Campus personnel behavior trajectory monitoring method, system, device and medium based on multi-source data fusion

Abstract

This invention belongs to the field of campus management and discloses a method for monitoring campus personnel behavior trajectories based on multi-source data fusion, including the following steps: acquiring multi-source data within the campus; preprocessing the multi-source data using a data weighted fusion algorithm; performing differentiated scene optimization processing on the collected facial images to output clear facial image frames and effective feature data; constructing real-time behavior trajectories of campus personnel based on a temporal modeling model using clear facial image frames, effective feature data, and preprocessed multi-source data; analyzing the real-time behavior trajectories through an anomaly detection mechanism to determine the trajectory anomaly status and classify the anomaly level; executing multi-channel alarm push according to the anomaly level; visually restoring the real-time behavior trajectories and historical behavior trajectories based on a campus spatial model; and absorbing new data and anomaly judgment results through an incremental learning algorithm to update the parameters of the temporal modeling model, anomaly detection mechanism, and differentiated scene optimization processing related models.

Description

Technical Field

[0001] This invention belongs to the field of campus management technology, and in particular relates to a method, system, device and medium for monitoring campus personnel behavior trajectories based on multi-source data fusion. Background Technology

[0002] With the deepening of smart campus construction, campus personnel flow is frequent and their geographical distribution is wide. Traditional trajectory monitoring solutions suffer from problems such as insufficient data fusion, low trajectory modeling accuracy, and lagging anomaly detection. In existing technologies, some solutions rely on a single data source or simple rules to determine anomalies, lacking the ability to deeply integrate multi-source data; some solutions use traditional machine learning algorithms for trajectory analysis, which makes it difficult to capture the temporal dependencies and complex anomaly patterns of trajectories; at the same time, trajectory visualization is mostly limited to a two-dimensional plane, resulting in insufficient backtracking accuracy and intuitiveness.

[0003] Of particular note is the significant challenge of differential recognition in campus facial recognition scenarios (lighting, pose, occlusion). Existing anomaly detection primarily focuses on mismatches between time, location, and behavioral calendars, lacking specific detection mechanisms for actions involving climbing over walls, railings, or other impassable areas. While geofencing algorithms exist, they are mostly applied to boundary control of authorized activity areas, failing to incorporate the characteristics of campus physical barriers (such as height and material) to construct accurate digital boundary models. This makes it difficult to effectively identify climbing-related anomalies, resulting in untimely warnings for safety hazards like climbing over walls and railings, impacting the comprehensiveness of campus security. Therefore, there is an urgent need for a technical solution that integrates multi-source data, specifically optimizes differential recognition and detection of climbing over impassable areas, and combines high-precision trajectory construction with intelligent anomaly detection to improve the intelligence and reliability of campus trajectory monitoring. Summary of the Invention

[0004] This application addresses the problems existing in the prior art by proposing a method, system, device, and medium for monitoring campus personnel behavior trajectories based on multi-source data fusion. Through core technologies such as illumination normalization, posture correction, occlusion completion, three-level preprocessing, improved face detection, Transformer temporal modeling, and special detection of impassable areas, it solves the pain points of differentiated face recognition and lack of trespass detection. It achieves accurate trajectory construction, intelligent identification of anomalies in all scenarios, and three-dimensional visualization backtracking, providing comprehensive technical support for campus security and prevention.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] Firstly, a method for monitoring campus personnel behavior trajectories based on multi-source data fusion includes the following steps: S1. Acquiring terminal data, device location data, and behavior calendar data within the campus; preprocessing the data using a multi-source data weighted fusion algorithm; optimizing the collected facial images through differentiated scenarios; and outputting clear facial image frames and effective feature data; S2. Constructing real-time behavior trajectories of campus personnel based on a temporal modeling model using the clear facial image frames, effective feature data, and preprocessed multi-source data; S3. Analyzing the real-time behavior trajectories through an anomaly detection mechanism, including trajectory boundary detection in trespassable areas, determining the trajectory anomaly status, and classifying the anomaly level; S4. Executing multi-channel alarm pushes based on the anomaly level, supporting the trajectories of historical behavior through terminal devices; S5. Visualizing and displaying the real-time and historical behavior trajectories based on a campus spatial model; S6. Updating the parameters of the temporal modeling model, anomaly detection mechanism, and differentiated scenario optimization models by absorbing new data and anomaly determination results through an incremental learning algorithm.

[0007] Optionally, in step S1, the terminal data includes video stream data, facial recognition data, event trigger data, and registration data generated by cameras, access control devices, facial recognition authentication terminals, and medical treatment terminals deployed on campus; the behavior calendar data includes one or more of the following: class schedule data, meal time data, bedtime data, library usage data, and medical appointment data; and the device location data includes the physical coordinate information of the device and the site level attribute information.

[0008] Optionally, step S1 includes the following steps:

[0009] S11. Real-time data collection from terminals via cameras, access control and authentication machines, electronic class signs, and medical clinic terminals deployed on campus;

[0010] S12. Collect equipment location data such as physical location coordinates and site hierarchy attributes, as well as behavioral calendar data such as class schedules, meals, sleep, and library usage.

[0011] S13. Employ a multi-source data weighted fusion algorithm to perform clearing, deduplication, and spatiotemporal alignment processing on terminal data, device location data, and behavior calendar data to obtain preprocessed multi-source data;

[0012] S14. The video stream data is enhanced by a three-level preprocessing model of illumination normalization, pose correction, and occlusion completion to output clear face image frames. The three-level preprocessing model includes an illumination compensation module based on Retinex-Net, a pose correction module based on 3DMM, and an occlusion feature reconstruction module based on Generative Adversarial Network (GAN).

[0013] S15. For the enhanced face image frame, the improved RetinaFace model is used to extract face key points and trajectory point data. The improved RetinaFace model introduces an attention mechanism and a multi-scale feature fusion module to improve the detection accuracy in differentiated scenarios.

[0014] Optionally, step S2 includes the following steps:

[0015] S21. Using the unique identifier of the person as an index, extract features such as timestamp, location coordinates, device identifier, and facial feature vector from the preprocessed multi-source data to construct a trajectory feature vector;

[0016] S22. Using the Transformer temporal modeling model, the input trajectory feature vector sequence is used to capture the dependency relationship of trajectory points at different time steps through the self-attention mechanism, and the output is a continuous, smooth real-time behavioral trajectory of each person. The trajectory includes attributes such as identity information, temporal location sequence, device interaction records, and dwell time.

[0017] S23. The trajectory is optimized by using the Kalman filter algorithm to fill in missing data segments, correct abnormal drift trajectory points, and ensure trajectory continuity.

[0018] Optionally, step S3 includes the following steps:

[0019] S31. Construct a digital boundary model for inaccessible areas of the campus, and use a Geographic Information System (GIS) to label the physical boundary coordinates and height attributes of walls and railings;

[0020] S32. Map the spatiotemporal coordinate information of the real-time behavior trajectory to the digital boundary model to establish a spatial association between the trajectory and the impassable area;

[0021] S33. Analyze whether the trajectory points cross the preset threshold range of the digital boundary model through the spatial collision detection algorithm to determine whether there is a climbing behavior;

[0022] S34. Anomaly levels are classified based on anomaly type, scope of impact, and urgency, including Level 1 anomalies, Level 2 anomalies, and Level 3 anomalies.

[0023] Optionally, step S4 includes the following steps:

[0024] S41. Based on the level of abnormality, a notification is sent through a multi-channel alarm push mechanism. Level 1 abnormalities trigger four channels: platform push, APP push, SMS, and campus broadcast. Level 2 abnormalities trigger the first three channels. Level 3 abnormalities trigger the first two channels.

[0025] S42. Based on the time window retrieval algorithm, it supports the input of identity identifiers and time ranges via PC / mobile terminal to quickly trace back historical trajectories with trajectory tracing accuracy down to the second level.

[0026] Optionally, step S5 includes the following steps:

[0027] S51. Receive campus floor plan or 3D model uploads and map the device location coordinates to 3D space using SLAM technology;

[0028] S52. It adopts a 3D point cloud reconstruction algorithm and combines trajectory time series data to dynamically restore the movement trajectory of personnel in the 3D model, and marks the time, equipment and event information corresponding to the trajectory points. It supports trajectory scaling, rotation and time axis dragging for viewing.

[0029] Secondly, the present invention provides a campus personnel behavior trajectory monitoring system based on multi-source data fusion, used to implement the campus personnel behavior trajectory monitoring method based on multi-source data fusion as described in the first aspect, comprising:

[0030] The multi-source data acquisition and preprocessing module is used to perform the operation in step S1, including a terminal data acquisition unit, a location data storage unit, a calendar data synchronization unit, a weighted fusion processing unit, a three-level preprocessing unit, and an improved RetinaFace detection unit;

[0031] The temporal trajectory construction module is used to perform the operation in step S2, including a feature vector extraction unit, a Transformer modeling unit, and a Kalman filter optimization unit;

[0032] The multi-model anomaly detection and classification module is used to perform the operation in step S3, including a rule matching detection unit, a deep learning detection unit, and a hierarchical clustering classification unit.

[0033] The intelligent alarm and trajectory backtracking module is used to perform the operation of step S4, including a multi-channel push unit and a time window retrieval unit;

[0034] The 3D trajectory visualization module is used to perform the operation in step S5, including a 3D spatial mapping unit and a point cloud reconstruction and fitting unit;

[0035] The model adaptive optimization module is used to perform the operation in step S6, including the incremental learning unit and the model parameter update unit.

[0036] Thirdly, the present invention provides a computer device, the computer device including a memory, a processor and a computer program, wherein when the computer program is executed by the processor, it implements the campus personnel behavior trajectory monitoring method based on multi-source data fusion as described in the first aspect.

[0037] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the campus personnel behavior trajectory monitoring method based on multi-source data fusion as described in the first aspect.

[0038] The beneficial effects of this application are as follows:

[0039] (1) This application achieves accurate identification of wall and railing climbing behavior by combining GIS digital modeling, spatial collision detection algorithm and intelligent algorithm verification, thus solving the pain point of existing technology lacking effective detection methods for climbing anomalies.

[0040] (2) Through a three-level preprocessing model of illumination normalization, pose correction and occlusion completion, combined with improved RetinaFace detection, the face detection accuracy is ≥98.5%, providing reliable identity support for trajectory association and anomaly recognition.

[0041] (3) This application has full-scene coverage of anomaly detection and precise hierarchical response: It integrates the detection of time anomalies, location anomalies, event anomalies and climbing anomalies to form a full-dimensional anomaly prevention and control system. The first-level anomaly is linked to the on-site sound and light alarm to improve the efficiency of emergency response. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating a scenario application of the campus personnel behavior trajectory monitoring method based on multi-source data fusion, as described in Embodiment 1 of this application.

[0043] Figure 2 This is a schematic diagram of a campus personnel behavior trajectory monitoring method based on multi-source data fusion, as described in Embodiment 1 of this application.

[0044] Figure 3 This is a schematic diagram of the anomaly detection mechanism steps in Embodiment 1 of this application;

[0045] Figure 4 This is a schematic diagram of a campus personnel behavior trajectory monitoring system based on multi-source data fusion, as described in Embodiment 2 of this application.

[0046] Figure 5 This is a structural diagram of the electronic device shown in Embodiment 3 of this application.

[0047] Figure label:

[0048] 300. Electronic device; 301. Processor; 302. Communication bus; 303. User interface; 304. Network interface; 305. Memory. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] like Figure 1 The diagram illustrates a scenario application of the campus personnel behavior trajectory monitoring method based on multi-source data fusion proposed in this application. Currently, campus multi-source monitoring data operates independently, resulting in data fragmentation, poor trajectory tracking accuracy, and an inability to accurately identify abnormal behaviors such as climbing over walls, making it difficult to achieve unified management and risk warning of personnel dynamics. This solution integrates multi-terminal data, optimizes facial recognition and trajectory modeling, and achieves precise monitoring of campus personnel across the entire process.

[0051] Example 1:

[0052] like Figures 2-3 As shown, a method for monitoring campus personnel behavior trajectories based on multi-source data fusion includes the following steps:

[0053] S1. Acquire terminal data, device location data, and behavior calendar data within the campus. Preprocess the data using a multi-source data weighted fusion algorithm. Optimize the collected facial images based on differentiated scenarios to output clear facial image frames and effective feature data.

[0054] S2. Based on the time-series modeling model, the real-time behavioral trajectory of campus personnel is constructed according to the clear face image frames, effective feature data and preprocessed multi-source data, including continuous trajectory data containing identity association information and spatiotemporal coordinate information;

[0055] S3. Analyze real-time behavioral trajectories through an anomaly detection mechanism, including trajectory boundary crossing detection in impassable areas, determine the trajectory anomaly status and classify the anomaly level; the impassable areas include campus walls, railings and other physical isolation areas that prohibit personnel from entering or climbing over;

[0056] S4. Perform multi-channel alarm push according to the anomaly level, and support the tracing of historical behavior trajectories through terminal devices;

[0057] S5. Based on the campus space model, the real-time behavior trajectory and historical behavior trajectory are visualized and displayed.

[0058] S6. The parameters of the time series modeling model, anomaly detection mechanism and differentiated scenario optimization processing related models are updated by absorbing new data and anomaly judgment results through incremental learning algorithm.

[0059] This embodiment achieves accurate identification of wall and railing climbing behavior through GIS digital modeling, spatial collision detection algorithms, and intelligent algorithm verification, addressing the pain point of existing technologies lacking effective detection methods for climbing-related anomalies. Through a three-level preprocessing model of illumination normalization, posture correction, and occlusion completion, combined with improved RetinaFace detection, the face detection accuracy is ≥98.5%, providing reliable identity support for trajectory association and anomaly identification. Anomaly detection covers all scenarios, with precise hierarchical response: integrating time anomaly, location anomaly, event anomaly, and climbing anomaly detection to form a comprehensive anomaly prevention and control system. Level 1 anomalies trigger on-site audio-visual alarms, improving emergency response efficiency. Enhanced visualization and backtracking capabilities: Impassable areas and boundary-crossing trajectories are highlighted in the 3D model, supporting trajectory and video-linked backtracking, allowing managers to intuitively reconstruct the climbing process and aiding in post-incident investigation. Strong adaptive optimization capabilities: incremental learning algorithms continuously optimize the model and detection parameters to adapt to changes in the campus environment, ensuring long-term stable operation.

[0060] In step S1, the terminal data includes video stream data, facial recognition data, event triggering data, and registration data generated by cameras, access control devices, facial recognition authentication terminals, and medical treatment terminals deployed on campus; the device location data includes the physical coordinate information of the device and the site level attribute information; the behavior calendar data includes class schedule data, meal time data, bedtime data, library usage data, medical appointment data, etc.

[0061] In this embodiment, step S1 includes the following steps:

[0062] S11. Real-time data collection is achieved through devices deployed on campus, including cameras, access control kiosks, electronic class signs, and medical clinic terminals. Cameras cover key areas such as teaching buildings, dormitories, canteens, and playgrounds, collecting real-time video stream data at a frame rate of 25fps and a resolution of 1080P. Access control kiosks and electronic class signs collect student identification (student ID / employee ID) and timestamps when users scan their faces. An AI edge computing device analyzes the camera video stream in real time, triggering event data and recording the location and time when a person falls or crosses a pre-set trip line. The medical clinic terminal collects students' identity information, treatment time, and symptom type when seeking medical care.

[0063] S12. Collect equipment location data such as physical location coordinates and site hierarchy attributes, as well as behavioral calendar data such as class schedules, meal times, bedtimes, and library usage. Location data is obtained by combining GPS positioning with the campus GIS system to acquire the physical location coordinates of each terminal device and label the site hierarchy (e.g., "Classroom 302, 3rd Floor, Building 1") and site security level (low / medium / high). Behavioral calendar data is collected through interfaces with the campus academic affairs system, logistics management system, and library management system to synchronize student / faculty / staff class schedules (including class times and classroom locations), meal times (breakfast 6:30-8:30, lunch 11:30-13:30, dinner 17:30-19:30), bedtimes (22:00-6:00 the next day), and library borrowing records in real time.

[0064] S13. A multi-source data weighted fusion algorithm is used to perform deduplication, spatiotemporal alignment, and clearing of terminal data, device location data, and behavior calendar data to obtain preprocessed multi-source data; the specific formula is shown below:

[0065] (1)

[0066] in, The merged data; This includes terminal data, location data, behavioral calendar data, and medical terminal data; Let represent the weight of the i-th data category. Weight calculation is based on data credibility scores, which are derived by weighting data transmission success rate, data error rate, and real-time performance. During the fusion process, the data is first cleaned (null values ​​and duplicates are removed). Then, a spatiotemporal alignment algorithm is used to align data of the same person collected from different devices by timestamp (time deviation ≤ 1 second). Finally, a structured fused dataset is output, with each data entry containing five dimensions: identity identifier, timestamp, location coordinates, device type, and data type.

[0067] S14. The video stream data is enhanced by a three-level preprocessing model consisting of illumination normalization, pose correction, and occlusion completion, outputting clear face image frames and effective feature data. The three-level preprocessing model includes an illumination compensation module based on Retinex-Net, a pose correction module based on 3DMM, and an occlusion feature reconstruction module based on Generative Adversarial Network (GAN).

[0068] S15. For the enhanced face image frame, the improved RetinaFace model is used to extract face key points and trajectory point data. The improved RetinaFace model introduces an attention mechanism and a multi-scale feature fusion module to improve the detection accuracy in differentiated scenarios.

[0069] In step S14, due to the complexity of campus camera deployment, various lighting scenarios exist, including direct sunlight, backlighting, and low light, resulting in uneven brightness and loss of detail in facial images. Therefore, the operation flow of the Retinex-Net-based illumination compensation module in this application includes the following steps:

[0070] S141a. Convert the RGB of the video frame to the HSV color space, and separate the luminance channel V and the chrominance channels H and S to avoid the effect of lighting adjustments on color accuracy.

[0071] S141b. Multi-scale Retinex decomposition is applied to the luminance channel V to calculate the reflection component R (inherent facial features) and the illumination component L (affected by ambient light). The specific formulas are shown below:

[0072] (2)

[0073] in, The Gaussian filter kernel is used; V is the HSV spatial luminance channel. This is a convolution operation; R is the reflection component.

[0074] S141c. Adaptive enhancement of the illumination component L is performed, and the dynamic range of brightness is adjusted through gamma correction. The specific formula is as follows:

[0075] (3)

[0076] in, L represents the enhanced illumination component; L represents the original illumination component. The gamma factor is dynamically adjusted based on light intensity, especially in low-light scenes. =0.3-0.5, strong light scene =1.5-2.0.

[0077] S141d. The enhanced reflection component R is recombined with the original chroma channels H and S, converted back to the RGB color space, and the image after illumination normalization is published.

[0078] In this embodiment, changes in posture such as walking, looking down, and turning one's face can cause facial feature points to shift, affecting recognition accuracy. Therefore, the operation flow of the 3DMM-based posture correction module in step S14 of this application includes the following steps:

[0079] S142a. Extract the coordinates of key feature points such as eyes, nose, mouth, and contours in a face image using a facial landmark detection algorithm.

[0080] S142b. Register the key point coordinates with the average face model of the 3D deformation model 3DMM, and solve for the attitude parameters and shape parameters; the state parameters include pitch angle α, yaw angle β, and roll angle γ.

[0081] S142c. Construct a rotation matrix based on the pose parameters, adjust the pose of the 3DMM model, and generate a 3D face model consistent with the input face pose; the formula for calculating the rotation matrix is ​​shown below:

[0082] (4)

[0083] Where R is the rotation matrix; The pitch angle; Yaw angle; This is the roll angle.

[0084] S142d. The adjusted 3D face model is projected onto a 2D plane using orthographic projection to obtain a frontal face image after pose correction; the calculation formula for orthographic projection is shown below:

[0085] (5)

[0086] in, Here are the vertex coordinates of the 3DMM model; T is the translation vector. These are the corrected two-dimensional coordinates.

[0087] In this embodiment, because students wearing masks, glasses, or other occlusions obscure key features such as the eyes and nose, recognition fails. This application employs a generative adversarial network (GAN) using U-Net+PatchGAN to achieve accurate completion of occluded regions. Step S14, the operation flow of the occlusion feature reconstruction module based on the GAN includes the following steps:

[0088] S143a. A generator is constructed using a U-Net structure. The input is a face image that is covered by a mask / glasses. The encoder extracts deep features, and the decoder outputs a face image with the occluded areas filled in.

[0089] S143b. Construct a PatchGAN discriminator to distinguish between the completed face image output by the generator and the real unoccluded face image, and calculate the adversarial loss function. The formula for calculating the total loss function is as follows:

[0090] (6)

[0091] in; Total loss; To combat the losses; To perceive loss; This is the L1 loss.

[0092] Combat losses To ensure the realism of the completed image, the specific formula is as follows:

[0093] (7)

[0094] Where x represents a real, unobstructed human face; z represents an obstructed human face; Output to the generator; E is the output of the discriminator; E is the expected value. This represents the true data distribution. To obscure the data distribution.

[0095] S143c. Introducing perceptual loss To ensure feature consistency in the completed image, features from the 1st, 3rd, and 5th convolutional layers of the VGG16 network are used for calculation, as shown in the following formula:

[0096] (8)

[0097] in, This represents the feature mapping of the i-th layer of the VGG16 network. , , These are the height, width, and number of channels of the feature map at layer i, respectively. L2 norm; L1 loss To ensure pixel consistency in the completed image, the specific formula is as follows:

[0098] (9)

[0099] in, It is an L1 norm; This represents the total number of pixels in the image.

[0100] S143d. By alternating the training of the generator and discriminator until the loss converges, the occluded region is accurately filled in. Several campus scene occluded face images and corresponding unoccluded face images are paired for training. After several iterations, the loss converges to below 0.02, and the cosine similarity between the filled region and the real region is ≥92%, effectively restoring the key features of the occluded region.

[0101] In this embodiment, in step S15, based on the three-level preprocessing, an improved RetinaFace model is used to extract facial features and trajectory point data, further improving the detection accuracy in differentiated scenarios. Specifically, the improvements of the improved RetinaFace model include the following steps:

[0102] S151. Add a coordinate attention module to the output of the backbone network MobileNetV3. By performing global average pooling on the x and y directions of the feature map, spatial location information is captured, enhancing the feature extraction capability of key facial regions (eyes and nose). The attention weight calculation formula is as follows:

[0103] (10)

[0104] in, F is the coordinate attention weight map; F is the MobileNetV3 output feature map; Global average pooling in the x-direction; Global average pooling in the y-direction; It is a two-layer fully connected network; The Sigmoid activation function is used to output an attention weight map.

[0105] S152. The Feature Pyramid Network (FPN) introduces cross-scale feature fusion, which fuses feature images through an adaptive weighted summation method. The weight parameters are automatically learned through backpropagation, which improves the detection capability of small-sized faces and blurred faces.

[0106] S153. Add a face occlusion judgment branch to the detection head and output the occlusion probability (0-1). When the occlusion probability is ≥0.5, the occlusion completion module in step S143 is automatically called for secondary processing.

[0107] S154. For the detected face image, extract the face feature vector (obtained by mapping key point features through a fully connected layer), and combine it with the device position coordinates, pose parameters, and occlusion probability to form trajectory point data. Each trajectory point contains position coordinates, face feature vector, pose parameters, occlusion probability, and timestamp to ensure a unique association between the person's identity and the trajectory.

[0108] In this embodiment, step S2 uses a deep learning model to capture the temporal dependencies of the trajectory, thus solving the problems of missing data and discontinuous trajectories in traditional trajectory construction. Specifically, step S2 includes the following steps:

[0109] S21. Using the unique identifier of the person as an index, extract features such as timestamp, location coordinates, device identifier, and facial feature vector from the preprocessed multi-source data to construct a 7-dimensional trajectory feature vector. The specific formula is shown below:

[0110] (11)

[0111] Where V is a 7-dimensional trajectory feature vector; The unique hot code for the identity identifier; T is the Unix timestamp; (X, Y) are the physical location coordinates of the device; Encode the device type; The degree of matching between trajectory and behavior calendar data; 128-dimensional facial feature vectors extracted for the improved RetinaFace model; This represents the occlusion probability.

[0112] S22. The Transformer temporal modeling model is adopted. It takes a sequence of trajectory feature vectors as input and captures the dependencies between trajectory points at different time steps through a self-attention mechanism. The output is a continuous, smooth, real-time behavioral trajectory for each person. The trajectory includes attributes such as identity information, temporal location sequence, device interaction records, and dwell time. Specifically, the Transformer temporal modeling model structure includes: an input layer that receives a 7-dimensional feature vector sequence of length N (N=3600, corresponding to a 1-hour trajectory); an embedding layer that maps the 7-dimensional features to a 128-dimensional embedding vector through a fully connected layer, adding positional encoding to preserve temporal information; an encoder layer with 4 encoder layers, each containing a multi-head self-attention mechanism (8 heads, 16 attention dimensions) and a Feed-Forward network (512 hidden layer dimensions); and an output layer that outputs a 3-dimensional trajectory vector through a fully connected layer. ,in, , Predict the location for the next time step. The state is considered as stationary (0 = moving, 1 = stationary). The training process uses real trajectory data from several students / faculty members on campus (each trajectory lasts 24 hours). The ratio of training set, validation set, and test set is 7:2:1. The Adam optimizer is used, and the loss function is MSE (mean squared error). Training stops when the loss on the validation set is ≤0.01. The final trajectory prediction position error is ≤0.5 meters.

[0113] S23. The trajectory is optimized using the Kalman filter algorithm to fill in missing data segments, correct abnormal drift points, and ensure trajectory continuity. Specifically, the Kalman filter algorithm optimization process involves the following steps:

[0114] S231. Using the trajectory vector output by the Transformer as the initial state, the specific formula is as follows:

[0115] (12)

[0116] in, Here is the Kalman filter state vector; X and Y are the position coordinates. , The velocity is in the x / y direction.

[0117] S232. Predict the next state based on the motion model, using the following formula:

[0118] (13)

[0119] in, For the predicted state vector; F is the updated state vector; F is the state transition matrix; B is the control matrix. It is the acceleration vector; This is process noise.

[0120] S233. Based on the actual collected trajectory point data, update the state estimate using Kalman gain. The specific formula is shown below:

[0121] (14)

[0122] in, Kalman gain; H is the prediction error covariance matrix; H is the observation matrix; To observe the noise covariance matrix; Find the inverse of the matrix.

[0123] S234. Repeated Prediction-Update Step: Fills in missing data segments (maximum missing time ≤ 30 seconds), corrects drift trajectory points, and outputs a continuous, smooth real-time behavior trajectory. The specific formula is shown below:

[0124] (15)

[0125] in, These are the actual observed coordinates.

[0126] In this embodiment, step S3 employs a dual-detection framework of rule matching and intelligent algorithms. Rule matching is based on the matching degree between spatiotemporal information and behavioral calendar data. The intelligent algorithm is one or more of the following: Isolation Forest algorithm, autoencoder model, and clustering algorithm. Specifically, rule matching includes time anomaly detection, location anomaly detection, and boundary crossing detection in impassable areas. Time anomaly detection determines the location by calculating the deviation between the trajectory timestamp and a preset time interval. Location anomaly detection uses a geofencing algorithm to determine whether the trajectory location exceeds a preset activity area. The intelligent algorithm determines abnormal trajectory points by calculating the anomaly score or reconstruction error of trajectory features and combining it with a preset threshold. For trajectory data in impassable areas, the intelligent algorithm analyzes the velocity changes and abrupt changes in direction of the trajectory to assist in verifying the authenticity of the boundary crossing behavior.

[0127] Specifically, the anomaly detection mechanism also includes the collection of abnormal events, combined with multi-event timeline analysis to improve the accuracy of boundary crossing detection in unpassable areas and eliminate false triggering situations.

[0128] In this embodiment, step S3 employs a dual framework of rule matching and intelligent algorithms, covering both traditional time and location anomaly detection and adding a specialized detection for behaviors that traverse impassable areas, achieving accurate identification and classification of anomalies across all scenarios. Specifically, the trajectory boundary crossing detection for impassable areas includes the following steps:

[0129] S311. Construct a digital boundary model for impassable areas on campus, using a Geographic Information System (GIS) to label the physical boundary coordinates and height attributes of walls and railings. Specifically, the digital boundary model is constructed using GIS technology to accurately model impassable areas on campus. First, three-dimensional coordinate data of walls and railings are obtained through LiDAR or on-site surveys. This data includes the longitude, latitude, and height information of the bottom and top boundaries. Second, polygon fitting technology is used to generate closed boundary contours from the discrete coordinate data, forming a digital model of the impassable areas. Finally, the threshold ranges for boundary crossing are labeled: a vertical threshold ≥ 1.2 meters (corresponding to an adult's shoulder height, considered climbing or scaling), and a horizontal threshold ≥ 0.5 meters (corresponding to the horizontal distance across the boundary, considered crossing or scaling).

[0130] Specifically, the formula for calculating the boundary of a closed polygon is as follows:

[0131] (16)

[0132] in, Let i be the i-th boundary segment; Let i be the three-dimensional coordinates of the i-th vertex of the polygon. ; Let be the parametric equation of the i-th boundary line segment; n is the number of boundary vertices; and t is the line segment parameter.

[0133] S312. Map the spatiotemporal coordinate information of the real-time behavior trajectory to the digital boundary model to establish a spatial association between the trajectory and the impassable area. Specifically, the real-time behavior trajectory data (including spatiotemporal coordinates and identity information) generated in step S2 is mapped to the same spatial coordinate system of the digital boundary model through a coordinate transformation algorithm (such as Gauss-Kruger projection) to establish a spatial association between the trajectory points and the impassable area; during the mapping process, a time synchronization mechanism is used to ensure that the timestamp of the trajectory point is accurately matched with the spatial data of the boundary model to avoid false detection caused by spatiotemporal misalignment.

[0134] S313. Analyze whether the trajectory point crosses the preset threshold range of the digital boundary model using a spatial collision detection algorithm to determine whether a breaching behavior exists. Specifically, spatial collision detection is implemented using the ray method. Rays are emitted from the trajectory point to the contour line of the digital boundary model, and the number of intersections between the ray and the contour line is counted. If the number of intersections is odd, and the vertical height of the trajectory point exceeds the top height threshold of the boundary model (≥1.2 meters), or the horizontal distance exceeds the horizontal threshold of the boundary model (≥0.5 meters), then a breaching behavior is initially determined. To improve detection accuracy, an intelligent algorithm is introduced to assist in verification: the isolated forest algorithm is used to analyze the characteristics of the trajectory point, such as speed changes (breaching behavior is usually accompanied by sudden speed changes in a short period of time, such as a decrease in speed during climbing and an increase in speed after breaching) and direction changes (the trajectory direction is perpendicular to the boundary or at an acute angle). An anomaly score is calculated, and when the score is ≥0.85, the breaching anomaly is confirmed, and false triggers caused by equipment errors or normal personnel approaching the boundary are excluded.

[0135] trajectory points The equations for the emitted rays are:

[0136] (17)

[0137] in, Here are the equations for the ray parameters; Let be the unit direction vector of the ray, taken as the direction perpendicular to the boundary contour line; These are the coordinates of the trajectory points.

[0138] Rays and boundary segments The intersection point determination requires satisfying a simultaneous equation:

[0139] (18)

[0140] Where s is the line segment parameter; t is the ray parameter; N is the number of intersection points; when t≥0 and s∈[0,1], there are valid intersection points. Count the number of valid intersection points N. If N is odd, the trajectory point is located inside the polygon boundary.

[0141] The preliminary criteria for determining the boundary threshold are as follows:

[0142] (19)

[0143] in, This represents the top height of the boundary model; Let the trajectory point Q be the boundary line segment. Horizontal distance; This is the vertical boundary threshold; This is the horizontal boundary threshold.

[0144] To improve detection accuracy, intelligent algorithms are introduced to assist in verification:

[0145] 1. Calculation of rate of change of velocity:

[0146] (20)

[0147] (twenty one)

[0148] in, The rate of change of velocity; The velocity of the k / k-1th trajectory point; The velocity of the k-th trajectory point; Let K be the coordinates of the k-th trajectory point; For time intervals.

[0149] Calculation of the angle between the trajectory directions:

[0150] (twenty two)

[0151] in, The angle between the trajectory directions; This is the direction vector of the previous trajectory segment; This is the direction vector of the current segment's trajectory; It is the dot product of vectors; Let be the vector magnitude.

[0152] Isolated Forest Anomaly Score:

[0153] (twenty three)

[0154] Where S represents the anomaly score; This represents the average path length of the trajectory points in the isolated forest. Here, n is the normalization coefficient; n is the number of training samples. It is a harmonic number.

[0155] The final criteria for determining an out-of-bounds exception are:

[0156] (twenty four)

[0157] Where S is the isolated forest anomaly score; the initial boundary crossing is the rule matching result; and the final boundary crossing is the final judgment result.

[0158] S314. Anomaly levels are classified based on anomaly type, scope of impact, and urgency, including Level 1 anomalies, Level 2 anomalies, and Level 3 anomalies. Among them, anomalies triggered by events such as climbing over impassable areas or falling / tripping intrusion (involving personal safety or campus security red lines, with the highest urgency) correspond to Level 1 anomalies; Level 2 anomalies correspond to a combination of time anomalies and location anomalies; and Level 3 anomalies correspond to a single time anomaly or location anomaly.

[0159] In this embodiment, step S4 includes the following steps:

[0160] S41. Based on the level of abnormality, a notification is sent through a multi-channel alarm push mechanism. Level 1 abnormalities trigger four channels: platform push, APP push, SMS, and campus broadcast. Level 2 abnormalities trigger the first three channels. Level 3 abnormalities trigger the first two channels.

[0161] S42. Based on the time window retrieval algorithm, it supports the input of identity identifiers and time ranges via PC / mobile terminal to quickly trace back historical trajectories with trajectory tracing accuracy down to the second level.

[0162] In this embodiment, step S5 includes the following steps:

[0163] S51. Receive the uploaded campus floor plan or 3D model, and map the device location coordinates to the 3D space using SLAM technology; specifically, the 3D space mapping uses SLAM technology to map the device location coordinates to the campus 3D model with a mapping error ≤ 0.3 meters.

[0164] S52. Employing a 3D point cloud reconstruction algorithm combined with trajectory time-series data, it dynamically reconstructs personnel movement trajectories in a 3D model, labeling the time, equipment, and event information corresponding to trajectory points, and supporting trajectory zooming, rotation, and timeline dragging for viewing. Specifically, in 3D point cloud reconstruction and fitting, each trajectory point corresponds to a point cloud in the 3D model, and a smooth trajectory line is generated using Bézier curve fitting. Viewpoint rotation and zooming are supported, and clicking on a trajectory point allows viewing video screenshots and identity information.

[0165] In this embodiment, step S6 involves continuously optimizing model performance using an incremental learning algorithm, periodically collecting new trajectory data, anomaly detection results, and differentiated scene face images; fusing incremental data with original training data to perform small-batch fine-tuning (BatchSize=32, learning rate=0.0001) on the Transformer model, autoencoder model, and improved RetinaFace model; dynamically adjusting the weights of the multi-source data weighted fusion algorithm and the parameters of the three-level preprocessing model; and conducting model performance evaluation quarterly, triggering full retraining when the anomaly detection accuracy is below 95%.

[0166] Example 2:

[0167] like Figure 4 As shown, this embodiment provides a campus personnel behavior trajectory monitoring system based on multi-source data fusion, used to implement the campus personnel behavior trajectory monitoring method based on multi-source data fusion as described in Embodiment 1, including:

[0168] The multi-source data acquisition and preprocessing module includes a terminal data acquisition unit, a location data storage unit, a calendar data synchronization unit, a weighted fusion processing unit, a three-level preprocessing unit (illumination compensation subunit, attitude correction subunit, occlusion completion subunit), and an improved RetinaFace detection unit.

[0169] The time-series trajectory construction module includes a feature vector extraction unit, a Transformer modeling unit, and a Kalman filter optimization unit.

[0170] The multi-model anomaly detection and classification module includes a rule matching detection unit, a deep learning detection unit, and a hierarchical clustering classification unit.

[0171] The intelligent alarm and trajectory tracing module includes a multi-channel push unit and a time window retrieval unit.

[0172] The 3D trajectory visualization module includes a 3D spatial mapping unit and a point cloud reconstruction and fitting unit.

[0173] The model adaptive optimization module includes an incremental learning unit and a model parameter update unit.

[0174] Example 3:

[0175] like Figure 5 As shown, this embodiment provides an electronic device, which may include: at least one processor, at least one network interface, a user interface, a memory, and at least one communication bus.

[0176] The communication bus can be used to enable communication between the various components mentioned above.

[0177] The user interface may include buttons, and optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0178] The network interface may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0179] The processor may include one or more processing cores. It connects various parts of the electronic device via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various functions and process data. Optionally, the processor can be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0180] The memory may include RAM or ROM. Optionally, the memory may include a non-transitory computer-readable medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. The memory, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an evaluation application. The processor can be used to call the evaluation application stored in the memory and execute the steps of the campus personnel behavior trajectory monitoring method based on multi-source data fusion mentioned in the foregoing embodiments.

[0181] Example 4:

[0182] This embodiment provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described instructions. Figure 2 One or more steps in the illustrated embodiment. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0183] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0184] Those skilled in the art will understand that all or part of the processes in the method of Embodiment 1 described above can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and the implementation scheme can be combined arbitrarily.

[0185] The above-described specific embodiments are preferred embodiments of a campus personnel behavior trajectory monitoring method, system, device, and medium based on multi-source data fusion according to this application. They are not intended to limit the specific scope of this application. The scope of this application includes but is not limited to these specific embodiments. All equivalent changes made in accordance with the shape and structure of this application are within the protection scope of this application.

Claims

1. A method for monitoring campus personnel behavior trajectories based on multi-source data fusion, characterized in that, Includes the following steps: S1. Acquire terminal data, device location data, and behavior calendar data within the campus. Preprocess the data using a multi-source data weighted fusion algorithm. Optimize the collected facial images based on differentiated scenarios to output clear facial image frames and effective feature data. S2. Based on the time-series modeling model, the real-time behavioral trajectory of campus personnel is constructed according to the clear face image frames, effective feature data and preprocessed multi-source data; S3. Analyze real-time behavioral trajectories through anomaly detection mechanisms, including trajectory boundary crossing detection in impassable areas, determining trajectory anomaly status and classifying anomaly levels; S4. Perform multi-channel alarm push according to the anomaly level, and support the tracing of historical behavior trajectories through terminal devices; S5. Based on the campus space model, the real-time behavior trajectory and historical behavior trajectory are visualized and displayed. S6. The parameters of the time series modeling model, anomaly detection mechanism and differentiated scenario optimization processing related models are updated by absorbing new data and anomaly judgment results through incremental learning algorithm.

2. The campus personnel behavior trajectory monitoring method based on multi-source data fusion according to claim 1, characterized in that, In step S1, the terminal data includes video stream data, facial recognition data, event trigger data, and registration data generated by cameras, access control devices, facial recognition authentication terminals, and medical treatment terminals deployed on campus; the behavior calendar data includes one or more of the following: class schedule data, meal time data, bedtime data, library usage data, and medical appointment data; and the device location data includes the physical coordinate information of the device and the site level attribute information.

3. The campus personnel behavior trajectory monitoring method based on multi-source data fusion according to claim 1, characterized in that, Step S1 includes the following steps: S11. Real-time data collection from terminals via cameras, access control and authentication machines, electronic class signs, and medical clinic terminals deployed on campus; S12. Collect equipment location data such as physical location coordinates and site hierarchy attributes, as well as behavioral calendar data such as class schedules, meals, sleep, and library usage. S13. Employ a multi-source data weighted fusion algorithm to perform clearing, deduplication, and spatiotemporal alignment processing on terminal data, device location data, and behavior calendar data to obtain preprocessed multi-source data; S14. The video stream data is enhanced by a three-level preprocessing model of illumination normalization, pose correction, and occlusion completion to output clear face image frames. The three-level preprocessing model includes an illumination compensation module based on Retinex-Net, a pose correction module based on 3DMM, and an occlusion feature reconstruction module based on Generative Adversarial Network (GAN). S15. For the enhanced face image frame, the improved RetinaFace model is used to extract face key points and trajectory point data. The improved RetinaFace model introduces an attention mechanism and a multi-scale feature fusion module to improve the detection accuracy in differentiated scenarios.

4. The campus personnel behavior trajectory monitoring method based on multi-source data fusion according to claim 1, characterized in that, Step S2 includes the following steps: S21. Using the unique identifier of the person as an index, extract features such as timestamp, location coordinates, device identifier, and facial feature vector from the preprocessed multi-source data to construct a trajectory feature vector; S22. Using the Transformer temporal modeling model, the input trajectory feature vector sequence is used to capture the dependency relationship of trajectory points at different time steps through the self-attention mechanism, and the output is a continuous, smooth real-time behavioral trajectory of each person. The trajectory includes attributes such as identity information, temporal location sequence, device interaction records, and dwell time. S23. The trajectory is optimized by using the Kalman filter algorithm to fill in missing data segments, correct abnormal drift trajectory points, and ensure trajectory continuity.

5. The campus personnel behavior trajectory monitoring method based on multi-source data fusion according to claim 1, characterized in that, Step S3 includes the following steps: S31. Construct a digital boundary model for inaccessible areas of the campus, and use a Geographic Information System (GIS) to label the physical boundary coordinates and height attributes of walls and railings; S32. Map the spatiotemporal coordinate information of the real-time behavior trajectory to the digital boundary model to establish a spatial association between the trajectory and the impassable area; S33. Analyze whether the trajectory points cross the preset threshold range of the digital boundary model through the spatial collision detection algorithm to determine whether there is a climbing behavior; S34. Anomaly levels are classified based on anomaly type, scope of impact, and urgency, including Level 1 anomalies, Level 2 anomalies, and Level 3 anomalies.

6. The campus personnel behavior trajectory monitoring method based on multi-source data fusion according to claim 1, characterized in that, Step S4 includes the following steps: S41. Based on the level of abnormality, a notification is sent through a multi-channel alarm push mechanism. Level 1 abnormalities trigger four channels: platform push, APP push, SMS, and campus broadcast. Level 2 abnormalities trigger the first three channels. Level 3 abnormalities trigger the first two channels. S42. Based on the time window retrieval algorithm, it supports the input of identity identifiers and time ranges via PC / mobile terminal to quickly trace back historical trajectories with trajectory tracing accuracy down to the second level.

7. The campus personnel behavior trajectory monitoring method based on multi-source data fusion according to claim 1, characterized in that, Step S5 includes the following steps: S51. Receive campus floor plan or 3D model uploads and map the device location coordinates to 3D space using SLAM technology; S52. It adopts a 3D point cloud reconstruction algorithm and combines trajectory time series data to dynamically restore the movement trajectory of personnel in the 3D model, and marks the time, equipment and event information corresponding to the trajectory points. It supports trajectory scaling, rotation and time axis dragging for viewing.

8. A campus personnel behavior trajectory monitoring system based on multi-source data fusion, characterized in that, The method for monitoring campus personnel behavior trajectories based on multi-source data fusion as described in any one of claims 1-7 includes: The multi-source data acquisition and preprocessing module is used to perform the operation in step S1, including a terminal data acquisition unit, a location data storage unit, a calendar data synchronization unit, a weighted fusion processing unit, a three-level preprocessing unit, and an improved RetinaFace detection unit; The temporal trajectory construction module is used to perform the operation in step S2, including a feature vector extraction unit, a Transformer modeling unit, and a Kalman filter optimization unit; The multi-model anomaly detection and classification module is used to perform the operation in step S3, including a rule matching detection unit, a deep learning detection unit, and a hierarchical clustering classification unit. The intelligent alarm and trajectory backtracking module is used to perform the operation of step S4, including a multi-channel push unit and a time window retrieval unit; The 3D trajectory visualization module is used to perform the operation in step S5, including a 3D spatial mapping unit and a point cloud reconstruction and fitting unit; The model adaptive optimization module is used to perform the operation in step S6, including the incremental learning unit and the model parameter update unit.

9. A computer device, the computer device comprising a memory, a processor, and a computer program, characterized in that, When the computer program is executed by the processor, it implements the campus personnel behavior trajectory monitoring method based on multi-source data fusion as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the campus personnel behavior trajectory monitoring method based on multi-source data fusion as described in any one of claims 1-7.

Images