Smart campus student activity trajectory tracking method and system based on front and back end fusion learning

Abstract

The invention discloses an intelligent campus student activity trajectory tracking method and system based on front and rear end fusion learning, and relates to the technical field of intelligent campus security. An embedded front end collects videos of a plurality of monitoring cameras of a campus, carries out target detection and extracts primary features of student targets; uploading the extracted primary features and the video to a back-end server; and the back-end server extracts advanced features of human body postures and gaits from the video by using a human body posture estimation algorithm and a gait recognition algorithm, fuses the advanced features and the primary features to obtain fused features, and clusters all detected student targets by using the fused features, thereby realizing cross-camera identity unification of the same student under different cameras, and improving the identification accuracy of the student. Face detection and recognition are carried out in each clustering group, a student information base is matched to determine student IDs, and the student IDs, video timestamps and monitoring camera position information are stored; and the client retrieves target information according to the input student ID or photo, and automatically clips and generates a time-space continuous complete cross-camera trajectory tracking video. According to the invention, efficient and intelligent analysis of campus monitoring data is realized, and targets can be accurately associated and tracked in a cross-camera scene.

Description

Technical Field

[0001] The present invention relates to the field of smart campus security technology, and in particular to a method and system for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning. Background Art

[0002] With the increasing demand for campus security management, issues such as missing students and accidental injuries are receiving increasing attention. The traditional method of tracking missing students relies on manually reviewing video data from each campus surveillance camera. This requires a significant amount of time, effort, and manpower to screen, search, and compare massive amounts of video data, extracting key images of the target student. However, due to the large number of campus surveillance cameras, the large volume of video data, and the complexity of the content, manual search for target information is often inefficient and prone to omissions and misjudgments, significantly delaying the crucial time to find the missing student.

[0003] Furthermore, while the introduction of deep learning and artificial intelligence technologies in recent years has enabled significant progress in the automated detection and processing of video data, achieving a certain degree of automated target recognition and behavior analysis, practical applications still face numerous challenges. First, the videos currently captured by campus surveillance systems mostly come from fixed cameras, which significantly limits the shooting angles and image content. When students fail to reveal their faces in certain situations, traditional facial recognition algorithms struggle to capture effective features, resulting in missed targets. Second, data collection between different surveillance cameras is isolated, and there is a lack of effective data correlation mechanisms between cameras, making it challenging to track the continuous movement of the same target student across multiple cameras. Furthermore, in real-world scenarios, video databases often contain a large number of surveillance videos unrelated to the target being retrieved. Existing automated detection methods often suffer from low retrieval efficiency and slow response times when faced with this noisy data, making it difficult to rapidly locate targets.

[0004] In summary, the existing methods have the following defects:

[0005] 1. Missed detection: When students are not fully visible, automatic detection algorithms such as facial recognition cannot effectively capture target features, resulting in the omission of key images.

[0006] 2. Difficulty in cross-camera tracking: Due to differences in viewing angles and timeframes among surveillance cameras, there is a lack of effective correlation between video data, making it difficult to automatically integrate and form a continuous activity trajectory.

[0007] 3. Low data processing efficiency: Manual retrieval of massive video data is not only time-consuming and labor-intensive, but also prone to human error, which can delay the optimal time to find missing students.

[0008] 4. Noisy data interference: The large amount of video data irrelevant to the target stored in the database poses a serious challenge to existing automatic detection technology, further reducing the accuracy and efficiency of retrieval. Summary of the Invention

[0009] In order to overcome the defects in the above-mentioned prior art, the present invention provides a method and system for tracking student activity trajectories on a smart campus based on front-end and back-end fusion learning. It overcomes the shortcomings of traditional methods by integrating the deep learning model of artificial intelligence with efficient data processing means, realizes efficient and intelligent analysis of campus surveillance data, and can accurately associate and track targets in cross-camera scenarios.

[0010] To achieve the above object, the present invention adopts the following technical solutions, including:

[0011] A method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning includes the following:

[0012] Collect videos from campus surveillance cameras;

[0013] Use the target detection algorithm to detect students in the video, remove invalid video frames without students, and record the video timestamps and surveillance camera location information corresponding to valid video frames with students;

[0014] Utilize attribute recognition algorithms to perform multi-label attribute classification on students detected in valid video frames and extract primary features;

[0015] Use the human posture estimation algorithm and gait recognition algorithm to process the effective video frames, extract high-level features including human posture and gait, and fuse the high-level features with the primary features to obtain fused features;

[0016] Using the fusion features of each student, all detected students are clustered, so that the cross-camera identity of the same student is unified between different cameras;

[0017] Perform face detection and recognition in each cluster group, and perform identity matching in the student information database to obtain the student ID corresponding to the cluster group;

[0018] storing the student ID, video timestamp, and surveillance camera location information in a data storage unit;

[0019] Subsequently, the target information is retrieved from the data storage unit based on the student ID or photo, and the spatiotemporal continuous tracking video of the queried target is automatically edited and generated.

[0020] Preferably, the primary features include but are not limited to: gender, age group, top type and color, bottom type and color, shoe type and color, whether wearing a hat, whether carrying a backpack, whether wearing glasses, whether wearing a mask, hairstyle, and hair color.

[0021] Preferably, the OpenPose deep neural network is used to estimate the human pose in the video, and the specific method is as follows:

[0022] S5011, extract features from the input image and output a feature map F;

[0023] S5012, generating a confidence map of key points of the human body;

[0024] The first branch of the OpenPose network is based on the feature map F and outputs the confidence map of J key points, that is, the probability distribution map of the key points of the human body, and each key point corresponds to a confidence map; among them, S j (p) represents the confidence that the jth keypoint exists at pixel position p;

[0025] As a supervisory signal for the network, the manually annotated key point position information is converted into an ideal confidence map in the form of a Gaussian distribution:

[0026]

[0027] in, represents the ideal confidence that the jth keypoint exists at pixel position p; x j,k represents the position of the jth key point of the kth student in the input image; σ is the standard deviation parameter, which controls the diffusion range of the Gaussian distribution;

[0028] S5013, generating a limb-related vector field;

[0029] The second branch of the OpenPose network is based on the feature map F and outputs C limb-related vector fields. Each limb-related vector field corresponds to a human limb and represents the direction information between two connected key points:

[0030] in, is the limb-related vector field of the c-th limb at pixel position p, which is a two-dimensional vector representing the orientation information of the limb;

[0031] As the supervision signal of the network, the manually annotated key points are Mapped to the true response of the limb-related vector field, that is, for the limb c formed by key points j1 and j2, its true limb-related vector field at pixel position p for:

[0032]

[0033] Where v is the unit direction vector between j1 and j2; Ω c is the limb area connected by the key point pair j1, j2;

[0034] S5014, multi-stage joint optimization;

[0035] The OpenPose network uses a multi-stage structure to j (p) and L c The prediction results of (p) are gradually refined and optimized, and output at each step and and optimize through supervisory signals;

[0036] Loss function for keypoint confidence maps Defined as:

[0037]

[0038] Loss function for limb-related vector fields Defined as:

[0039]

[0040] Where W(p) is the availability mask at pixel position p, which is used to exclude the loss calculation of occluded or unlabeled areas;

[0041] S5015, keypoint connectivity score based on limb association vector field;

[0042] Using the predicted limb-related vector field information, any two key point candidate pairs are Score the connection relationship and calculate its connection energy E in the vector field:

[0043]

[0044] Among them, the connection energy E reflects whether there is a real physical connection between two key points. Represents the continuous points on the key point line, parameter u∈[0,1];

[0045] In practice, by sampling M points at equal intervals on the limb connection line for discrete approximation, the connection energy E is obtained as:

[0046]

[0047] Where v is and The direction vector between

[0048] S5016, human skeleton graph construction;

[0049] Based on the obtained connection energy E, a K-part graph is constructed. All key points and their connections are regarded as nodes and edges in the graph. The graph matching algorithm is used to complete the assembly of multiple human skeletons. The assembled complete skeleton is represented as a set of key points:

[0050] P={x1,x2,...,x J}

[0051] in, is the two-dimensional pixel coordinate of the j-th key point;

[0052] S5017, posture feature vectorization;

[0053] First, all keypoint coordinates are expanded into raw skeleton vectors in sequence:

[0054]

[0055] Then, centering and scale normalization are performed:

[0056] Skeleton Centralization:

[0057]

[0058] Skeleton normalization: Use shoulder width or height as the scale factor s to normalize the coordinates of all key points:

[0059]

[0060] Finally, the posture feature vector F is obtained pose ∈R 2J .

[0061] Preferably, the GaitSet network is used to perform gait recognition on the video, and the specific method is as follows:

[0062] S5021: Based on the contour image extracted from the student target detection frame, a set of gait contour images of the student in the video is obtained, which is recorded as:

[0063]

[0064] Among them, s i represents the gait profile image of the student at the i-th frame, and n is the number of valid frames of the student in the current video;

[0065] S5022, for each frame of gait profile image s i , using the ResNet-based convolutional neural network F CNN Extract frame-level gait local features f i , f i =F CNN (si );

[0066] S5023, using Set Pooling mechanism to set local feature sets of frame-level gait Extract the overall gait features and obtain the set-level gait features F set ,

[0067] S5024, horizontal pyramid mapping HPM is used to extract partitioned gait features;

[0068] F set The image is divided into L horizontal partitions along the height direction, and the gait features of the local regions are extracted to obtain the partition gait features F gait ,

[0069] Partition gait features F gait Perform L2 normalization:

[0070]

[0071] The partitioned gait feature F output after L2 normalization gait That is the gait feature of the student in the current video.

[0072] Preferably, the primary features, human posture features and gait features are first mapped to the same dimension through linear transformation, and then the primary features, human posture features and gait features are fused by weighted fusion to obtain fused features and normalize them.

[0073] Preferably, the K-means algorithm is used to perform unsupervised clustering on the fused features to unify the student identities across cameras.

[0074] Preferably, through face detection and recognition, identity matching is performed in the student information database to obtain the student ID corresponding to the cluster group, as shown below:

[0075] S701, extracting candidate student face regions within each cluster group;

[0076] S702, input each candidate face region into the FaceNet network for feature extraction to obtain the corresponding face feature vector;

[0077] S703, calculating the similarity between the facial feature vector extracted in step S702 and all facial feature vectors stored in the student information database, and selecting the student ID corresponding to the maximum similarity;

[0078] S704 , performing identity voting decision within the cluster group, and selecting the student ID with the most occurrences and the highest average similarity as the student ID corresponding to the cluster group.

[0079] Preferably, an attribute recognition algorithm is used to perform multi-label attribute classification on students detected in valid video frames and extract primary features. The specific method is as follows:

[0080] For the detected student target area, we first use a deep convolutional neural network as the backbone network to extract global feature representation and obtain a feature map.

[0081] The extracted feature map is compressed into a feature vector of fixed length after global average pooling;

[0082] The feature vector is mapped to the prediction results of each attribute category through the fully connected layer.

[0083] The present invention provides a smart campus student activity trajectory tracking system based on front-end and back-end integrated learning, which is applicable to the above-mentioned smart campus student activity trajectory tracking method based on front-end and back-end integrated learning. The system includes: an embedded front-end, a back-end server and a client;

[0084] The embedded front end includes a target detection unit and a primary feature extraction unit; the embedded front end collects videos from multiple surveillance cameras on campus, and detects student targets in the collected videos through the target detection unit, then inputs the obtained detection results into the primary feature extraction unit to extract primary features of the student targets, and uploads the extracted primary features and the video after target detection to the back end server;

[0085] The back-end server includes an advanced feature extraction unit, a cross-camera clustering unit, an identity binding unit and a data storage unit; the advanced feature extraction unit uses a human posture estimation algorithm and a gait recognition algorithm to process the uploaded video, extracts advanced features including human posture and gait, and fuses the advanced features with the primary features to obtain fused features; the cross-camera clustering unit uses the fused features to cluster all detected student targets to achieve cross-camera identity unification of the same student under different cameras; the identity binding unit performs face detection and recognition in each cluster group, matches the student information database and determines the student ID; the data storage unit is used to store the student ID, video timestamp and surveillance camera location information;

[0086] The client is used to input student IDs or photos, retrieve corresponding target information from a data storage unit, and automatically edit and generate a complete cross-camera trajectory tracking video that is continuous in time and space.

[0087] The present invention provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the method for tracking student activity trajectories in a smart campus based on front-end and back-end fusion learning.

[0088] The advantages of the present invention are:

[0089] (1) The present invention provides a method and system for tracking student activity trajectories in a smart campus based on front-end and back-end fusion learning. By integrating the deep learning model of artificial intelligence with efficient data processing methods, the system overcomes the shortcomings of traditional methods, realizes efficient and intelligent analysis of campus monitoring data, and can accurately associate and track targets in cross-camera scenarios.

[0090] (2) The present invention uses a target detection unit and a primary feature extraction unit to automatically detect students in the video and extract primary features using an attribute recognition algorithm. This automated detection and extraction process not only significantly reduces the workload and risk of misjudgment associated with manually reviewing massive amounts of video data, but also provides accurate and reliable basic data for subsequent high-level feature fusion processing and cross-camera target identity preservation.

[0091] (3) The present invention realizes multi-source feature fusion by extracting advanced features such as human posture and gait through an advanced feature extraction unit and a cross-camera clustering unit. This fusion feature greatly improves the accuracy of the model's cross-camera clustering work, so that the cross-camera identity of the same student between different cameras remains unique, and the target identity is maintained across cameras.

[0092] (4) The present invention performs face detection and recognition in each cluster group through the identity binding unit, realizing the mapping from meaningless cross-camera identities to searchable student IDs, and facilitating the client's retrieval work.

[0093] (5) The present invention supports rapid retrieval based on student ID or photos through the collaborative design of the data storage unit and the client. The client can automatically edit and generate temporally and spatially continuous tracking videos, providing an efficient and intuitive auxiliary decision-making tool for campus safety management and emergency rescue.

[0094] (6) The present invention realizes a fully automated operation process from video acquisition, preprocessing, feature extraction to identity association and trajectory generation, avoiding the omission and misjudgment problems that are prone to occur in traditional manual retrieval, significantly improving the efficiency of tracking and retrieving missing students, and ensuring the timeliness and efficiency of campus safety management.

[0095] (7) The present invention adopts the front-end and back-end fusion learning technology of the embedded front-end and back-end server to realize the efficient processing of the video data of multiple surveillance cameras on campus, so that the system can collect and process the video data of multiple surveillance cameras on campus in real time and efficiently. The front-end can collect and process data continuously without waiting for the back-end. After processing a batch of data, the back-end can continue to process the data accumulated by the front-end during this period. This two-level pipeline working mode greatly improves the system throughput and computing power utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 This is a structural diagram of a smart campus student activity trajectory tracking system based on front-end and back-end integrated learning.

[0097] Figure 2 This is a flowchart of a method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning. DETAILED DESCRIPTION

[0098] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0099] Depend on Figure 1 As shown, a smart campus student activity trajectory tracking system based on front-end and back-end integrated learning includes: an embedded front-end 1, a back-end server 2 and a client 3.

[0100] The embedded front-end 1 includes a target detection unit 11 and a primary feature extraction unit 12. The embedded front-end 1 is used to collect videos from multiple surveillance cameras on campus, and detect student targets in the collected videos through the target detection unit 11. The obtained detection results are then input into the primary feature extraction unit 12 to extract primary features of the student targets, and the extracted primary features and the video after target detection are uploaded to the back-end server 2.

[0101] The backend server 2 includes a high-level feature extraction unit 21, a cross-camera clustering unit 22, an identity binding unit 23, and a data storage unit 24. The high-level feature extraction unit 21 processes the uploaded video using a human posture estimation and gait recognition algorithm to extract high-level features such as human posture and gait, and fuses the high-level features with the primary features to obtain fused features. The cross-camera clustering unit 22 uses the fused fused features to cluster all detected student targets, achieving cross-camera identity unification for the same student under different cameras. The identity binding unit 23 performs face detection and recognition within each cluster group, matches the student information database, determines the student ID, and stores the student ID, video timestamp, and surveillance camera location information in the data storage unit 24.

[0102] The client 3 is used to input a student ID or photo, and retrieve corresponding target information from the data storage unit 24 according to the input student ID or photo, and automatically edit and generate a complete cross-camera trajectory tracking video that is continuous in time and space.

[0103] Depend on Figure 2 As shown in FIG, a method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning includes the following steps:

[0104] S1, uses the embedded front end 1 to collect video data from campus surveillance cameras.

[0105] The embedded front end 1 is connected to the campus surveillance camera terminal and transmits the collected video data to the target detection unit 11 in real time.

[0106] S2, the target detection unit 11 uses the target detection algorithm to detect students frame by frame in the video data, eliminates invalid video frames without students, and effectively compresses the video data. At the same time, it records the video timestamps and surveillance camera position information corresponding to the valid video frames with students.

[0107] S3 , the primary feature extraction unit 12 extracts primary features from the students detected by the target detection unit 11 using an attribute recognition algorithm.

[0108] Among them, primary features include but are not limited to: gender, age group, top type and color, bottom type and color, shoe type and color, whether wearing a hat, whether carrying a backpack, whether wearing glasses, whether wearing a mask, hairstyle, hair color, etc.

[0109] S4, the embedded front end 1 uploads the extracted primary features and the valid video frames obtained after target detection to the back end server 2.

[0110] S5, the high-level feature extraction unit 21 processes the uploaded video data using the human posture estimation and gait recognition algorithm, extracts high-level features including human posture and gait, and performs multi-source fusion of the high-level features with the primary features extracted by the front end to obtain fused features.

[0111] S6, the cross-camera clustering unit 22 uses the fusion features of each student to cluster all detected student targets, so that the cross-camera identity (cross-camera ID) of the same student in different cameras is unified, and the target identity is maintained across cameras.

[0112] In step S7, the identity binding unit 23 performs face detection and recognition within each cluster group and matches the identity in the student information database. For multiple identity candidates within a cluster, the one with the highest confidence score is selected as the final student ID binding result for that cluster group, achieving the mapping from cross-camera identity (cross-camera ID) to student ID.

[0113] S8, storing the student ID, video timestamp and surveillance camera location information in the data storage unit 24.

[0114] S9, the client 3 retrieves the target information from the data storage unit 24 according to the student ID or photo, and automatically edits and generates a spatiotemporal continuous tracking video of the queried target.

[0115] Among them, the photo-based query method specifically involves first performing face recognition on the input photo to extract the face feature vector, and then matching the corresponding face feature vector in the student information database to obtain the corresponding student ID for query.

[0116] In step S1, the video data from the campus surveillance camera is collected and preprocessed, which specifically includes the following steps:

[0117] S101, connecting the surveillance camera terminal to the embedded front end 1, and obtaining the original video data of the surveillance camera in the campus in real time.

[0118] S102: Process the original H264 video data frame by frame and transcode it into an MP4 format that can be directly processed.

[0119] S103 , transmitting the formatted video data frame by frame to the target detection unit 11 .

[0120] In step S2, the target detection unit 11 detects the student target and eliminates invalid video frames, specifically including the following steps:

[0121] S201, crop the video into frame images, and use the YOLO algorithm to divide each image into S×S grids.

[0122] In step S202, each grid predicts B bounding boxes, and the output of each bounding box is (x, y, w, h, C), where (x, y) represents the offset of the center of the bounding box relative to the grid, and the value range is [0, 1]; w and h are the normalized width and height of the bounding box, respectively, and C is the confidence level of the object in the bounding box.

[0123] S203: All predicted bounding boxes in each grid are filtered using the non-maximum suppression (NMS) algorithm, that is, all bounding boxes are sorted according to confidence, and the intersection over union (IoU) of any two bounding boxes is calculated. If the IoU exceeds the set threshold, the bounding boxes with higher confidence are retained, and the redundant bounding boxes with higher overlap rates are eliminated.

[0124] S204, retain all bounding boxes with confidence C higher than the preset threshold. If the detection box set is empty, it is determined that no student target is detected in the frame, which is regarded as an invalid video frame, and the frame image is directly discarded to reduce data redundancy; if it is not empty, it is used as the final result of student target detection in each frame image, and all detection box information is stored for subsequent processing.

[0125] In step S3, the primary feature extraction unit 12 extracts primary features from the student target detected in step S2, specifically including the following steps:

[0126] S301, performing attribute recognition on the student target detected in step S2.

[0127] S302, constructing a front-end primary feature vector.

[0128] S303: Store the timestamp and camera position information corresponding to the video frame.

[0129] In step S301, the attribute recognition algorithm is used to perform multi-label attribute classification on the student target detected in step S2. The specific method is as follows:

[0130] S3011, feature extraction network construction;

[0131] For the detected student target area, we first use a deep convolutional neural network (such as ResNet-50) as the backbone network to extract the global feature representation. Let the input image be I, and the feature map F∈R is extracted by the convolutional neural network. H×W×C , where H and W are the height and width of the feature map respectively, and C is the number of channels.

[0132] The convolution feature extraction formula is:

[0133] F=CNN(I)

[0134] S3012, multi-label attribute prediction;

[0135] The extracted feature map F is compressed into a fixed-length feature vector f∈R after global average pooling (GAP). C , and its calculation formula is:

[0136]

[0137] The feature vector is then mapped to the prediction results of each attribute category through a fully connected layer:

[0138]

[0139] Where W∈R K×C is the weight matrix, b∈R K is the bias vector, K is the total number of attribute categories,

[0140] σ(·) is the Sigmoid activation function, defined as:

[0141]

[0142] Because it is a multi-label classification task, each attribute is predicted independently, so Sigmoid is used as the activation function instead of Softmax.

[0143] S3013, attribute recognition training loss function;

[0144] During the training process, Binary Cross-Entropy (BCE) loss is used as the optimization objective, which is defined as follows:

[0145]

[0146] Among them, y k ∈{0,1} is the true label of the k-th attribute, is the predicted value.

[0147] By minimizing this loss function, the classification performance of all attributes can be optimized simultaneously.

[0148] The final output attribute prediction results include but are not limited to: gender, age group, top type and color, bottom type and color, shoe type and color, whether wearing a hat, whether carrying a backpack, whether wearing glasses, whether wearing a mask, hairstyle, hair color, etc.

[0149] In step S303, the timestamp and camera position information corresponding to the video frame are stored;

[0150] The embedded front-end 1 acquires the surveillance video stream frame by frame in real time, obtains the timestamp t and camera position l corresponding to each frame image, and stores them as auxiliary information for subsequent trajectory generation and query.

[0151] The output of the final step S3 is:

[0152] Front-end primary feature vector F init , obtained by attribute identification;

[0153] Timestamp t corresponding to each frame;

[0154] The camera position information l corresponding to each frame.

[0155] The above data serves as the main information transmitted from the front-end to the back-end for subsequent advanced feature extraction, feature fusion, and cross-camera target identity association.

[0156] In step S5, the high-level feature extraction unit 21 of the backend server 2 extracts high-level features and fuses multi-source features, which specifically includes the following steps:

[0157] S501, performing human posture estimation on the video data obtained in step S4;

[0158] S502, performing gait recognition on the video data acquired in step S4;

[0159] S503: Fusing the primary features obtained in step S4 with the extracted high-level features (human body posture features, gait features).

[0160] In step S501, the OpenPose deep neural network is used to estimate the human pose of the video data obtained in step S4. The specific method is as follows:

[0161] S5011, extracts features from the input image based on the VGG-19 feature extraction backbone network;

[0162] For each frame of input image obtained in step S4, it is recorded as:

[0163] I∈R W×H×3

[0164] OpenPose uses the first 10 layers of VGG-19 as the base network to extract features from the image and output feature maps:

[0165] F∈R W'×H'×C

[0166] Among them, W' and H' are the downsampled image sizes, C is the number of feature channels, and the feature map serves as the basic feature for posture detection.

[0167] S5012, generation of confidence maps of human key points;

[0168] The first branch of the OpenPose network is based on the feature map F and outputs the confidence map of J key points, that is, the probability distribution map of the key points of the human body. Each key point corresponds to a confidence map:

[0169] S j ∈R W'×H' (j=1,2,...,J)

[0170] Among them, S j (p) represents the confidence that the jth keypoint exists at pixel position p.

[0171] In order to serve as a supervisory signal for the network, the manually annotated key point position information is converted into an ideal confidence map in the form of a Gaussian distribution, which is defined as follows:

[0172]

[0173] Among them, x j,k represents the position of the jth key point of the kth student in the image, and σ is the standard deviation parameter, which controls the diffusion range of the Gaussian distribution.

[0174] The purpose of this confidence map is to mark the true position x of the key point j,k A high response (close to 1) is formed near the key point, and the response gradually decreases (close to 0) at positions far away from the key point, allowing the neural network to learn the spatial distribution characteristics of the key point instead of just locating a single pixel. This allows OpenPose to smoothly model the spatial position of the key point during training, improving the robustness and generalization ability of the network.

[0175] S5013, PartAffinityFields (PAF) generation;

[0176] The second branch of the OpenPose network is based on the feature map F and outputs C PAFs to describe the spatial and directional associations between adjacent key points. Each PAF corresponds to a human limb (e.g. shoulder-elbow, elbow-wrist), representing the directional information between two connected key points. Its representation is:

[0177] L c ∈R W'×H'×2 (c=1,2,...,C)

[0178] in, It is a two-dimensional vector at pixel position p, representing the orientation information of the limb.

[0179] In order to serve as a supervisory signal for the network, the manually annotated key points are Mapped to the true response of PAF, that is, for the limb c formed by key points j1 and j2, its true PAF at pixel position p is:

[0180]

[0181] Where v is the unit direction vector between j1 and j2;

[0182] Ω c It is the limb area connected by the key point pair j1, j2, which is specifically defined as the neighborhood range with a certain radius around the limb centerline.

[0183] Through this PAF, OpenPose can not only locate key points, but also directly predict the spatial topological structure between key points, thereby realizing skeleton segmentation and assembly in multi-person scenes.

[0184] S5014, multi-stage joint optimization;

[0185] OpenPose adopts a multi-stage structure to j and L c The prediction results are gradually refined and optimized, and output at each step and And optimized through supervisory signals.

[0186] Among them, the loss function of the key point confidence map is defined as:

[0187]

[0188] The loss function of PAF is defined as:

[0189]

[0190] Where W(p) is the availability mask of pixel position p, which is used to exclude the loss calculation of occluded or unlabeled areas.

[0191] Through gradual optimization, the network can more accurately predict key point confidence maps and limb association vector fields, significantly improving the accuracy of pose estimation.

[0192] S5015, PAF-based keypoint connectivity score;

[0193] Using the predicted PAF information, for any two key point candidate pairs Score the connection relationship and calculate its connection energy E in the vector field:

[0194]

[0195] in, Represents the continuous points on the key point line, u∈[0,1]. In practice, the integral is approximated by sampling M points at equal intervals on the limb line:

[0196]

[0197] in, is the direction vector.

[0198] The connection energy E reflects whether there is a real physical connection between two key points.

[0199] S5016, human skeleton graph construction;

[0200] Based on the connection energy obtained in S5015, a K-part graph is constructed. All key points and their connections are regarded as nodes and edges in the graph. A graph matching algorithm (such as the Hungarian algorithm or greedy matching) is used to complete the assembly of multiple human skeletons. The assembled complete skeleton is represented as a set of key points:

[0201] P={x1,x2,...,x J}

[0202] in, is the two-dimensional coordinate of the j-th key point, in pixel coordinates.

[0203] If some key points are missing in some frames, the integrity and stability of the skeleton are guaranteed by allowing the skeletons with some missing key points to be connected during the graph matching process.

[0204] S5017, posture feature vectorization;

[0205] In order to facilitate subsequent feature fusion and student identity association, the human skeleton P is further converted into a fixed-length posture feature vector F pose .

[0206] First, all keypoint coordinates are expanded into raw skeleton vectors in sequence:

[0207]

[0208] Then, scale normalization and centering are performed:

[0209] Skeleton Centralization:

[0210]

[0211] Skeleton normalization: Use shoulder width or height as the scale factor s to normalize the coordinates of all key points:

[0212]

[0213] The final Fpose ∈R 2J , which is the human posture feature vector used for subsequent multi-source feature fusion.

[0214] In step S502, the GaitSet network is used to perform gait recognition on the video data obtained in step S4. The specific method is as follows:

[0215] S5021, performing gait contour preprocessing on the input contour image set;

[0216] In this step, we first extract the contour image sequence from the student target detection frame uploaded in step S4 to obtain the student's gait contour set in the video, which is recorded as:

[0217]

[0218] Among them, s i ∈R H×W represents the student's binary outline image in the i-th frame, n is the valid frame number of the student in the current video, H and W are the height and width of the outline image respectively.

[0219] S5022, extracting local gait features using a frame-level feature extraction network;

[0220] For each frame of gait profile s i , using the ResNet-based convolutional neural network F CNN Extract its frame-level gait local features:

[0221] f i =F CNN (s i ),f i ∈R C×H'×W'

[0222] Among them, C is the number of feature channels, H and W are the height and width of the local gait feature map respectively. The gait feature f of each frame i Describes the local gait feature distribution of the frame.

[0223] S5023, using the SetPooling module to extract gait features of the frame set;

[0224] Considering that gait is a periodic movement, GaitSet innovatively uses the SetPooling mechanism to pool gait frames. Extract overall gait features instead of modeling time series. The set-level gait features are expressed as:

[0225]

[0226] Among them, SetPooling usually adopts average pooling, specifically:

[0227]

[0228] F set ∈R C×H'×W' That is, the set-level gait features of the gait frame set.

[0229] S5024, horizontal pyramid mapping (HPM) is used to extract partitioned gait features;

[0230] This step is done by set The image is divided into L horizontal partitions along the height direction, and the gait features of the local regions are extracted respectively. The gait features of each partition are mapped into a fixed-length partition gait feature vector through pooling and fully connected layers. Expressed as:

[0231]

[0232] Where d is the total dimension of gait features.

[0233] Gait characteristics F gait Then perform L2 normalization:

[0234]

[0235] Output F gait That is, the gait feature of the student in the current video clip, which is used for subsequent multi-source feature fusion and cross-camera identity association.

[0236] In step S503, the primary features and the advanced features are fused into a multi-source feature. The specific method is as follows:

[0237] S5031, feature preparation;

[0238] First, receive multi-source features extracted from the front-end and back-end, including:

[0239] (1) Primary features Extracted by the front end, mainly including static appearance features such as facial information and clothing color;

[0240] (2) Human body posture characteristics Output from step S501 describes the distribution of key points of the target student's body in the current video frame;

[0241] (3) Gait characteristics The output from step S502 describes the dynamic gait characteristics of the target student.

[0242] Among them, d1, d2 and d3 are the dimensions of three features respectively.

[0243] S5032, weighted fusion of multi-source features;

[0244] In order to make full use of the discriminative ability of different features, a weighted fusion strategy is used to uniformly model multi-source features. The fused features are expressed as:

[0245] F fused =α1·F init +α2·F pose +α3·F gait

[0246] Among them, α1, α2, α3∈R + is a learnable weight that satisfies:

[0247] α1+α2+α3=1

[0248] The fusion weights are learned through back-propagation joint training and can adaptively adjust the importance of each feature.

[0249] S5033, Feature Dimension Unification and Normalization;

[0250] Because F init 、F pose 、F gait The dimensions are different, so we first need to map each feature to the same dimension d through linear transformation:

[0251] F' init =FC init (F init ),F' pose =FC pose (F pose ),F' gait =FC gait (F gait )

[0252] Among them, FC * (·) is the fully connected transformation function.

[0253] The fusion formula is adjusted to:

[0254] F fused =α1·F' init +α2·F' pose +α3·F' gait

[0255] The obtained fusion feature F fused ∈R d As the global description vector of the student.

[0256] S5034, fusion feature normalization.

[0257] In order to improve the stability of feature matching and subsequent clustering, the fusion features are L2 normalized:

[0258]

[0259] So far, the obtained F fused This is the complete multi-source fusion feature, which is subsequently used for identity association and trajectory generation across cameras.

[0260] In step S6, cross-camera ReID clustering is performed by fusion features, which specifically includes the following steps:

[0261] S601, collect fusion features of all student detection targets;

[0262] In the above step S5, the primary feature F is completed. init , posture feature F pose , gait characteristics F gait The weighted fusion of , obtains the fusion feature vector set of each student:

[0263]

[0264] Among them, N is the number of student targets to be associated at the current moment, is the fusion feature vector of the i-th student.

[0265] S602, clustering student identities based on K-means clustering algorithm;

[0266] The K-means algorithm is used to perform unsupervised clustering on the N fusion features mentioned above to automatically complete the identity association of student targets across cameras. The objective function of K-means clustering is:

[0267]

[0268] in, For c i The cluster center of the class, c i ∈1,2,...,K represents the category label assigned to the i-th student target, and K is the preset number of cluster categories, which is dynamically set according to the actual number and distribution of students.

[0269] S603, iteratively updating cluster centers;

[0270] By continuously optimizing the objective function J, iteratively updating each cluster center μ c , until the objective function converges or the preset maximum number of iterations is reached. This process ensures that even if a student's face is not visible in some video frames, their fused features can still be associated with facial features at other times, thereby maintaining the student's unique identity and ensuring cross-camera trajectory consistency.

[0271] In step S7, student identity binding is performed through face detection and recognition, which specifically includes the following steps:

[0272] S701, extracting student face candidate regions in each cross-camera clustering group;

[0273] For each student trajectory set obtained by clustering in step S6, the corresponding video frames are scanned frame by frame, and face detection is performed in each frame to extract possible face areas.

[0274] Assume that the set of video frames contained in the i-th cluster group is:

[0275]

[0276] in, represents the j-th frame image in the i-th cluster group.

[0277] In each frame, a face detection algorithm (such as MTCNN) is used to detect a set of face candidate frames:

[0278]

[0279] in, Indicates The k-th face candidate box detected in .

[0280] S702, performing feature extraction on the extracted face area;

[0281] Each detected face candidate region Input into the FaceNet network for feature extraction to obtain the corresponding facial feature vector:

[0282]

[0283] in, is a 128-dimensional normalized feature vector.

[0284] FaceNet uses triplet loss for training. The goal is to make the facial features of the same identity closer and the facial features of different identities farther apart. Its loss function is:

[0285]

[0286] Where: f a is the anchor face feature; f p is the facial feature of the same identity; f n are facial features of different identities; α is the set interval hyperparameter;

[0287] S703, performing identity matching in the student information database;

[0288] The facial feature vector extracted in step S702 Calculate the similarity with all facial feature vectors stored in the student information database.

[0289] Assume that the facial feature template in the student information database is:

[0290]

[0291] in, Represents the facial feature vector of the mth student.

[0292] Cosine similarity is used for matching, and the similarity calculation formula is:

[0293]

[0294] For each Calculate its and all Similarity, select the student ID corresponding to the maximum similarity:

[0295]

[0296] And record the maximum similarity score:

[0297]

[0298] S704, performing identity voting decision within the cluster group;

[0299] Since each cluster group may detect multiple candidate faces in different frames, identity voting is required to finally determine the unique student ID.

[0300] Suppose the set of all detected candidate student IDs of the i-th cluster group is:

[0301]

[0302] Define the number of occurrences of each student ID in the group as:

[0303]

[0304] Where δ(·) is the Kronecker function, which takes the value 1 when the condition in the brackets is met and takes the value 0 otherwise.

[0305] The average similarity of student ID s is defined as:

[0306]

[0307] Finally, the student ID with the most occurrences and the highest average similarity is selected as the binding result of the cluster group:

[0308]

[0309] S705, output the bound student ID;

[0310] The value determined in step S704 The student identity of cluster group i is used as the student identity, and is associated with all detection targets, video timestamps and surveillance camera location information in the group and stored in the data storage unit 24 for subsequent retrieval by the client.

[0311] At this point, the student identity binding across camera cluster groups has been completed through face detection and recognition.

[0312] In step S8, the data storage unit 24 is designed, which specifically includes the following steps:

[0313] S801, establish a student tracking information database;

[0314] This embodiment uses MySQL as the backend database system, and designs a dedicated data table Student_Tracks to store students' track information.

[0315] S802, Student_Tracks data table structure design;

[0316] Field Name type illustrate StudentID VARCHAR Student ID Timestamp DATETIME Video timestamp Location VARCHAR Surveillance camera location information

[0317] S803, storage strategy;

[0318] For each student track, the system automatically writes its student ID, video timestamp, and surveillance camera location information into the Student_Tracks table for subsequent client and analysis system calls.

[0319] In step S9, the client 3 performs retrieval and video editing, which specifically includes the following steps:

[0320] S901, feature-based student information retrieval;

[0321] The client supports two retrieval methods:

[0322] Student ID search: Search directly based on StudentID.

[0323] Student photo retrieval: Input the photo of the student to be tested, extract their facial features, perform similarity matching in the student information database, obtain the corresponding student ID, and then search based on the StudentID;

[0324] S902, automatic retrieval and editing of target student tracking videos;

[0325] All retrieved video clips of the target student are sorted in ascending order by timestamp to form a continuous spatiotemporal trajectory of the student. The system automatically stitches the videos together to generate a complete tracking video. For surveillance videos capturing the same student simultaneously, the video is played back per camera, ensuring a continuous trajectory between each segment. Finally, the generated video file is encoded and compressed using OpenCV and transcoded into a video format (H264) that can be directly played on the web.

[0326] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for tracking student activity trajectories in a smart campus based on front-end and back-end fusion learning, characterized in that: Includes the following: Collect videos from campus surveillance cameras; Use the target detection algorithm to detect students in the video, remove invalid video frames without students, and record the video timestamps and surveillance camera location information corresponding to valid video frames with students; Utilize attribute recognition algorithms to perform multi-label attribute classification on students detected in valid video frames and extract primary features; Use the human posture estimation algorithm and gait recognition algorithm to process the effective video frames, extract high-level features including human posture and gait, and fuse the high-level features with the primary features to obtain fused features; Using the fusion features of each student, all detected students are clustered, so that the cross-camera identity of the same student is unified between different cameras; Perform face detection and recognition in each cluster group, and perform identity matching in the student information database to obtain the student ID corresponding to the cluster group; storing the student ID, video timestamp, and surveillance camera location information in a data storage unit; Subsequently, the target information is retrieved from the data storage unit according to the student ID or photo, and the spatiotemporal continuous tracking video of the queried target is automatically edited and generated.

2. The method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning according to claim 1 is characterized in that: The primary features include but are not limited to: gender, age group, top type and color, bottom type and color, shoe type and color, whether wearing a hat, whether carrying a backpack, whether wearing glasses, whether wearing a mask, hairstyle, and hair color.

3. The method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning according to claim 1 is characterized in that: The OpenPose deep neural network is used to estimate human poses in videos. The specific method is as follows: S5011, extract features from the input image and output a feature map F; S5012, generating a confidence map of key points of the human body; The first branch of the OpenPose network is based on the feature map F and outputs the confidence map of J key points, that is, the probability distribution map of the key points of the human body, and each key point corresponds to a confidence map; among them, S j (p) represents the confidence that the jth keypoint exists at pixel position p; As a supervisory signal for the network, the manually annotated key point position information is converted into an ideal confidence map in the form of a Gaussian distribution: in, represents the ideal confidence that the jth keypoint exists at pixel position p; x j,k represents the position of the jth key point of the kth student in the input image; σ is the standard deviation parameter, which controls the diffusion range of the Gaussian distribution; S5013, generating a limb-related vector field; The second branch of the OpenPose network is based on the feature map F and outputs C limb-related vector fields. Each limb-related vector field corresponds to a human limb and represents the direction information between two connected key points: in, is the limb-related vector field of the c-th limb at pixel position p, which is a two-dimensional vector representing the orientation information of the limb; As the supervision signal of the network, the manually annotated key points are Mapped to the true response of the limb-related vector field, that is, for the limb c formed by key points j1 and j2, its true limb-related vector field at pixel position p for: Where v is the unit direction vector between j1 and j2; Ω c is the limb area connected by the key point pair j1, j2; S5014, multi-stage joint optimization; The OpenPose network uses a multi-stage structure to j (p) and L c The prediction results of (p) are gradually refined and optimized, and output at each step and and optimize through supervisory signals; Loss function for keypoint confidence maps Defined as: Loss function for limb-related vector fields Defined as: Where W(p) is the availability mask at pixel position p, which is used to exclude the loss calculation of occluded or unlabeled areas; S5015, keypoint connectivity score based on limb association vector field; Using the predicted limb-related vector field information, any two key point candidate pairs are Score the connection relationship and calculate its connection energy E in the vector field: Among them, the connection energy E reflects whether there is a real physical connection between two key points. Represents the continuous points on the key point line, parameter u∈[0,1]; In practice, by sampling M points at equal intervals on the limb connection line for discrete approximation, the connection energy E is obtained as: Where v is and The direction vector between S5016, human skeleton graph construction; Based on the obtained connection energy E, a K-part graph is constructed. All key points and their connections are regarded as nodes and edges in the graph. The graph matching algorithm is used to complete the assembly of multiple human skeletons. The assembled complete skeleton is represented as a set of key points: P={x1,x2,...,x J } in, is the two-dimensional pixel coordinate of the j-th key point; S5017, posture feature vectorization; First, all keypoint coordinates are expanded into raw skeleton vectors in sequence: Then, centering and scale normalization are performed: Skeleton Centralization: Skeleton normalization: Use shoulder width or height as the scale factor s to normalize the coordinates of all key points: Finally, the posture feature vector F is obtained pose ∈R 2J .

4. The method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning according to claim 1 is characterized in that: Gait recognition is performed on the video using the GaitSet network. The specific method is as follows: S5021: Based on the contour image extracted from the student target detection frame, a set of gait contour images of the student in the video is obtained, which is recorded as: Among them, s i represents the gait profile image of the student at the i-th frame, and n is the number of valid frames of the student in the current video; S5022, for each frame of gait profile image s i , using the ResNet-based convolutional neural network F CNN Extract frame-level gait local features f i , f i =F CNN (s i ); S5023, using Set Pooling mechanism to set local feature sets of frame-level gait Extract the overall gait features and obtain the set-level gait features F set , S5024, horizontal pyramid mapping HPM is used to extract partitioned gait features; F set The image is divided into L horizontal partitions along the height direction, and the gait features of the local regions are extracted to obtain the partition gait features F gait , Partition gait features F gait Perform L2 normalization: The partitioned gait feature F output after L2 normalization gait That is the gait feature of the student in the current video.

5. The method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning according to claim 1 is characterized in that: First, the primary features, human posture features and gait features are mapped to the same dimension through linear transformation, and then the primary features, human posture features and gait features are fused by weighted fusion to obtain the fused features and perform normalization.

6. The method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning according to claim 1 is characterized in that: The K-means algorithm is used to perform unsupervised clustering on the fused features to unify the student identities across cameras.

7. The method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning according to claim 1 is characterized in that: Through face detection and recognition, identity matching is performed in the student information database to obtain the student ID corresponding to the cluster group, as shown below: S701, extracting candidate student face regions within each cluster group; S702, input each candidate face region into the FaceNet network for feature extraction to obtain the corresponding face feature vector; S703, calculating the similarity between the facial feature vector extracted in step S702 and all facial feature vectors stored in the student information database, and selecting the student ID corresponding to the maximum similarity; S704 , performing identity voting decision within the cluster group, and selecting the student ID with the most occurrences and the highest average similarity as the student ID corresponding to the cluster group.

8. The method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning according to claim 1 is characterized in that: The attribute recognition algorithm is used to perform multi-label attribute classification on the students detected in the valid video frames and extract primary features. The specific method is as follows: For the detected student target area, we first use a deep convolutional neural network as the backbone network to extract global feature representation and obtain a feature map. The extracted feature map is compressed into a feature vector of fixed length after global average pooling; The feature vector is mapped to the prediction results of each attribute category through the fully connected layer.

9. A smart campus student activity trajectory tracking system based on front-end and back-end integrated learning, applicable to a smart campus student activity trajectory tracking method based on front-end and back-end integrated learning as described in any one of claims 1 to 8, characterized in that: The system comprises: an embedded front end (1), a back end server (2) and a client (3); The embedded front end (1) includes a target detection unit (11) and a primary feature extraction unit (12); the embedded front end (1) collects videos from multiple surveillance cameras on campus, and detects student targets in the collected videos through the target detection unit (11), then inputs the obtained detection results into the primary feature extraction unit (12) to extract primary features of the student targets, and uploads the extracted primary features and the video after target detection to the back end server (2); The back-end server (2) includes an advanced feature extraction unit (21), a cross-camera clustering unit (22), an identity binding unit (23) and a data storage unit (24); the advanced feature extraction unit (21) processes the uploaded video using a human posture estimation algorithm and a gait recognition algorithm, extracts advanced features including human posture and gait, and fuses the advanced features with the primary features to obtain fused features; the cross-camera clustering unit (22) clusters all detected student targets using the fused features to achieve cross-camera identity unification of the same student under different cameras; the identity binding unit (23) performs face detection and recognition in each cluster group, matches the student information database and determines the student ID; the data storage unit (24) is used to store the student ID, video timestamp and surveillance camera location information; The client (3) is used to input a student ID or photo, retrieve corresponding target information from a data storage unit (24), and automatically edit and generate a complete cross-camera trajectory tracking video that is continuous in time and space.

10. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements a method for tracking student activity trajectories in a smart campus based on front-end and back-end integrated learning as described in any one of claims 1-8.

Images