A multi-lens pedestrian recognition and tracking method based on computer vision

By adopting computer vision technology in the road monitoring system, combining JDE model and Kalman filter, the problem of multi-object tracking across cameras is solved, efficient pedestrian identification and tracking is achieved, and the processing capability and accuracy of the monitoring system are improved. Suitable for public safety and commercial data management.

CN112132873BActive Publication Date: 2024-12-03TIANJIN FENGWU TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011013830.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-24
Publication Date
2024-12-03
Estimated Expiration
2040-09-24

AI Technical Summary

Technical Problem

The existing road monitoring systems face the difficulties of multi-target tracking across cameras when processing large amounts of high-resolution video data, resulting in server processing capabilities and data management challenges, and lack of effective pedestrian identification and tracking solutions.

Method used

Using a multi-lens pedestrian recognition and tracking method based on computer vision, the camera is deployed in the monitoring area, using JDE model and Kalman filter for trajectory tracking, combined with deep network and re-ID model, and using Hungarian algorithms and binary tree search to optimize feature matching , solve the pedestrian ID-SWITCH problem and realize cross-camera trajectory matching.

Benefits of technology

It improves the accuracy and efficiency of pedestrian identification and tracking, reduces manual operations, and can provide efficient and accurate pedestrian behavior analysis and early warning functions under large-scale video data, suitable for community security and business scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112132873B_ABST
    Figure CN112132873B_ABST
Patent Text Reader

Abstract

A method for multi-lens pedestrian recognition and tracking based on computer vision, comprising the following steps: Step 1: deployment of camera equipment and video stream acquisition; Step 2: tracking module under a single camera; Step 4: cross-camera tracking; Step 5: re_label module to solve row ID-SWITCH. The present invention proposes an ID error correction module for pedestrians who exchange IDs when they meet. Through analysis, it is found that the gait characteristics of pedestrians during ID exchange can be used to distinguish the trajectory, thereby achieving the purpose of ID correction. It can be used for community security, early warning of lost children and dangerous behaviors, crowd trajectory analysis in supermarkets, etc.; through computer vision methods, it can be processed uninterruptedly in the background, and the efficiency and accuracy can be significantly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video surveillance, and in particular to a multi-lens pedestrian recognition and tracking method based on computer vision. Background Art

[0002] At present, road monitoring systems have emerged in the world, and urban monitoring in various countries around the world has entered a stage of expansion and change. Under this demand change, security monitoring systems will need more integrated solutions with AI artificial intelligence. Modern public security is no longer limited to unlimited expansion of video monitoring coverage density, breadth and pursuit of ultra-high-definition resolution, but through these AI artificial intelligence means and tools, the traditional security monitoring era has taken a step further and turned to the AI ​​artificial intelligence security monitoring era that focuses on data collection, application and management. With the massive increase in the number of monitoring equipment and the continuous improvement of image resolution, the amount of data collected by public security images and pictures has shown a geometric growth. Coupled with the improvement of image resolution, the processing capacity and utilization rate of the server have also created a higher threshold. Therefore, security monitoring image monitoring faces huge challenges in image retrieval, access control data, data storage, and computing technologies.

[0003] Cross-camera multi-target tracking is a very important research topic in the field of surveillance video. The following content of this article is simply referred to as MTMC. There are some good solutions for single-camera single-target tracking and multi-target tracking, but the field of MTMC has not yet formed some solutions, and there is a lot of room for research.

[0004] To this end, this case invented a cross-camera pedestrian tracking method. The data collected by different cameras is used to generate the behavior trajectory of the person, conduct behavioral analysis and issue early warnings for special groups of people, obtain some behavioral habits through trajectory analysis, and retrieve the target person sequence to reduce manual operations. Summary of the invention

[0005] The purpose of the present invention is to provide a method for multi-lens pedestrian recognition and tracking based on computer vision.

[0006] To achieve this object, the present invention adopts the following technical solutions:

[0007] A method for multi-lens pedestrian recognition and tracking based on computer vision is provided, comprising the following steps:

[0008] Step 1: Deployment of camera equipment and video stream acquisition

[0009] Cameras are deployed at important entrances, paths, intersections and other places in the monitoring area to track and identify pedestrians. When obtaining a certain camera or a certain video, the internal switching network of the intranet is used. The protocol is fully matched and uses the RTSP access protocol to set a limit on the access connection.

[0010] Step 2: Tracking module under single camera

[0011] Given a video, the JDE model processes each frame and outputs a bounding box and a corresponding appearance embedding; calculates the association matrix between the embedding of the observation and the embedding in the pool of previously existing tracks; uses the Hungarian algorithm to assign observations to tracks; a Kalman filter is used to smooth the tracklet and predict the location of the previous tracklet in the current frame; if the assigned observation is too far away from the predicted location in space, the assignment will be rejected; the embedding of a tracker is then updated as follows: if no observation is assigned to a Tracklet, the Tracklet is marked as lost; if the loss time is greater than a given threshold, the track marked as lost will be removed from the current tracking pool; or it will be found again in the assignment step; an early attempt to jointly learn a detector and an embedding model (JDE) in a single deep network was adopted; the proposed JDE uses a single network to simultaneously output detection results and the corresponding appearance embedding of the detection box; in contrast, the SDE method and the two-stage method are characterized by resampled pixels (bounding boxes) and feature maps, respectively; both bounding boxes and feature maps are fed into a separate re-ID model to extract appearance features; the loss function of JDE adopts trihard loss, this loss takes into account the contribution of difficult samples to the final loss based on the triple loss; assuming that the two input images and , get the features through network forward propagation and , the Euclidean distance between the feature vectors of the two images is: Picture a and picture p are a pair of positive samples, picture a and picture n are a pair of negative samples, and Triplet loss is: In the formula represents max(x,0), Trihard loss with manually set threshold parameters: For each training batch, randomly select pedestrians with P IDs, and randomly select K different pictures for each pedestrian, that is, a batch contains PxK, A is the picture set with the same ID as a, and the rest of the picture sets with different IDs are B;

[0012] Step 3: re_id merges pedestrians with the same ID in a single shot

[0013] Combine global features with multi-granularity local features. Global features are responsible for extracting the common features of the whole macroscopic view. Then divide the image into different blocks, each with different granularity, to extract features at different levels or layers. Combine global and local features together, so that there is rich information and details to characterize the complete situation of the input image.

[0014] Step 4: Tracking Across Cameras

[0015] The pedestrian features of a whole trajectory are averaged and set to a as the feature of this trajectory, and then re-ranking is used to further optimize the ranking; for the cross-camera tracking problem, a binary tree search algorithm is used for traversal, and the feature matching uses the extracted re-id features, which are related to the robustness of the model;

[0016] Step 5: re_label module solves row ID-SWITCH

[0017] By connecting pedestrians at different locations monitored by multiple surveillance cameras, when visual appearance features are unreliable, such as wearing masks and makeup, analysis based on behavioral features (such as gait) becomes an optional solution to the problem of human re-identification; by analyzing the single-lens tracking results, after merging the same ID through the reid feature, there are still some ID-SWITCH situations that come from the exchange of IDs between two people when they meet, and these IDs need to be resolved by the correction module; this module includes two sub-modules: target object gait feature extraction and object re-identification. The target object gait feature extraction module detects objects through the foreground and extracts the target object silhouette, gait cycle and walking angle, and finally fuses the gait features into the object re-identification module; the object re-identification module searches for the top three re-identification results that best match the target object in the candidate object receipt set.

[0018] Furthermore, the cameras deployed in step 1 include a plurality of cameras installed in a horizontal arrangement, and the plurality of cameras can rotate at a wide angle without interfering with each other.

[0019] Furthermore, in step 3, the image is divided into different blocks, each with a different granularity, and each block is assigned a different value n (n=1, 2, 3...n). Different values ​​n correspond to different levels of features. As the value of n increases, the detail of the model's detailed information increases.

[0020] Furthermore, in addition to the three overlapping camera pairs in step 4, some merging of single-camera tracks is also performed during cross-camera tracking.

[0021] Furthermore, the implementation method of gait feature extraction in step 5 is as follows:

[0022] 1) Image preprocessing, pedestrian ROI and scale change;

[0023] 2) Extract the GEI gait energy map of the image;

[0024] 3) Extract HOG features from the above GEI;

[0025] 4) Construct training set and test set to train SVM classifier;

[0026] 5) Use the trained model to label pedestrians.

[0027] Furthermore, in step 5, the ID correction implementation method predicts the target spatial position of each frame through Kalman filtering. When the target is blocked, it will bring a large prediction error. A reasonable area search method is used to correct some IDs. Assuming that the center of the search area is , with radius r, Any detection in the current frame The center position of the rectangular area, if the conditions are met: The number of frames in which the target is in an unrelated state; then the similarity is calculated according to the gait characteristics of pedestrians in the area, and the next re-matching is performed. The beneficial effects of the present invention: it can be used for community security, such as tracking the elderly can be used for warning of falls, tracking children, warning of children's loss and dangerous behavior; it can also be used for crowd trajectory analysis in supermarkets, thereby helping supermarkets to reasonably analyze data to achieve intelligent shelf management.

[0028] When the amount of video data is large, we can abandon manual retrieval to ensure high efficiency and accuracy. Through computer vision methods, we can process it uninterruptedly in the background, which can significantly improve efficiency and accuracy.

[0029] The re_label module is used to solve the ID-SWITCH problem. When two people exchange IDs at an intersection, a series of features such as the object's gait features can be used for re-identification, and the previous tracking can be continued to avoid interference and errors, with higher overall accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments of the present invention are briefly introduced below.

[0031] Figure 1 It is the overall framework process of the present invention;

[0032] Figure 2 The re_label module provided by the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0034] Among them, the drawings are only used for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product.

[0035] Reference Figure 1 and Figure 2 A method for multi-lens pedestrian recognition and tracking based on computer vision is shown, comprising the following steps:

[0036] Step 1: Deployment of camera equipment and video stream acquisition

[0037] Cameras are deployed at important entrances, paths, intersections and other places in the monitoring area to track and identify pedestrians. When obtaining a certain camera or a certain video, the internal switching network of the intranet is used. The protocol is fully matched and uses the RTSP access protocol to set a limit on the access connection.

[0038] Step 2: Tracking module under single camera

[0039] Given a video, the JDE model processes each frame and outputs a bounding box and a corresponding appearance embedding; calculates the association matrix between the embedding of the observation and the embedding in the pool of previously existing tracks; uses the Hungarian algorithm to assign observations to tracks; a Kalman filter is used to smooth the tracklet and predict the location of the previous tracklet in the current frame; if the assigned observation is too far away from the predicted location in space, the assignment will be rejected; the embedding of a tracker is then updated as follows: if no observation is assigned to a Tracklet, the Tracklet is marked as lost; if the loss time is greater than a given threshold, the track marked as lost will be removed from the current tracking pool; or it will be found again in the assignment step; an early attempt to jointly learn a detector and an embedding model (JDE) in a single deep network was adopted; the proposed JDE uses a single network to simultaneously output detection results and the corresponding appearance embedding of the detection box; in contrast, the SDE method and the two-stage method are characterized by resampled pixels (bounding boxes) and feature maps, respectively; both bounding boxes and feature maps are fed into a separate re-ID model to extract appearance features; the loss function of JDE adopts trihard loss, this loss takes into account the contribution of difficult samples to the final loss based on the triple loss; assuming that the two input images and , get the features through network forward propagation and , the Euclidean distance between the feature vectors of the two images is: Picture a and picture p are a pair of positive samples, picture a and picture n are a pair of negative samples, and Triplet loss is: In the formula represents max(x,0), Trihard loss with manually set threshold parameters: For each training batch, randomly select pedestrians with P IDs, and randomly select K different pictures for each pedestrian, that is, a batch contains PxK, A is the picture set with the same ID as a, and the rest of the picture sets with different IDs are B;

[0040] Step 3: re_id merges pedestrians with the same ID in a single shot

[0041] Combine global features with multi-granularity local features. Global features are responsible for extracting the common features of the whole macroscopic view. Then divide the image into different blocks, each with different granularity, to extract features at different levels or layers. Combine global and local features together, so that there is rich information and details to characterize the complete situation of the input image.

[0042] Step 4: Tracking Across Cameras

[0043] The pedestrian features of a whole trajectory are averaged and set to a as the feature of this trajectory, and then re-ranking is used to further optimize the ranking; for the cross-camera tracking problem, a binary tree search algorithm is used for traversal, and the feature matching uses the extracted re-id features, which are related to the robustness of the model;

[0044] Step 5: re_label module solves row ID-SWITCH

[0045] By connecting pedestrians at different locations monitored by multiple surveillance cameras, when visual appearance features are unreliable, such as wearing masks and makeup, analysis based on behavioral features (such as gait) becomes an optional solution to the problem of human re-identification; by analyzing the single-lens tracking results, after merging the same ID through the reid feature, there are still some ID-SWITCH situations that come from the exchange of IDs between two people when they meet, and these IDs need to be resolved by the correction module; this module includes two sub-modules: target object gait feature extraction and object re-identification. The target object gait feature extraction module detects objects through the foreground and extracts the target object silhouette, gait cycle and walking angle, and finally fuses the gait features into the object re-identification module; the object re-identification module searches for the top three re-identification results that best match the target object in the candidate object receipt set.

[0046] The cameras deployed in step 1 include several cameras installed in a horizontal arrangement, and the cameras can rotate at a wide angle without interfering with each other.

[0047] Among them, in step 3, the image is divided into different blocks, each with a different granularity, and each block is assigned a different value n (n=1, 2, 3...n). Different values ​​n correspond to different levels of features. As the value of n increases, the detail of the model detail information increases.

[0048] In addition to the three overlapping camera pairs in step 4, some single-camera track merging work is also done during cross-camera tracking.

[0049] The implementation method of gait feature extraction in step 5 is as follows:

[0050] 1) Image preprocessing, pedestrian ROI and scale change;

[0051] 2) Extract the GEI gait energy map of the image;

[0052] 3) Extract HOG features from the above GEI;

[0053] 4) Construct training set and test set to train SVM classifier;

[0054] 5) Use the trained model to label pedestrians.

[0055] Among them, the ID correction implementation method in step 5 predicts the target spatial position of each frame through Kalman filtering. When the target is blocked, it will cause prediction errors; a reasonable area search method is used to correct some IDs. Assume that the center of the search area is , with radius r, Any detection in the current frame The center position of the rectangular area, if the conditions are met: is the number of frames in which the target is in an unassociated state; then the similarity is calculated based on the gait characteristics of the pedestrians in the area for subsequent re-matching.

[0056] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A multi-lens pedestrian recognition and tracking method based on computer vision, It is characterized in that The following steps are involved: Step 1: Deployment of camera equipment and video stream acquisition Cameras are deployed at important entrances, paths, and intersections of the monitoring area to track and identify pedestrians. When obtaining a camera or a video, the internal switching network of the intranet is used. The protocol is fully matched, using the RTSP access protocol, and a restriction is set on the access connection. Step 2: Tracking module under single camera Given a video, the JDE model processes each frame and outputs bounding boxes and corresponding appearance embeddings; The association matrix between the embedding of the observation and the embeddings in the pool of previously existing tracks is calculated; the observations are assigned to tracks using the Hungarian algorithm; the Kalman filter is used to smooth the tracks and predict the position of the previous track in the current frame; if the assigned observation is too far away from the predicted position in space, the assignment is rejected; the embedding of a tracker is then updated as follows, if no observation is assigned to a tracklet, the tracklet is marked as lost; If the lost time is greater than a given threshold, the track marked as lost will be removed from the current tracking pool; or it will be found again in the assignment step; an early attempt to jointly learn the detector and embedding model in a single deep network was adopted; the proposed JDE uses a single network to simultaneously output the detection result and the corresponding appearance embedding of the detection box; The loss function of JDE adopts trihardloss, which takes into account the contribution of hard examples to the final loss based on the triple loss. Assuming two input images and , get the features through network forward propagation and , the Euclidean distance between the feature vectors of the two images is: Picture a and picture p are a pair of positive samples, picture a and picture n are a pair of negative samples, and Tripletloss is: In the formula Indicates max(x,0), which is a manually set threshold parameter Trihard loss: For each training batch, randomly select pedestrians with P IDs, and randomly select K different pictures for each pedestrian, that is, a batch contains PxK, A is the picture set with the same ID as a, and the rest of the picture sets with different IDs are B; Step 3: re_id merges pedestrians with the same ID in a single shot Combine global features with multi-granularity local features. Global features are responsible for extracting the common features of the whole macroscopic view. Then divide the image into different blocks, each with different granularity, to extract features at different levels or layers. Combine global and local features together, so that there is rich information and details to characterize the complete situation of the input image. Step 4: Tracking Across Cameras The pedestrian features of a whole trajectory are averaged and set to a as the feature of this trajectory, and then re-ranking is used to further optimize the ranking; for the cross-camera tracking problem, a binary tree search algorithm is used for traversal, and the feature matching uses the extracted re-id features, which are related to the robustness of the model; Step 5: re_label module solves row ID-SWITCH By linking pedestrians located in different locations monitored by multiple surveillance cameras, when visual appearance features are unreliable, analysis based on behavioral features becomes an optional solution to the problem of human re-identification; by analyzing the single-lens tracking results, after merging the same ID through the re-id feature, there are still some ID-SWITCH situations that come from the ID exchange between two people when they meet, and these IDs need to be solved by the correction module; this module includes two sub-modules: target object gait feature extraction and object re-identification. The target object gait feature extraction module detects the object through the foreground and extracts the target object silhouette, gait cycle and walking angle, and finally fuses the gait features into the object re-identification module; the object re-identification module searches for the top three re-identification results that best match the target object in the candidate object receipt set.

2. According to the method of multi-lens pedestrian recognition and tracking based on computer vision in claim 1, It is characterized in that The cameras deployed in step 1 include several cameras arranged horizontally, and the cameras can rotate at a wide angle without interfering with each other.

3. The method for multi-lens pedestrian recognition and tracking based on computer vision according to claim 1, It is characterized in that In step 3, the image is divided into different blocks, each with a different granularity, and each block is assigned a different value n. Different values ​​n correspond to different levels of features. As the value of n increases, the detail of the model's detailed information increases.

4. The method for multi-lens pedestrian recognition and tracking based on computer vision according to claim 1, It is characterized in that In addition to the three overlapping camera pairs in step 4, some merging of single-camera tracks is also done during cross-camera tracking.

5. The method for multi-lens pedestrian recognition and tracking based on computer vision according to claim 1, It is characterized in that The implementation method of gait feature extraction in step 5 is as follows: 1) Image preprocessing, pedestrian ROI and scale change; 2) Extract the GEI gait energy map of the image; 3) Extract HOG features from the above GEI; 4) Construct training set and test set to train SVM classifier; 5) Use the trained model to label pedestrians.

6. The method for multi-lens pedestrian recognition and tracking based on computer vision according to claim 1, It is characterized in that In step 5, the ID correction implementation method predicts the target spatial position of each frame through Kalman filtering. When the target is blocked, it will cause prediction errors. A reasonable area search method is used to correct some IDs. Assume that the center of the search area is , with radius r, Any detection in the current frame The center position of the shape area, if the conditions are met: is the number of frames in which the target is in an unassociated state; then the similarity is calculated based on the gait characteristics of the pedestrians in the area for subsequent re-matching.