A pedestrian tracking and re-identification method based on deep learning

Abstract

A kind of pedestrian tracking and re-identification method based on deep learning, comprising the following steps;Step 1: pedestrian target detection is carried out on video image frame by frame;Step 2: the features of the pedestrians detected in step 1 are extracted using the DeepSort model, generating a.npy file;Step 3: perform pedestrian re-identification detection using Fastreid, extract features according to the preset pedestrian picture database, generate.npy file;Step 4: the feature extraction result of each pedestrian target is calculated with the feature extraction result of the specific pedestrian database, if it is greater than the threshold γ, it is judged as a specific pedestrian target that needs to be re-identified, and the tracking of the pedestrian is performed, otherwise the target tracking is not performed;The present application can accurately locate the specific pedestrian across time, across region, across camera, and can infer and detect through real-time video, and through a series of improvements to achieve the optimal effect, finally complete the project landing, which can be widely used in intelligent monitoring, intelligent security and other systems.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent monitoring and security, and particularly relates to a pedestrian tracking and re-identification method based on deep learning. BACKGROUND

[0002] With the development of science and technology, monitoring videos have been widely used in commercial, security, search and other fields, and play a very important role in people's daily life. Since the rise of face recognition technology, pedestrian re-identification technology has developed into a major direction of computer vision. Although the face recognition technology is relatively mature, in cases such as high-density crowds, low resolution of capture cameras or camera angle bias, the face recognition technology often cannot achieve the desired effect, but the person re-identification technology can continue to play an important role, timely positioning and identifying specific pedestrians in the monitoring video, which has great significance for the investigation of criminal cases, search and rescue of missing persons, etc.

[0003] Up to now, enterprises in the field of artificial intelligence at home and abroad have conducted in-depth research on pedestrian re-identification technology, and there are still the following research difficulties and problems in pedestrian re-identification:

[0004] (1) Real-world pedestrians may be obstructed by obstacles, change from day to night, change clothes, and other complex and variable situations, making it difficult to achieve the accuracy of experimental algorithms.

[0005] (2) Cross-region identification has security and privacy problems, and it is difficult to obtain a data set, that is, how to obtain a model algorithm with high robustness under the condition of uneven samples is extremely challenging.

[0006] (3) When tracking across cameras, the light and shade, obstacles, and camera resolution will change with the change of the camera, and how to identify the same target without being limited by the tracking range is a problem that needs to be solved in pedestrian re-identification technology. SUMMARY

[0007] In order to overcome the above technical problems, the present application provides a pedestrian tracking and re-identification method based on deep learning, which combines an improved YOLOv5-Lite target detection algorithm and an improved Deepsort target tracking algorithm, can accurately locate specific pedestrians across time, region and camera, and can infer and detect through real-time video, and through a series of improvement measures to make the system model achieve the best effect. This method can be widely used in intelligent monitoring, intelligent security and other system applications.

[0008] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0009] A pedestrian tracking and re-identification method based on deep learning, comprising the following steps:

[0010] Step 1: Improved YOLOv5-Lite model is used to detect pedestrian targets frame by frame for video images;

[0011] Step 2: DeepSort model is used to extract features of the pedestrians detected in each frame in step 1, and.npy files are generated;

[0012] Step 3: Fastreid is used for pedestrian re-identification detection, and features are extracted according to the preset pedestrian picture database to generate.npy files;

[0013] Step 4: The feature extraction results of each pedestrian target in step 2 and the feature extraction results of the specific pedestrian database in step 3 are calculated by cosine similarity, as shown in formula (1), x1 and x2 are two non-zero vectors, if greater than threshold γ, it is judged as a specific pedestrian target that needs to be re-identified, and the tracking strategy of the improved Deepsort model is used to track the pedestrian, otherwise the target tracking is not performed.

[0014]

[0015] Further, the step 1 comprises the following sub-steps:

[0016] Step 1.1: The pictures of the data set are input into the improved YOLOv5-Lite model network structure, and the BiFPN module is added on the basis of YOLOv5-Lite, which realizes the combination of cross-scale bidirectional connection and fast normalization, inputs different feature weights, and lets the network learn by itself, which adopts the Softmax-based fusion method to normalize the weights to between 0 and 1 as formula (2):

[0017]

[0018] Among them, w i and w j are learnable weights;

[0019] Step 1.2: The convolutional neural network is used to extract features of the pictures, and then the feature maps are output, at the same time the pictures are divided into small squares and anchor frames are generated, the labeled prediction frame and the feature map are associated, and finally the loss function is established and the end-to-end training is started, wherein the loss function is shown in formula (3).

[0020]

[0021] Among them Euclidean distance between the center points of the predicted and ground truth boxes, c represents the diagonal distance of the minimum closed region that can contain both the predicted and ground truth boxes, v is used to measure the consistency of the relative proportion of the two rectangular boxes, and a is a weight coefficient:

[0022]

[0023] Further, the step 3 includes the following sub-steps:

[0024] Step 3.1: After the pre-processing of the input picture, a pre-trained model ResNet50 is called as the Backbone, and then the output feature map is aggregated through Gem Pooling, so that one target is represented by a feature vector, and then the feature vector obtained in the foregoing is changed through the Bnneck module, and finally the Triplet loss is defined to learn the similarity within the classification and the discrimination within the class, so that the direct discrimination between different feature vectors is more obvious, and the same feature vectors are more convergent.

[0025] The Triplet loss input is a triplet, including an anchor (Anchor) example, a positive (Positive) example and a negative (Negative) example, and the similarity calculation between samples is realized by optimizing the distance between the anchor example and the positive example to be less than the distance between the anchor example and the negative example, a: anchor, anchor example; p: positive, sample of the same class as a; n: negative, sample of different classes from a; margin is a constant greater than 0:

[0026] L = max (d (a, p) - d (a, n) + margin, 0) (5)

[0027] Further, the tracking strategy in the step 4 includes the following sub-steps:

[0028] Step 4.1: The selected specific pedestrian target of re-identification is tracked by using the NSA Kalman filter algorithm, and the appearance state of the i-th track at frame t is updated in the form of exponential moving average

[0029]

[0030] where f i t is the current matching detection appearance embedding, and a = 0.9 is the momentum term.

[0031] Meanwhile, adaptive noise is added to enhance the robustness of tracking, wherein the covariance of the adaptive noise is As shown in formula (7):

[0032]

[0033] where R k is a pre-set constant measurement noise covariance, c k is the detection confidence score at state k, and instead of using only appearance feature distance in the matching process, both appearance and motion information are considered simultaneously;

[0034] A global linear assignment is used instead of the matching cascade, where the assignment matrix C is a weighted sum of the appearance cost A a and the motion cost A m :

[0035] C = λA a + (1 - λ)A m (8)

[0036] where the weight factor λ is set to 0.98;

[0037] Step 4.2: After the track is predicted by the Kalman filter algorithm, a track is predicted for the current frame, if the prediction is confirmed (pedestrian or vehicle), the detection (detection) is performed on the current frame, then the detection frame and the confirmed track frame are matched in cascade, and the detection frame of the track is updated after the matching is completed;

[0038] If the track matching fails, proceed to IoU matching, if it can be matched successfully, then update, and then repeat the prediction-observation-update tracking process. IoU matching failure is divided into observation matching failure and track matching failure: for observation matching failure, a new track is established at this time, and three observations are performed thereafter, if it is still the actual target (pedestrian or vehicle), it is modified to be confirmed; for track matching failure, it is judged whether it is confirmed as a pedestrian or a vehicle, if it is in an unconfirmed state, it is deleted, otherwise, a threshold is set for it, if it is greater than the threshold max_age, it is deleted, it is considered to have moved out of the observation range, if it is less than the threshold, it is also observed three times, returning to the initial stage.

[0039] Advantages of the present application.

[0040] The application realizes real-time tracking and re-identification of specific pedestrian targets. Compared with the algorithm before improvement, the YOLOv5-Lite detection module model has a 3% improvement in recognition accuracy while maintaining the average accuracy, and the detection accuracy can reach 92%; the Deepsort tracking module model has different degrees of improvement in various indicators for evaluating tracking performance, and obtains better tracking effect; the feature extraction logic of the Fastreid re-identification module is optimized, and the algorithm speed has a leap-forward improvement; the overall model proposed in the application can achieve high accuracy in real-time detection, thus meeting the needs of actual video monitoring and having broad application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 Overall process of the pedestrian re-identification system

[0042] Figure 2 Improved YOLOv5-Lite model network structure.

[0043] Figure 3 Tracking algorithm Deepsort improvement strategy diagram.

[0044] Figure 4 The pedestrian pictures to be detected are named bag and red from left to right.

[0045] Figure 5 Pedestrian bag re-identification effect in area 1.

[0046] Figure 6 Pedestrian red re-identification effect in area 1.

[0047] Figure 7 Pedestrian bag re-identification effect in area 2.

[0048] Figure 8 Pedestrian red re-identification effect in area 2. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings and tables in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0050] A pedestrian tracking and re-identification method based on deep learning comprises the following steps:

[0051] Step 1: using an improved YOLOv5-Lite model to detect pedestrian targets frame by frame for video images;

[0052] Step 2: Feature extraction of the pedestrians detected in each frame in step 1 using the DeepSort model, generating a.npy file;

[0053] Step 3: Perform pedestrian re-identification detection using Fastreid, and perform feature extraction according to the preset pedestrian picture database, generating a.npy file;

[0054] Step 4: Calculate the cosine similarity between the feature extraction results of each pedestrian target in step 2 and the feature extraction results of the specific pedestrian database in step 3, as shown in formula (1). If it is greater than the threshold value γ, it is judged as a specific pedestrian target that needs to be re-identified, and the tracking strategy of the improved Deepsort model is used to track the pedestrian, otherwise the target is not tracked

[0055]

[0056] The step 1 includes the following sub-steps:

[0057] Step 1.1: Input the picture of the data set into the improved YOLOv5-Lite model network structure, as shown in the following figure: Figure 2 The present application adds a BiFPN module (a weighted bidirectional feature pyramid network) to the original YOLOv5-Lite, effectively enhancing feature extraction. BiFPN realizes the combination of cross-scale bidirectional connection and fast normalization, inputs different feature weights, and lets the network learn by itself. It uses a Softmax-based fusion method to normalize the weight to between 0 and 1 as formula (2):

[0058]

[0059] where w i and w j are learnable weights.

[0060] Step 1.2: Feature extraction of the picture using the convolutional neural network, then output the feature map, at the same time the picture is divided into small squares and anchor boxes are generated, the labeled prediction box and the feature map are associated, and finally the loss function is established and the end-to-end training is started, wherein the loss function is shown in formula (3).

[0061]

[0062] where represents the Euclidean distance between the center points of the prediction box and the real box. c represents the diagonal distance of the smallest closed region that can contain the prediction box and the real box. v is used to measure the consistency of the relative proportion of the two rectangular boxes, and α is the weight coefficient:

[0063]

[0064] The step 3 comprises the following sub-steps:

[0065] Step 3.1: after pre-processing the input picture, a pre-trained model ResNet50 is called as Backbone, then through Gem Pooling, the output feature map is represented by an aggregated feature vector, and then the Bnneck module is used to change the feature vector obtained before, finally, the Triplet loss is defined to learn the similarity within the classification and the distinction within the class, so that the distinction between different feature vectors is more obvious, and the same feature vectors are more convergent.

[0066] The Triplet loss input is a triplet, including an anchor (Anchor) example, a positive (Positive) example and a negative (Negative) example, the similarity between samples is calculated by optimizing the distance between the anchor example and the positive example to be less than the distance between the anchor example and the negative example, a: anchor, anchor example; p: positive, sample of the same class as a; n: negative, sample of different classes from a; margin is a constant greater than 0:

[0067] L = max (d (a, p) - d (a, n) + margin, 0) (5)

[0068] The tracking strategy in DeepSort is improved, and the tracking strategy in the above step 4 comprises the following sub-steps:

[0069] Step 4.1: the selected specific pedestrian target of re-identification is tracked by using the NSA Kalman filter algorithm. Specifically, the appearance state of the i-th track at frame t is updated in the form of exponential moving average

[0070]

[0071] Where f i t is the current matching detection appearance embedding, and alpha=0.9 is the momentum term.

[0072] In the algorithm, adaptive noise is also added to enhance the robustness of tracking. The covariance of the adaptive noise is As shown in formula (7):

[0073]

[0074] Where R k is a pre-set constant noise covariance, ck is the detection confidence score in state k, and in the matching process not only the appearance feature distance is used, but also the appearance and motion information are considered.

[0075] In order to solve the problem that the additional prior constraints limit the matching accuracy, a common global linear assignment is used instead of the matching cascade. In which, the assignment matrix C is the weighted sum of the appearance cost A a and the motion cost A m :

[0076] C = λA a + (1 - λ)A m (8)

[0077] In which the weight factor λ is set to 0.98.

[0078] Step 4.2: After the track trajectory is predicted by the Kalman filtering algorithm, a trajectory is predicted for the current frame, if the prediction is confirmed (pedestrian or car), the detection (detection) is carried out for the current frame, then the detection frame and the confirmed track frame are matched, and after the matching is completed, the detection frame of the track is updated.

[0079] If the track matching fails, proceed to IoU matching, if it can be matched successfully, then update, and then repeat the prediction-observation-update tracking process. IoU matching failure is divided into observation matching failure and track matching failure: for observation matching failure, at this time a new track is established for it, and then it is investigated three times, if it is still the actual target (pedestrian or car) it is modified as confirmed; for track matching failure, at this time it is judged whether it is confirmed as a pedestrian or a car, if it is not confirmed, it is deleted, otherwise, a threshold is set for it, if it is greater than the threshold max_age, it is deleted, it is considered to have moved out of the observation range, if it is less than the threshold, it is also investigated three times, back to the beginning stage.

[0080] Embodiment:

[0081] As Figure 1As shown: first intercept the target pedestrian picture, then through the Fastreid feature extraction model, the intercepted pedestrian database is extracted, and the corresponding.npy file is generated, the video to be detected is read in, and YOLOv5-Lite target detection algorithm is used to detect all pedestrians in the current video frame, then Deepsort algorithm is used to extract the features of the detected pedestrians, and.npy file is generated, at this time, the cosine similarity of the two.npy files generated above is calculated, if the similarity is greater than the threshold value γ, it is judged as the target pedestrian, at this time, the target pedestrian is tracked through the Deepsort algorithm, and the similarity less than the threshold value γ is the non-target pedestrian, finally the whole process is displayed through simple visualization.

[0082] As shown in Figure 2 The original network head Concat module is replaced by BiFPN_Concat module.

[0083] As shown in Figure 3 The ordinary Kalman filter is replaced by NSA Kalman filter algorithm, and ordinary global linear distribution is used instead of matching cascade, and the appearance of the track is updated in the form of exponential moving average (EMA).

[0084] As shown in Figure 4 The picture of the pedestrian to be searched is intercepted in advance.

[0085] As shown in Figures 5-6 The search result of the pedestrian to be searched in area 1 is shown.

[0086] As shown in Figures 7-8 The search result of the pedestrian to be searched in area 2 is shown.

[0087] Table 1 Comparison of various indexes before and after improvement of YOLOv5-Lite algorithm

[0088]

[0089] Table 2 Comparison of indexes before and after improvement of Deepsort algorithm

[0090]

[0091] Table 3 Comparison of indexes before and after improvement of Deepsort algorithm (continued)

[0092]

[0093] As shown in Table 1, under the premise that the picture input size is 640*640, the overall size of the improved model has a small increase, the mAP_0.5, mAP_0.5:0.95 before and after improvement is basically flat, the recall and frame rate FPS decrease slightly, the accuracy of the model increases from 0.89 to 0.92, which increases by 3%, indicating that the accuracy of the improved model has a certain degree of increase, and compared with other algorithms, the model size and accuracy are improved, and the performance on the test set is better.

[0094] As shown in Tables 2-3, the IDR index increases from 21.7 to 24.9, the IDP index increases from 71.8 to 74.7, and the IDF1 index increases from 33.3 to 37.4, indicating that the recall value and detection score of correct identification have been significantly improved; Rcll increases from 27.2 to 31.3, and Prcn increases from 89.9 to 94.0, indicating that the improved Deepsort algorithm has a significant improvement in accuracy; FAR decreases from 0.63 to 0.42, that is, the number of false recognitions per frame decreases; MT increases from 25 to 30, and ML decreases from 339 to 307, indicating that the number of GT tracks with more than 80% of the total number of successfully tracked frames increases, and the number of GT tracks with less than 20% of the total number of successfully tracked frames decreases; FP is the number of false positives, which decreases from 3352 to 2214, and FN is the number of false negatives, which decreases from 80411 to 75817; IDs increases from 218 to 239, so the frequency of ID-switch increases after the model is changed; FM increases from 1121 to 1190, indicating that the algorithm has improved the ability to continue tracking after the target is blocked; MOTA increases from 23.9 to 29.1, and MOTP increases from 78.4 to 78.5, indicating that the detection quality and tracking accuracy have improved to a certain extent. Through analysis and comparison, it can be seen that after the algorithm is improved, the tracking performance and accuracy of Deepsort have been greatly improved, and the performance on the same data set is better after improvement.

[0095] The innovation of the application lies in:

[0096] Improvement I, for the branch model v5Lite-g of YOLOv5-Lite, modify the network head, replace all Concat with BiFPN_Concat.

[0097] Improvement II, replace the ordinary Kalman filter with the NSA Kalman filter algorithm, and introduce an adaptive noise covariance

[0098]

[0099] where R kis a pre-set constant measurement noise covariance, c k is the detection confidence score at state k, and instead of using only appearance feature distance in the matching process, both appearance and motion information are considered.

[0100] The cost matrix C is a weighted sum of the appearance cost A a and the action cost A m :

[0101] C = λA a + (1 - λ)A m (2)

[0102] where the weight factor λ is set to 0.98, and in addition, to solve the problem that additional prior constraints will limit the matching accuracy, a normal global linear assignment is used instead of the matching cascade.

[0103] The appearance state of the i-th track at frame t is updated in an exponential moving average manner

[0104]

[0105] where f i t is the appearance embedding of the current matching detection, and α = 0.9 is the momentum term.

[0106] Improvement three, convert the Fastreid model file with.pth suffix to the model file with.onnx suffix.

[0107] Improvement four, change the pedestrian detection to frame separation detection, that is, every other frame uses the YOLOv5-Lite detection model to detect all pedestrians in the video, and at the same time, add a frame rate display module to the real-time video visualization interface.

Claims

1. A pedestrian tracking and re-identification method based on deep learning, characterized in that, It comprises the following steps: Step 1: using the improved YOLOv5-Lite model to detect pedestrian targets frame by frame for video images; Step 2: using the DeepSort model to extract features of the pedestrians detected in each frame in step 1, and generating a.npy file; Step 3: using Fastreid to detect pedestrian re-identification, extracting features according to the preset pedestrian picture database, and generating a.npy file; Step 4: calculating the cosine similarity between the feature extraction results of each pedestrian target in step 2 and the feature extraction results of the specific pedestrian database in step 3, as shown in formula (1), x1 and x2 are two non-zero vectors, if greater than threshold γ, it is judged as a specific pedestrian target that needs to be re-identified, and the tracking strategy of the improved Deepsort model is used to track the pedestrian, otherwise the target is not tracked The step 1 comprises the following sub-steps: Step 1.1: input the picture of the data set into the improved YOLOv5-Lite model network structure, which adds a BiFPN module based on YOLOv5-Lite, BiFPN realizes the combination of cross-scale bidirectional connection and fast normalization, inputs different feature weights, and lets the network learn by itself, which adopts Softmax-based fusion to normalize the weight to 0 and 1 as formula (2): where w i and w j are learnable weights; Step 1.2: using the convolutional neural network to extract features of the picture, then outputting the feature map, at the same time the picture is divided into small squares and anchor boxes are generated, the labeled prediction box and the feature map are associated, and finally the loss function is established and the end-to-end training is started, wherein the loss function is shown as formula (3); where ρ 2 (b, b gt ) represents the Euclidean distance between the center points of the predicted and true boxes, c represents the diagonal distance of the minimum closed region that can contain both the predicted and true boxes; v is used to measure the consistency of the relative proportion of the two rectangular boxes, and a is a weight coefficient:

2. The pedestrian tracking and re-identification method based on deep learning according to claim 1, characterized in that, The step 3 comprises the following sub-steps: Step 3.1: after the input picture is preprocessed, the pre-trained model ResNet50 is called as Backbone, then the output feature map is aggregated through Gem Pooling, a feature vector is used to represent a target, and then the Bnneck module is used to change the feature vector obtained before, finally the Triplet loss is defined to learn the similarity within the classification and the discrimination within the class, so that the discrimination between different feature vectors is more obvious, and the same feature vectors are more convergent; The Triplet loss input is a three-tuple, including anchor example, positive example and negative example, by optimizing the distance between anchor example and positive example to be less than the distance between anchor example and negative example, the similarity calculation between samples is realized, a: anchor, anchor example; P: positive, the same class sample as a; n: negative, different class sample from a; margin is a constant greater than 0: L = max (d (a, p) - d (a, n) + margin, 0) (5). 3.The method of claim 1, wherein, The tracking strategy in step 4 comprises the following sub-steps: Step 4.1: Track the selected re-identified specific pedestrian target using the NSA Kalman filter algorithm, specifically updating the appearance state of the ith trajectory at frame t in an exponentially moving average manner where f i t is the current match detection appearance embedding, and a = 0.9 is a momentum term; An adaptive noise is also added to enhance the robustness of the tracking, where the covariance of the adaptive noise is As shown in equation (7): where R k is a pre-set constant measurement noise covariance, c k is the detection confidence score at state k, and instead of using only the appearance feature distance in the matching process, both appearance and motion information are considered simultaneously; Instead of a matching cascade, a common global linear assignment is used, where the assignment matrix C is a weighted sum of the appearance costs A a and the action costs A m ​ C = λA a + (1 - λ)A m (8) Wherein the weight factor λ is set to 0.98; Step 4.2: After the track is predicted by Kalman filter algorithm, a track is predicted for the current frame, if the prediction is confirmed, detection is performed on the current frame, then the detection box and the confirmed track box are matched, after the matching is completed, the detection box of the track is updated; If the track matching fails, IoU matching is performed, if the matching is successful, the track is updated, and then the prediction-observation-update tracking process is repeated, IoU matching failure is divided into observation matching failure and track matching failure: for observation matching failure, a new track is established for it, and then it is investigated three times, if it is still the actual target, it is modified as confirmed; for track matching failure, it is judged whether it is a confirmed pedestrian or vehicle, if it is in an unconfirmed state, it is deleted, otherwise, a threshold is set for it, if it is greater than the threshold max_age, it is deleted, considering that it has moved out of the observation range, if it is less than the threshold, it is also investigated three times, returning to the initial stage.

Images