Monocular multi-target tracking and speed measuring method based on deep learning

Abstract

The invention provides a monocular multi-target tracking and speed measuring method based on deep learning, and the method comprises the following steps: preparing a training data set, collecting the data of real vehicles and pedestrians in real time through a camera under a traffic real scene, adding an open source data set, and carrying out the real-time collection of the data; the method can effectively improve the robustness and generalization of subsequent reasoning of a model, adopts YOLOv5 as a backbone network architecture for model training, trains the vehicle and pedestrian data sets obtained in the S1 to obtain a target detection model, and uses the vehicle and pedestrian target detection model trained in the S2 to quickly and accurately detect all traffic objects in a video frame. And then the detection results are transmitted to a tracking algorithm, and the movement speed of the traffic object is measured and calculated. The monocular multi-target tracking and speed measuring method based on deep learning provided by the invention is high in operation speed in practical application reasoning, very high in adaptability in hardware deployment, simple and small in model packaging, and low in hardware computing power performance requirement.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a monocular multi-target tracking and speed measurement method based on deep learning. Background Art

[0002] With the rapid development of artificial intelligence, deep learning technology is constantly updating and iterating, which can better empower cutting-edge industry applications. Among them, urban road monitoring systems play an important role in public security prevention and control. Further optimizing the monitoring system and expanding its functions can not only reduce its construction and maintenance costs, but also provide technical support for various types of traffic, public security, etc., greatly improving the level and efficiency of law enforcement management. At present, in the field of intelligent traffic monitoring, when people solve the problems of (vehicle, personnel) target tracking and speed measurement, they mostly use discriminant models, using the principle of correlation filtering to distinguish target objects and background environments in monitoring video sequences, that is, learning a classifier or a filter to distinguish targets and non-targets, and using the learned classifier or filter as a discriminant model in the video frame to search and locate targets. In subsequent new frames, multiple candidate areas are evaluated, and the area with the highest score output by the discriminant model is selected as the new position of the target, so as to achieve the tracking effect, and then mathematical analysis is performed to calculate the movement speed of the tracked target.

[0003] A discriminant model is used to track targets using the principle of correlation filtering. The effect of target tracking depends largely on the correct prediction of the approximate position of the target. The discriminant model is fine-tuned to provide a more accurate tracking effect. If the appearance of the tracked object is very simple and does not change during movement, and there is no object occlusion, the tracking effect will be very good. However, in actual traffic scenarios, the appearance of the tracked object is likely to change significantly, and there will also be a large number of tracked objects occluding each other. The discriminant model is easily disturbed, resulting in poor tracking effect and the accuracy of subsequent speed measurement analysis is also affected.

[0004] Therefore, it is necessary to provide a monocular multi-target tracking and speed measurement method based on deep learning to solve the above technical problems. Summary of the invention

[0005] The present invention provides a monocular multi-target tracking and speed measurement method based on deep learning, which solves the problem that in actual traffic scenarios, the appearance of the tracked objects is likely to change significantly, and there will also be a large number of tracked objects occluding each other. The discriminant model is easily disturbed, resulting in poor tracking effect and affecting the accuracy of subsequent speed measurement analysis.

[0006] In order to solve the above technical problems, the present invention provides a monocular multi-target tracking and speed measurement method based on deep learning, comprising the following steps:

[0007] S1. Prepare training data sets. Use cameras to collect real vehicle and pedestrian data in real traffic scenes. Also, add open source data sets to effectively improve the robustness and generalization of subsequent model reasoning.

[0008] S2, using YOLOv5 as the backbone network architecture for model training, training the vehicle and pedestrian data sets obtained in S1 to obtain a target detection model;

[0009] S3, using the vehicle and pedestrian detection model trained in S2, quickly and accurately detect all traffic objects in the video frame, and then pass these detection results to the tracking algorithm;

[0010] S4. Calculate the movement speed of the traffic object.

[0011] Preferably, the open source data sets in S1 include (COCO2017, KITTI, VisDrone2019, OpenImagesV7, etc.), and all data are screened, cleaned, and labeled to produce a large-scale training data set suitable for human and vehicle target detection.

[0012] Preferably, YOLOv5 is selected as the deep learning target detection network architecture in S2, because YOLOv5 further improves the performance and flexibility based on the success of the previous YOLO version in the target detection task, and has cutting-edge performance in terms of accuracy and speed. At the same time, the YOLOv5 network architecture model is better adapted to the hardware rk3588 development board to achieve a more lightweight deployment. The present invention adds a lightweight, parameter-free convolutional neural network attention mechanism SimAM module to the YOLOv5 network framework, and quotes the wiou loss function.

[0013] Preferably, the SimAM module generates attention weights by calculating the local self-similarity of the feature map, and adding the SimAM module does not require the introduction of any additional parameters, which can effectively improve the convolution learning ability during model training. The core idea of ​​the SimAM module is based on the local self-similarity of the image. In the image, adjacent pixels usually have a strong similarity, while the similarity between distant pixels is weaker. SimAM uses this feature to generate attention weights by calculating the similarity between each pixel in the feature map and its adjacent pixels.

[0014] Preferably, the SimAM running process first uses the global average pooling operation to extract the global features of the feature map, then calculates each pixel to obtain the local features of the pixel, and then uses the global features and local features to calculate to obtain the attention weight of each pixel, and uses the attention weight to perform weighted fusion on the local features to obtain the final feature representation, and its calculation expression is:

[0015] A(x)=σ(W×G(x)+b)

[0016] In the expression, A(x) represents the attention weight, x is the local feature, G(x) is the global feature, W is the weight matrix, b is the bias term, and σ is the activation function;

[0017] Since the current training data set is large and contains various open source data, there are inconsistencies in data annotation standards and quality. When improving the model, the wiou loss function is used to solve the balance problem between samples with good data quality and samples with poor data quality, which helps to improve the generalization ability of the model;

[0018] The wiou loss function needs to use IOU to find the intersection-over-union ratio of the predicted anchor box and the true anchor box, L IOU =1-IOU, where L IOU Expressed as IOU loss, the target anchor box outlier β is defined, and the outlier β is the sum of the current IOU loss and L IOU The mean ratio of

[0019]

[0020] The smaller the outlier degree β of the target anchor box, the higher the quality of the anchor box. A small gradient gain is assigned to make the bounding box regression focus on the anchor box of normal quality. When the outlier degree β of the target anchor box is large, a smaller gradient gain is assigned to effectively prevent the influence of low-quality label data.

[0021] Preferably, the process of tracking traffic objects in S3 is divided into the following steps:

[0022] S31, initializing data information: first create an empty trajectory set to store the tracking information of each traffic object in the video;

[0023] S32, data target detection: perform traffic target detection on the video to be tracked and analyzed (the target detection here is to use the model trained in S2 for reasoning), and output the position anchor frame of the predicted target and the confidence information of its predicted target anchor frame;

[0024] S33, classifying the target detection results: setting a specific threshold for the confidence information of the target anchor box obtained in S2 to distinguish between high confidence detection and low confidence detection, where the threshold is selected as 0.6;

[0025] S34, traffic object trajectory prediction: use the Kalman algorithm to predict the new position of each existing trajectory in the current frame;

[0026] S35, first data association: using similarity metrics, associate existing trajectories with high-confidence detections;

[0027] S36, updating the remaining unmatched detections and trajectories: after S5, searching for the remaining unmatched detection anchor boxes and trajectory information;

[0028] S37, second data association: Use similarity measurement to associate the remaining tracks of S6 with the low confidence detections in S3;

[0029] S38, cleaning up the trajectory information: deleting all the trajectory information not matched in S7 from the trajectory set;

[0030] S39, initializing new track information: for the remaining unmatched detections after the second association, a new track is initialized for each detection and added to the track set;

[0031] S310, returning the traffic object trajectory result: returning an updated trajectory set, that is, the tracking path information of all traffic objects.

[0032] Preferably, the Kalman algorithm in S34 is used to predict the target position during multi-target tracking. To implement the Kalman algorithm calculation, the target state and its covariance matrix to be tracked must first be initialized, and then the state of the tracked target at the next moment must be predicted, and the target state and covariance matrix information must be updated. The mathematical expression is:

[0033]

[0034] in Indicates the state of the predicted tracking target, represents the predicted covariance matrix, F t is the state transfer matrix, B t is the control input matrix, u t is the control input vector, Q t is the process noise covariance matrix, where t represents the tth moment and t-1 represents the t-1th moment.

[0035] Preferably, in S4, it is first necessary to determine that the image video collector is fixedly placed and has clear position information, and then use the above S2 and S3 to detect and track the traffic target objects in the image in real time, extract the detection anchor frame of each target, and further mathematically convert each detected traffic target anchor frame into its pixel coordinates in the image to deduce the actual distance between each traffic target and the image video collector;

[0036] Then we start to calculate the speed of the tracked target and analyze the continuous frames of the real-time video. After successfully detecting and tracking the traffic target, we record the position change of each time step and the target detection anchor frame. The mathematical expression is:

[0037]

[0038] That is, the instantaneous speed Δv is equal to the ratio of distance to time, where Δs is the actual distance change between the homologous traffic target and the image video collector in two adjacent frames, and the calculation formula of Δt is:

[0039]

[0040] Δt represents the time difference between two adjacent frames, and fps is the frequency at which images appear continuously on the display, that is, the frame rate of the image video collector. Here, Δt is converted into seconds to facilitate subsequent measurement and recording, thus realizing the speed measurement function of the tracked target.

[0041] Compared with the related art, the monocular multi-target tracking and speed measurement method based on deep learning provided by the present invention has the following beneficial effects:

[0042] The present invention provides a monocular multi-target tracking and speed measurement method based on deep learning. The proposed technical solution first establishes a detection model based on the deep learning model, obtains prior target information based on the powerful detection model, and references the association strategy to achieve high-precision tracking and speed measurement functions for traffic objects. It has a fast running speed in actual application reasoning and can realize real-time monitoring. It has strong adaptability in hardware deployment, the model encapsulation is simple and compact, and does not require high hardware computing power performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic structural diagram of a preferred embodiment of a monocular multi-target tracking and speed measurement method based on deep learning provided by the present invention;

[0044] Figure 2 This is a schematic diagram of image and video acquisition;

[0045] Figure 3 Schematic diagram of target tracking execution strategy. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0047] First embodiment

[0048] Please refer to Figure 1 , Figure 2 and Figure 3 ,in, Figure 1 A schematic diagram of a preferred embodiment of a monocular multi-target tracking and speed measurement method based on deep learning provided by the present invention; Figure 2 This is a schematic diagram of image and video acquisition; Figure 3 Schematic diagram of target tracking execution strategy. A monocular multi-target tracking and speed measurement method based on deep learning includes the following steps:

[0049] S1. Prepare training data sets. Use cameras to collect real vehicle and pedestrian data in real traffic scenes. Also, add open source data sets to effectively improve the robustness and generalization of subsequent model reasoning.

[0050] S2, using YOLOv5 as the backbone network architecture for model training, training the vehicle and pedestrian data sets obtained in S1 to obtain a target detection model;

[0051] S3, using the vehicle and pedestrian detection model trained in S2, quickly and accurately detect all traffic objects in the video frame, and then pass these detection results to the tracking algorithm;

[0052] S4. Calculate the movement speed of the traffic object.

[0053] The open source data sets in S1 include (COCO2017, KITTI, VisDrone2019, OpenImagesV7, etc.). All data are screened, cleaned, and labeled to produce a large-scale training data set suitable for human and vehicle target detection.

[0054] In the S2, YOLOv5 is selected as the deep learning target detection network architecture, because YOLOv5 further improves the performance and flexibility based on the success of the previous YOLO version in the target detection task, and has cutting-edge performance in terms of accuracy and speed. At the same time, the YOLOv5 network architecture model is better adapted to the hardware rk3588 development board to achieve a lighter deployment. The present invention adds a lightweight, parameter-free convolutional neural network attention mechanism SimAM module to the YOLOv5 network framework, and quotes the wiou loss function.

[0055] YOLOv5 is selected as the deep learning target detection network architecture here, because YOLOv5 further improves the performance and flexibility based on the success of the previous YOLO version in the target detection task, and has cutting-edge performance in both accuracy and speed. At the same time, the YOLOv5 network architecture model is better adapted to the hardware rk3588 development board to achieve a lighter deployment. The present invention adds a lightweight, parameter-free convolutional neural network attention mechanism SimAM module to the YOLOv5 network framework, and quotes the wiou loss function.

[0056] The SimAM module generates attention weights by calculating the local self-similarity of the feature map, and adding the SimAM module does not require the introduction of any additional parameters, which can effectively improve the convolution learning ability during model training. The core idea of ​​the SimAM module is based on the local self-similarity of the image. In the image, adjacent pixels usually have strong similarities, while the similarity between distant pixels is weaker. SimAM uses this feature to generate attention weights by calculating the similarity between each pixel in the feature map and its adjacent pixels.

[0057] The SimAM running process first uses the global average pooling operation to extract the global features of the feature map, then calculates each pixel to obtain the local features of the pixel, and then uses the global features and local features to calculate to obtain the attention weight of each pixel. The attention weight is used to perform weighted fusion on the local features to obtain the final feature representation, and its calculation expression is:

[0058] A(x)=σ(W×G(x)+b)

[0059] In the expression, A(x) represents the attention weight, x is the local feature, G(x) is the global feature, W is the weight matrix, b is the bias term, and σ is the activation function;

[0060] Since the current training data set is large and contains various open source data, there are inconsistencies in data annotation standards and quality. When improving the model, the wiou loss function is used to solve the balance problem between samples with good data quality and samples with poor data quality, which helps to improve the generalization ability of the model;

[0061] The wiou loss function needs to use IOU to find the intersection-over-union ratio of the predicted anchor box and the true anchor box, L IOU =1-IOU, where L IOU Expressed as IOU loss, the target anchor box outlier β is defined, and the outlier β is the sum of the current IOU loss and L IOU The mean ratio of

[0062]

[0063] The smaller the outlier degree β of the target anchor box, the higher the quality of the anchor box. A small gradient gain is assigned to make the bounding box regression focus on the anchor box of normal quality. When the outlier degree β of the target anchor box is large, a smaller gradient gain is assigned to effectively prevent the influence of low-quality label data.

[0064] The process of tracking traffic objects in S3 is divided into the following steps:

[0065] S31, initializing data information: first create an empty trajectory set to store the tracking information of each traffic object in the video;

[0066] S32, data target detection: perform traffic target detection on the video to be tracked and analyzed (the target detection here is to use the model trained in S2 for reasoning), and output the position anchor frame of the predicted target and the confidence information of its predicted target anchor frame;

[0067] S33, classifying the target detection results: setting a specific threshold for the confidence information of the target anchor box obtained in S2 to distinguish between high confidence detection and low confidence detection, where the threshold is selected as 0.6;

[0068] S34, traffic object trajectory prediction: use the Kalman algorithm to predict the new position of each existing trajectory in the current frame;

[0069] S35, first data association: using similarity metrics, associate existing trajectories with high-confidence detections;

[0070] S36, updating the remaining unmatched detections and trajectories: after S5, searching for the remaining unmatched detection anchor boxes and trajectory information;

[0071] S37, second data association: Use similarity measurement to associate the remaining tracks of S6 with the low confidence detections in S3;

[0072] S38, cleaning up the trajectory information: deleting all the trajectory information not matched in S7 from the trajectory set;

[0073] S39, initializing new track information: for the remaining unmatched detections after the second association, a new track is initialized for each detection and added to the track set;

[0074] S310, returning the traffic object trajectory result: returning an updated trajectory set, that is, the tracking path information of all traffic objects.

[0075] The Kalman algorithm in S34 is used to predict the target position during multi-target tracking. To implement the Kalman algorithm calculation, the target state and its covariance matrix to be tracked must first be initialized, and then the state of the tracked target at the next moment is predicted, and the target state and covariance matrix information are updated. The mathematical expression is:

[0076]

[0077] in Indicates the state of the predicted tracking target, represents the predicted covariance matrix, F t is the state transfer matrix, B t is the control input matrix, u t is the control input vector, Q t is the process noise covariance matrix. Here t represents the tth moment, and t-1 represents the t-1th moment.

[0078] In S4, it is first necessary to determine that the image video collector is fixed and has clear position information, and then use the above S2 and S3 to detect and track the traffic target objects in the image in real time, extract the detection anchor frame of each target, and further mathematically convert each detected traffic target anchor frame into its pixel coordinates in the image to infer the actual distance between each traffic target and the image video collector;

[0079] Then we start to calculate the speed of the tracked target and analyze the continuous frames of the real-time video. After successfully detecting and tracking the traffic target, we record the position change of each time step and the target detection anchor frame. The mathematical expression is:

[0080]

[0081] That is, the instantaneous speed Δv is equal to the ratio of distance to time, where Δs is the actual distance change between the homologous traffic target and the image video collector in two adjacent frames, and the calculation formula of Δt is:

[0082]

[0083] Δt represents the time difference between two adjacent frames, and fps is the frequency at which images appear continuously on the display, that is, the frame rate of the image video collector. Here, Δt is converted into seconds to facilitate subsequent measurement and recording, thus realizing the speed measurement function of the tracked target.

[0084] Compared with the related art, the monocular multi-target tracking and speed measurement method based on deep learning provided by the present invention has the following beneficial effects:

[0085] The present invention provides a monocular multi-target tracking and speed measurement method based on deep learning. The proposed technical solution first establishes a detection model based on the deep learning model, obtains prior target information based on the powerful detection model, and references the association strategy to achieve high-precision tracking and speed measurement functions for traffic objects. The present invention runs fast in actual application reasoning, can realize real-time monitoring, has strong adaptability in hardware deployment, has a simple and compact model package, and does not require high hardware computing power performance.

[0086] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A monocular multi-target tracking and speed measurement method based on deep learning, characterized in that: The following steps are involved: S1. Prepare training data sets. Use cameras to collect real vehicle and pedestrian data in real traffic scenes. Also, add open source data sets to effectively improve the robustness and generalization of subsequent model reasoning. S2, using YOLOv5 as the backbone network architecture for model training, training the vehicle and pedestrian data sets obtained in S1 to obtain a target detection model; S3, using the vehicle and pedestrian detection model trained in S2, quickly and accurately detect all traffic objects in the video frame, and then pass these detection results to the tracking algorithm; S4. Calculate the movement speed of the traffic object.

2. The method for monocular multi-target tracking and speed measurement based on deep learning according to claim 1, characterized in that: The open source data sets in S1 include (COCO2017, KITTI, VisDrone2019, OpenImagesV7, etc.). All data are screened, cleaned, and labeled to produce a large-scale training data set suitable for human and vehicle target detection.

3. The monocular multi-target tracking and speed measurement method based on deep learning according to claim 1, characterized in that: In the S2, YOLOv5 is selected as the deep learning target detection network architecture, because YOLOv5 further improves the performance and flexibility based on the success of the previous YOLO version in the target detection task, and has cutting-edge performance in terms of accuracy and speed. At the same time, the YOLOv5 network architecture model is better adapted to the hardware rk3588 development board to achieve a lighter deployment. The present invention adds a lightweight, parameter-free convolutional neural network attention mechanism SimAM module to the YOLOv5 network framework, and quotes the wiou loss function.

4. The method for monocular multi-target tracking and speed measurement based on deep learning according to claim 3, characterized in that: The SimAM module generates attention weights by calculating the local self-similarity of the feature map, and adding the SimAM module does not require the introduction of any additional parameters, which can effectively improve the convolution learning ability during model training. The core idea of ​​the SimAM module is based on the local self-similarity of the image. In the image, adjacent pixels usually have strong similarities, while the similarity between distant pixels is weaker. SimAM uses this feature to generate attention weights by calculating the similarity between each pixel in the feature map and its adjacent pixels.

5. The monocular multi-target tracking and speed measurement method based on deep learning according to claim 1, characterized in that: The SimAM running process first uses the global average pooling operation to extract the global features of the feature map, then calculates each pixel to obtain the local features of the pixel, and then uses the global features and local features to calculate to obtain the attention weight of each pixel. The attention weight is used to perform weighted fusion on the local features to obtain the final feature representation, and its calculation expression is: A(x)=σ(W×G(x)+b) In the expression, A(x) represents the attention weight, x is the local feature, G(x) is the global feature, W is the weight matrix, b is the bias term, and σ is the activation function; Since the current training data set is large and contains various open source data, there are inconsistencies in data annotation standards and quality. When improving the model, the wiou loss function is used to solve the balance problem between samples with good data quality and samples with poor data quality, which helps to improve the generalization ability of the model; The wiou loss function needs to use IOU to find the intersection-over-union ratio of the predicted anchor box and the true anchor box, L IOU =1-IOU, where L IOU Expressed as IOU loss, the target anchor box outlier β is defined, and the outlier β is the sum of the current IOU loss and L IOU The mean ratio of The smaller the outlier degree β of the target anchor box, the higher the quality of the anchor box. A small gradient gain is assigned to make the bounding box regression focus on the anchor box of normal quality. When the outlier degree β of the target anchor box is large, a smaller gradient gain is assigned to effectively prevent the influence of low-quality label data.

6. The monocular multi-target tracking and speed measurement method based on deep learning according to claim 1, characterized in that: The process of tracking traffic objects in S3 is divided into the following steps: S31, initializing data information: first create an empty trajectory set to store the tracking information of each traffic object in the video; S32, data target detection: perform traffic target detection on the video to be tracked and analyzed (the target detection here is to use the model trained in S2 for reasoning), and output the position anchor frame of the predicted target and the confidence information of its predicted target anchor frame; S33, classifying the target detection results: setting a specific threshold for the confidence information of the target anchor box obtained in S2 to distinguish between high confidence detection and low confidence detection, where the threshold is selected as 0.6; S34, traffic object trajectory prediction: use the Kalman algorithm to predict the new position of each existing trajectory in the current frame; S35, first data association: using similarity metrics, associate existing trajectories with high-confidence detections; S36, updating the remaining unmatched detections and trajectories: after S5, searching for the remaining unmatched detection anchor boxes and trajectory information; S37, second data association: Use similarity measurement to associate the remaining tracks of S6 with the low confidence detections in S3; S38, cleaning up the trajectory information: deleting all the trajectory information not matched in S7 from the trajectory set; S39, initializing new track information: for the remaining unmatched detections after the second association, a new track is initialized for each detection and added to the track set; S310, returning the traffic object trajectory result: returning an updated trajectory set, that is, the tracking path information of all traffic objects.

7. The monocular multi-target tracking and speed measurement method based on deep learning according to claim 1, characterized in that: The Kalman algorithm in S34 is used to predict the target position during multi-target tracking. To implement the Kalman algorithm calculation, the target state and its covariance matrix to be tracked must first be initialized, and then the state of the tracked target at the next moment is predicted, and the target state and covariance matrix information are updated. The mathematical expression is: in Indicates the state of the predicted tracking target, represents the predicted covariance matrix, F t is the state transfer matrix, B t is the control input matrix, u t is the control input vector, Q t is the process noise covariance matrix, where t represents the tth moment and t-1 represents the t-1th moment.

8. The monocular multi-target tracking and speed measurement method based on deep learning according to claim 1, characterized in that: In S4, it is first necessary to determine that the image video collector is fixed and has clear position information, and then use the above S2 and S3 to detect and track the traffic target objects in the image in real time, extract the detection anchor frame of each target, and further mathematically convert each detected traffic target anchor frame into its pixel coordinates in the image to infer the actual distance between each traffic target and the image video collector; Then we start to calculate the speed of the tracked target and analyze the continuous frames of the real-time video. After successfully detecting and tracking the traffic target, we record the position change of each time step and the target detection anchor frame. The mathematical expression is: That is, the instantaneous speed Δv is equal to the ratio of distance to time, where Δs is the actual distance change between the homologous traffic target and the image video collector in two adjacent frames, and the calculation formula of Δt is: Δt represents the time difference between two adjacent frames, and fps is the frequency at which images appear continuously on the display, that is, the frame rate of the image video collector. Here, Δt is converted into seconds to facilitate subsequent measurement and recording, thus realizing the speed measurement function of the tracked target.

Images