A detection-based multi-vehicle tracking method and system

Abstract

The present invention discloses a detection-based multi-vehicle tracking method and system, and relates to the fields of computer vision and artificial intelligence technology. Compared with the previous vehicle tracking detection method, the present invention solves the problem that the prior art cannot ensure that the real-time requirements and tracking accuracy in the tracking process are met simultaneously due to data association matching and ID conversion problems caused by occlusion in the tracking process; the lightweight detection model through the GSConv module solves the drawbacks of the previous algorithm in real-time processing, and the VOVGSCSP module improves the problem of inaccurate detection caused by occlusion in the vehicle detection process. In the tracking module, a method for reconstructing the spatial state vector of the Kalman filter is proposed. Taking into account the shortcomings of the original Byte Track algorithm in trajectory processing, the problem of frequent ID conversion and insufficient tracking accuracy caused by occlusion of the vehicle is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a detection-based multi-vehicle tracking method and system. Background Art

[0002] The current mainstream research directions of vehicle tracking algorithms can be divided into detection-based tracking and joint detection tracking algorithms. The former mainly relies on the performance of detection to improve the tracking effect, while the latter mainly uses joint learning and weight sharing to enhance the coupling between the functional modules of the two detection tasks and tracking tasks. In contrast, the detection-based tracking method has certain advantages in modular design and algorithm flexibility. In the detection-based tracking system, when the tracking module loses the target, the detection algorithm can still rediscover the target in subsequent frames and perform corresponding initialization. Gradually, the detection-based vehicle tracking algorithm has occupied the mainstream position. The detection-based multi-vehicle tracking method used in the present invention can be divided into two modules: a detection module and a tracking module.

[0003] Detection algorithms are the core research direction in the field of computer vision and an important part of the field of artificial intelligence. Since the 1990s, with the development of machine learning technology, traditional target detection methods have begun to emerge, including support vector machines and adaptive boosting algorithms. However, the shortcomings of this algorithm are also very obvious, such as the inability to adapt to complex environments, poor real-time and robustness, and high cost. Compared with traditional target detection algorithms, target detection algorithms based on deep learning can better cope with complex environments and have faster detection speeds and higher detection accuracy.

[0004] In 2012, convolutional neural networks were successfully applied to the field of computer vision, which started the rapid development of convolutional neural networks in this field. Subsequently, two-stage target detection algorithms were gradually proposed. The two-stage target detection algorithm first needs to use a detector to generate candidate regions, and then use a classifier to classify these candidate regions. However, this type of algorithm generally has the disadvantages of large number of parameters and large amount of calculation, resulting in poor real-time performance.

[0005] In 2015, Joseph Redmon and others proposed a single-stage target detection algorithm, YOLO. This algorithm can directly classify and locate target objects through convolutional neural networks, avoiding the cumbersome process of two-step two-stage target detection algorithms, greatly speeding up the detection speed. From 2017 to 2020, based on YOLO, YOLOv2, YOLOv3, YOLOv4, and YOLOv5 with better performance were successively proposed. Currently, the more mature YOLO algorithm is represented by YOLOv7. The rapid development of target detection algorithms has not only been applied in many fields, but also has a profound impact on people's daily lives. However, due to the complexity of the algorithm, the YOLOv7 algorithm cannot meet the needs of high-precision real-time detection during target detection, and there is still room for improvement in improving the detection speed of the algorithm.

[0006] In the blue ocean of target tracking development in the past few years, detection-based tracking methods have achieved outstanding performance. At that time, the SORT algorithm was a classic algorithm for detection and tracking based on deep neural networks. The algorithm uses a faster regional convolutional neural network (Faster Region-CNN, Faster R-CNN) as a detector, and on this basis, uses the Kalman filter and the Hungarian algorithm to achieve trajectory prediction and state update of the target, as well as the matching between the prediction results and the detection results. For the design of the tracking framework, the SORT algorithm adopted a simple solution, which had a unique real-time online tracking performance at the time. However, since the SORT algorithm only uses the intersection over union (IOU) strategy for detection results and prediction results, it is difficult to achieve stable tracking in occluded environments, and the overall robustness still needs to be improved.

[0007] Subsequently, various improved versions such as Deep Sort, Strong Sort, Bot Sort, and Byte Track were proposed based on SORT. Corresponding improvement measures were proposed for the problem of ID conversion caused by occlusion during data association matching and tracking. However, there has been no effective solution to the occlusion problem in the tracking process. This has always been the focus of researchers, and how to ensure that the real-time requirements and tracking accuracy can be met during the tracking process has always been the mainstream direction of research.

[0008] In order to solve the above problems, the present invention proposes a detection-based multi-vehicle tracking method and system. Summary of the invention

[0009] The purpose of the present invention is to provide a detection-based multi-vehicle tracking method and system to solve the problems raised in the background technology:

[0010] The existing technology has problems with data association matching and ID conversion caused by occlusion during tracking, and cannot guarantee that both real-time requirements and tracking accuracy can be met during the tracking process.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] A detection-based multi-vehicle tracking method comprises the following steps:

[0013] S1: Build the YOLOv7 multi-vehicle detection algorithm model;

[0014] S2: The GSConv module is used to perform lightweight processing on the YOLOv7 multi-vehicle detection algorithm model, and the VOVGSCSP module is introduced into the head network of the YOLOv7 multi-vehicle detection algorithm to improve the detection accuracy;

[0015] S3: Shape-iou combined with Soft-NMS is used in the loss function of the YOLOv7 multi-vehicle detection algorithm to build an improved YOLOv7 multi-vehicle detection algorithm model;

[0016] S4: Build the Byte Track multi-vehicle tracking algorithm model;

[0017] S5: Reconstruct the spatial state vector of the Byte Track multi-vehicle tracking algorithm model and build an improved Byte Track multi-vehicle tracking algorithm model;

[0018] S6: Detect and track multiple vehicles based on the improved YOLOv7 multi-vehicle detection algorithm model and the improved Byte Track multi-vehicle tracking algorithm model.

[0019] Preferably, in S2, the GSConv module uses group convolution and channel shuffling to perform lightweight processing of the model; the group convolution limits each convolution kernel to operate only with channels in the same group, and adjusts the number of restricted groups; the channel shuffling is used to promote information exchange and fusion between different channels.

[0020] Preferably, in S2, a cross-stage local network VOVGSCSP module designed based on an aggregation algorithm performs multi-level learning on features, and the VOVGSCSP module includes a GS bottleneck module, in which GSConv and Shortcut are connected, and the Shortcut implements point-by-point operation.

[0021] Preferably, in S4, for the occlusion problem in detection, the YOLOv7 multi-vehicle detection algorithm model is further improved by combining Shape-iou with Soft-NMS method. The Shape-iou calculates the shape cost by considering the width and height, and evaluates the shape similarity of the two bounding boxes. The Shape-iou score is calculated based on the following formula, that is, the loss function L Shape-iou The details are as follows:

[0022] L Shape-iou =1-IOU+distance shape +0.5*shape_cost

[0023] Among them, distance shape is the center distance of shape normalization; shape_cost is the shape cost; IOU means intersection-over-union score;

[0024]

[0025] Among them, B and B gt Represent the predicted box and the true bounding box respectively;

[0026] When an object is occluded, the Soft-NMS method distinguishes the detection frame of the occluded object from the repeated detection frames of the same object through secondary processing of the threshold.

[0027] Preferably, in S5, the spatial state vector of the Kalman filter in the Byte Track multi-vehicle tracking algorithm model is (x, y, a, h, vx, vy, va, vh), where (x, y) are the coordinates of the center point of the target frame, h is the frame height, a is the aspect ratio of the target frame, vx, vy, va and vh are the first-order derivatives of the variables, and the spatial state vector of the Kalman filter is reconstructed as (x, y, w, h, vx, vy, vw, vh), where w is the target frame width and vw is the first-order derivative of the width variable.

[0028] Preferably, in S6, the specific steps are as follows:

[0029] S6.1: Collect video frames and perform target detection based on the improved YOLOv7 multi-vehicle detection algorithm model;

[0030] S6.2: Divide the detection results into two categories: high confidence score detection boxes and low confidence score detection boxes based on the detection threshold;

[0031] S6.3: First, the high confidence score detection box is first associated with the result predicted by the Kalman filter, the associated result is predicted based on the Kalman filter, and then the low confidence score detection box is used to perform a second data association on the remaining unmatched tracks;

[0032] S6.4: Track the matched vehicles in the two association results, update the vehicle trajectory based on the Kalman filter, and update it to the prediction result before the first data association.

[0033] Preferably, the first data association and the second data association in S6.3 are both based on the Hungarian matching algorithm for data association; during the first data association process, a new track is created for the unmatched detection box, and during the second data association process, the tracking is considered lost for the second unmatched track.

[0034] A detection-based multi-vehicle tracking system comprising:

[0035] Data reading module: used to read image data, video files, real-time video streams from cameras, and vehicle detection or tracking modes selected by users;

[0036] Vehicle detection module: used to detect image data, video files and camera real-time video streams based on user selection through a detector built based on the improved YOLOv7 multi-vehicle detection algorithm model;

[0037] Vehicle tracking module: used to track the corresponding vehicles according to the user's selection results through the tracker built based on the improved Byte Track multi-vehicle tracking algorithm model;

[0038] Tracking display module: used to visualize the detection or tracking results.

[0039] Preferably, in the system, only a vehicle detection function is provided for image data; and two functions, vehicle detection and vehicle tracking, are provided for video files and real-time camera video streams, wherein the vehicle detection function detects and feeds back each frame of image.

[0040] Preferably, in the vehicle tracking module, when tracking multiple vehicles, an ID is assigned to each target for real-time tracking.

[0041] Compared with the prior art, the present invention provides a detection-based multi-vehicle tracking method and system, which has the following beneficial effects:

[0042] The present invention solves the drawbacks of previous algorithms in real-time processing by lightweighting the detection model through the GSConv module, and improves the problem of inaccurate detection due to occlusion during vehicle detection through the VOVGSCSP module. In the tracking module, a method for reconstructing the spatial state vector of the Kalman filter is proposed. Taking into account the shortcomings of the original Byte Track algorithm in trajectory processing, the problems of frequent ID conversion and insufficient tracking accuracy caused by occlusion of vehicles are solved. A multi-vehicle detection and tracking system that can be applied to urban road traffic has also been developed. The system supports web-based deployment and supports access through mobile terminals in the same local area network. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the method mentioned in Example 1 of the present invention;

[0044] Figure 2 This is a schematic diagram of the YOLOv7 multi-vehicle detection algorithm model structure mentioned in Example 1 of the present invention;

[0045] Figure 3 Schematic diagram of the GSConv module mentioned in Example 1 of the present invention;

[0046] Figure 4 Schematic diagram of the VOVGSCSP module based on GSConv mentioned in Example 1 of the present invention;

[0047] Figure 5 This is a schematic diagram of the overlapped blocks mentioned in Embodiment 1 of the present invention;

[0048] Figure 6 It is a pseudo code diagram of Shape-iou mentioned in Example 1 of the present invention;

[0049] Figure 7 This is a schematic diagram of the change of regression loss during the training process using Shape-iou and original CIOU mentioned in Example 1 of the present invention;

[0050] Figure 8 This is a schematic diagram of the structure of the improved YOLOv7 multi-vehicle detection algorithm model mentioned in Example 1 of the present invention;

[0051] Fig. 9 It is a schematic diagram of the detection effect before and after the improvement of the YOLOv7 multi-vehicle detection algorithm model mentioned in Example 1 of the present invention;

[0052] Fig.10 This is a schematic diagram showing that the vehicles mentioned in Embodiment 1 of the present invention may have the same aspect ratio;

[0053] Fig.11This is a flow chart of the improved YOLOv7 multi-vehicle detection algorithm + improved ByteTrack vehicle detection and tracking algorithm mentioned in Example 1 of the present invention;

[0054] Fig.12 This is a schematic diagram of the occlusion situation in the tracking process combined with the improved YOLOv7 multi-vehicle detection algorithm + the improved ByteTrack vehicle detection and tracking algorithm mentioned in Example 1 of the present invention;

[0055] Fig.13 This is a system block diagram mentioned in Embodiment 2 of the present invention;

[0056] Fig.14 This is the overall system flow chart mentioned in Example 2 of the present invention;

[0057] Fig.15 This is a schematic diagram of the data set preprocessing results mentioned in Example 3 of the present invention;

[0058] Fig.16 This is a schematic diagram of the initial page of the web page system mentioned in Embodiment 3 of the present invention;

[0059] Fig.17 This is a schematic diagram of the detection and tracking results of the web page system mentioned in Example 3 of the present invention;

[0060] Fig.18 This is a schematic diagram of the initial page of the mobile terminal system mentioned in Embodiment 3 of the present invention;

[0061] Fig.19 This is a schematic diagram of the mobile terminal system detection results mentioned in Example 3 of the present invention.

[0062] Meaning of the symbols in the figure:

[0063] 100, data reading module; 200, vehicle detection module; 300, vehicle tracking module; 400, tracking display module. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0065] The vehicle detection and tracking model of improved YOLOv7+improved Byte Track designed by the present invention uses the GSConv module throughout the whole model for lightweighting in the detection part, introduces the VOVGSCSP module into the head network of YOLOv7 to improve the accuracy, uses the Shape-iou combined with Soft-NMS solution in the loss function part of the model, and improves the reconstruction of the original Kalman filter after analyzing the spatial state vector in the tracking part.

[0066] By lightweighting the detection model through the GSConv module, the shortcomings of the previous algorithm in real-time processing are optimized, and the problem of inaccurate detection due to occlusion during vehicle detection is improved through the VOVGSCSP module. In the tracking module, the spatial state vector of the Kalman filter is reconstructed. Considering the shortcomings of the original Byte Track algorithm in trajectory processing, the problem of frequent ID conversion and insufficient tracking accuracy caused by occlusion of vehicles is solved. A multi-vehicle detection and tracking system that can be applied to urban road traffic has also been developed. The system supports web-based deployment and can be accessed through mobile terminals in the same local area network. The specific content includes the following.

[0067] Embodiment 1:

[0068] See also Figure 1-12 The present invention provides a detection-based multi-vehicle tracking method, comprising:

[0069] S1: Build the YOLOv7 multi-vehicle detection algorithm model; the details are as follows:

[0070] Reference Figure 2 The YOLOv7 network structure, the YOLOv7 multi-vehicle detection algorithm model consists of three parts: Input, Backbone, and Head. The entire model first pre-processes the input image, aligns it into an image of 640*640*3 size and inputs it into the Backbone network. According to the three outputs in the Backbone network, the Head layer performs feature fusion and generates three feature maps of different sizes through Rep and Conv. After generating the anchor box in the image, the best anchor box is selected through the loss function to form the final detection result.

[0071] S2: The GSConv module is used to lightweight the YOLOv7 multi-vehicle detection algorithm model, and the VOVGSCSP module is introduced into the Head network of the YOLOv7 multi-vehicle detection algorithm to improve the detection accuracy; the details are as follows:

[0072] Reference Figure 3-4GSConv is a convolution operation in a neural network that combines the concepts of group convolution and channel shuffling. Compared with ordinary convolution operations, group convolution has a higher degree of parameter sharing. Group convolution promotes the extraction of local features and increases the diversity of features by limiting each convolution kernel to operate only with channels in the same group. By adjusting the number of groups, the network capacity can be effectively controlled, the number of parameters can be reduced, and the model complexity can be reduced, while reducing the risk of overfitting and improving computational efficiency. Channel shuffling promotes information exchange and fusion between different channels, enhances feature interaction capabilities, and improves the expressiveness of the model, so that the model can learn more comprehensive and diverse feature representations, which plays a positive role in model performance.

[0073] Taking YOLOv7 as the baseline model, GSConv is first used to improve the model. In order to verify the lightweight effect after adding GSConv and the change in accuracy after the model is lightweight, and to explore the best improvement position, three solutions are designed in YOLOv7: Backbone only, Head only, and GSConv is used throughout the whole model. The experimental results are shown in Table 1:

[0074] Table 1 Experimental results of introducing GSConv module

[0075]

[0076] The experimental results show that by replacing the original Conv with GSConv convolution in the baseline model, the model can be effectively lightweighted. Whether it is used in the backbone network, head network or through-model, the size of the model can be reduced, and the number of GFLOPs floating-point operations is reduced, and the model requires less memory when running. The above three sets of comparative experiments demonstrate the effectiveness of introducing GSConv for model lightweight improvement. Among them, the through-model achieved the best lightweight results, and the model size was reduced by 29.4%, but it was also inevitable that the model made some sacrifices in detection accuracy, with a decrease of 1.3% in detection accuracy, 0.4% in recall rate, and 0.8% in mAP@.5. When introduced separately in the Backbone and Head networks, the performance of the two in indicators such as detection accuracy is not much different, and both have decreased compared to the baseline. In terms of lightweight effect, the model size is reduced by 12% and 17.4% respectively.

[0077] In the detection-based vehicle tracking algorithm, the accuracy of the detector is of great significance to the accuracy of tracking. After the model is lightweight, the detection accuracy will be more or less negatively affected. Therefore, in the embodiment, the VOVGSCSP module is introduced on the basis of the lightweight model to improve the detection accuracy.

[0078] The YOLOv7 algorithm innovatively proposed the ELAN (Efficient Layer Aggregation Network) module, and proposed an improved E-ELAN (Extended-ELAN) structure in the subsequent enhanced version. The VOVGSCSP module based on GSConv has a simpler overall structure than E-ELAN, but it is better than the ELAN structure in feature extraction. The cross-stage local network VOVGSCSP designed based on the aggregation algorithm can learn features at multiple levels, thereby enhancing the reuse of features. VOVGSCSP contains the GS bottleneck module, in which the GSConv and Shortcut connections can reduce information loss while extracting features, and Shortcut has the advantages of alleviating gradient vanishing and accelerating training. Furthermore, since Shortcut implements point-by-point operations, it does not increase the depth of the network, but can improve the feature extraction performance of the VOVGSCSP network.

[0079] Based on the above analysis, the VOVGSCSP module is used to improve the overall model using the GSConv module for corresponding research. The experimental results are shown in Table 2:

[0080] Table 2 Experimental results of VOVGSCSP module based on GSConv

[0081]

[0082] The experimental results show that after using GSConv convolution, whether the VOVGSCSP module is introduced in the backbone network or the head network, the accuracy and mAP value of the model detection have been improved to a certain extent. Although the introduction in the Head leads to a large loss in the recall rate of the model, the overall performance is better. In terms of model lightweighting, the improved model is reduced by 14.3%, the FPS value is slightly improved, and the GFLOPs is reduced by 13.3. The computing resources required by the model are reduced. The overall improvement plan sacrifices part of the recall rate and has been effectively improved in terms of lightweight, accuracy, and real-time performance. After the introduction of VOVGSCSP in Backbone, the model size becomes larger than the baseline, the FPS value decreases, and the performance in lightweight and real-time performance is far inferior to the former.

[0083] S3: In the loss function part of the YOLOv7 multi-vehicle detection algorithm, Shape-iou combined with Soft-NMS is used to build an improved YOLOv7 multi-vehicle detection algorithm model; the details are as follows:

[0084] After making corresponding improvements to the network part of YOLOv7, a technical solution combining Shape-iou and Soft-NMS is proposed to solve the occlusion problem in detection. Compared with the CIOU (Complete-IoU) used by the baseline model, the Shape-iou loss function takes into account the information of the bounding box itself and can select objects more accurately. When the target is occluded, Soft-NMS can distinguish the detection box of the occluded object from the repeated detection box of the same object. For details, please refer to Figure 5 .

[0085] The pseudo code of Shape-iou can be referred to Figure 6 , where cw and ch are the total width and height of the overlapping frame, c2 is the square of its diagonal, and eps is a fixed value, which is set to 10 according to experience. -7 , the final Shape-iou score, that is, the loss function L Shape-iou The details are as follows:

[0086] L Shape-iou =1-IOU+distance shape +0.5*shape_cost

[0087] Among them, distance shape is the center distance of shape normalization; shape_cost is the shape cost; IOU means intersection-over-union score;

[0088]

[0089] Among them, B and B gt Represent the predicted box and the true bounding box respectively;

[0090] The scale value in Shape-iou is set to 0.13. The Shape-iou used in this embodiment calculates the shape cost by considering the width and height, and evaluates the shape similarity of the two bounding boxes. Finally, the final Shape-iou score is calculated by combining the intersection-over-union score, the normalized center distance, and the shape cost. This score not only reflects the degree of overlap between the bounding boxes, but also comprehensively considers their positional relationship and shape similarity, providing a new method for the comprehensive evaluation of the similarity between bounding boxes.

[0091] When the target is occluded, the Soft-NMS method uses a secondary processing of the threshold to distinguish the detection frame of the occluded object from the repeated detection frame of the same object.

[0092] Reference Figure 7In order to evaluate the impact of the improved scheme combining Shape-iou and Soft-NMS on the model training process, the changes in regression loss during training using Shape-iou and the original CIOU are compared. The curve closer to the coordinate axis is the convergence curve during training of the improved model. The results show that the proposed improved scheme accelerates the convergence speed of the model and has certain advantages in model training.

[0093] After integrating the above-mentioned improvement methods, compared with the baseline network, the improved model uses GSConv convolution for lightweight processing, and on this basis introduces the VOVGSCSP module. In the loss function part, the Shape-iou combined with Soft-NMS method is proposed. The overall model experimental results are shown in Table 3.

[0094] Table 3 Experimental results of the fused model

[0095]

[0096] The experimental results show that the improvement scheme proposed in this embodiment has a slight loss in recall rate, but other indicators have been improved to a certain extent. In terms of detection accuracy, the vehicle detection accuracy rate has increased by 1.7%, and mAP@.5 has increased by 0.8%. Although the overall improvement is not large, in terms of lightweight, the improved model is finally reduced by 18.3%, the FPS value is slightly improved, and the computing resources required for model operation are also reduced. The overall improvement scheme can meet the real-time requirements, and the detection accuracy can also adapt to most scenarios. The improvement scheme is effective.

[0097] Reference Figure 8 , based on the YOLOv7 multi-vehicle detection algorithm model and the improved YOLOv7 multi-vehicle detection algorithm model, the actual vehicle detection is performed, referring to Fig. 9 , Fig. 9 (a) shows the detection effect before the model is improved. The two cars are severely occluded. Only the back half of the first car is exposed. Due to the small number of available features, the detector has limited information learned after feature extraction, so it is mistakenly marked as a van; only the front half of the second car is visible. Despite the limited features, the detector still correctly identifies it as a car with a confidence level of 0.44. Fig. 9 (b) shows the detection effect after the model is improved. Under the same occlusion condition, the detection model improved by this embodiment can accurately identify the two vehicles and assign correct labels to both. At the same time, the recognition confidence is also improved.

[0098] S4: Build the Byte Track multi-vehicle tracking algorithm model; the details are as follows:

[0099] Byte Track relies on the Kalman filter for trajectory prediction and update. In the Kalman filter, the spatial state vector is a very important concept, which is used to represent the state of a dynamic system at any time, and contains all the variables required to define the current state of the dynamic system. When new observation data is available, the Kalman filter will combine the previous estimate and the new measurement data to update the state vector, making the estimate closer to the actual state.

[0100] The spatial state vector of the Kalman filter in the Byte Track multi-vehicle tracking algorithm model is (x, y, a, h, vx, vy, va, vh), where (x, y) is the coordinate of the center point of the target box, h is the box height, a is the aspect ratio of the target box, and vx, vy, va and vh are the first-order derivatives of the variables.

[0101] S5: Reconstruct and improve the spatial state vector of the Byte Track multi-vehicle tracking algorithm model to build an improved Byte Track multi-vehicle tracking algorithm model; the details are as follows:

[0102] The spatial state vector of the Kalman filter is reconstructed as (x, y, w, h, vx, vy, vw, vh), where w is the width of the target box and vw is the first-order derivative of the width variable.

[0103] The spatial state vector before the improvement used the aspect ratio solution, and after the improvement, the width and height of the target box solution was used. The two solutions were analyzed under different vehicle occlusion conditions:

[0104] (1) When the vehicle obstruction is not obvious

[0105] The dynamics of straight-moving vehicles in the video can be roughly divided into two categories: one is moving toward or away from the camera, and the other is approximately lateral movement within the frame. When the vehicle occlusion in the video is not obvious, as the tracked vehicle approaches or moves away, the predicted target box will change as the size of the vehicle moves, which means that the width and height of the target box will continue to change, while the aspect ratio may remain unchanged. In addition, different types of vehicles may have the same aspect ratio, but there will be differences in width and height. See Fig.10, comparing the two, if the width and height variables are used, the size change process of the target can be accurately reflected, and the size change is often continuous when there is no occlusion or slight occlusion. Such slow and continuous changes will be more accurate for the Kalman filter to update the state vector and predict the position of the target. The aspect ratio is the ratio of the width and height of the target frame, which reflects a proportional relationship. If the aspect ratio solution is used, when the target is far away or close, once the aspect ratio remains unchanged, the variable will almost remain zero or close to zero, which will not provide additional valuable information. When the Kalman filter updates the state and predicts the target position, the variables that are fixed for a long time will gradually be ignored, and the filter will lack corresponding support when predicting, and the filter's adaptive ability on this variable will also be reduced. Therefore, in the process of vehicle tracking, the width and height solution will provide more effective information.

[0106] (2) When the vehicle obstructs the view

[0107] When there are multiple vehicles in the picture at the same time, it is very easy for obscuration to occur due to the speed difference between the two vehicles or overtaking or lane changing. In this case, the slower vehicle will be blocked by the faster vehicle, and then retreat to the rear or even completely disappear from sight and finally reappear.

[0108] During the entire motion process, if the aspect ratio solution is used, if the value of the aspect ratio changes slowly, the filter can achieve relatively stable tracking; however, if the aspect ratio remains unchanged, it will gradually lead to the Kalman filter making the assumption that the aspect ratio remains unchanged. Under this assumption, once the vehicle is occluded, the aspect ratio of the detection frame of the occluded vehicle will change immediately, and the filter may not be able to adapt quickly and overturn the previous assumptions. If the entire system is not adjusted effectively in time, it may cause deviations in state estimation, and the system will make incorrect predictions and updates, which will affect the accuracy of tracking. On the contrary, if the width and height solution is used, the two variables will intuitively reflect the actual size of the target frame from beginning to end, and the filter will not make incorrect assumptions similar to fixed variables during the motion process. When occlusion occurs, it can more sensitively capture the changes in variables and adjust the state estimation in time according to the new observation data.

[0109] Combined with the above analysis under different vehicle occlusion conditions, the use of width and height spatial state vectors can capture the changes in the target frame more delicately and accurately, and can provide more accurate data when the Kalman filter is predicted and updated.

[0110] S6: Detect and track multiple vehicles based on the improved YOLOv7 multi-vehicle detection algorithm model and the improved Byte Track multi-vehicle tracking algorithm model. The details are as follows:

[0111] Reference Fig.11 The improved YOLOv7+improved Byte Track vehicle detection and tracking model is used to detect and track vehicles. During the detection process, the detection results are divided into high-confidence score detection frames and low-confidence score detection frames based on the detection threshold, and then the detection frames with high confidence scores and low confidence scores are used for the first data association and the second data association, respectively. Among them, the second data association is predicted and associated based on the result of the first data association, and both data associations are based on the Hungarian matching algorithm for data association; in the results of the two data associations, the matched vehicles are tracked, and the vehicle trajectory is updated based on the Kalman filter, and updated to the prediction result of the first data association. For the detection frame that is not matched in the first data association, a new trajectory is created, and for the trajectory that is not matched in the second data association, it is considered that the tracking is lost.

[0112] Reference Fig.12 When a vehicle is blocked, only a small part of the vehicle body can be seen. At this time, the detector alone cannot accurately determine the specific position of the vehicle at this time, including the width and height of the bounding box, which may deviate from the actual situation. At this time, detection and tracking work together. In the previous video frame, Fig.12 (a), the detector can frame the entire vehicle before occlusion occurs. When occlusion occurs, the Kalman filter in the tracker will predict the possible location of the vehicle in subsequent frames based on the previous state. The width and height scheme is used in the Kalman filter spatial state vector, such as Fig.12 (b)~(d) When occlusion occurs, the width and height of the bounding box and the possible location of the vehicle can be predicted based on the change pattern of previous data. In the figure, even if the vehicle is severely occluded, a reasonable prediction is still made for the vehicle, and no ID conversion occurs during the whole process, which handles the tracking of the vehicle under occlusion very well.

[0113] Embodiment 2:

[0114] See also Figure 13-14 The present invention provides a detection-based multi-vehicle tracking system, comprising:

[0115] Data reading module 100: used to read image data, video files and camera real-time video streams, as well as vehicle detection or tracking modes selected by the user;

[0116] Vehicle detection module 200: used for detecting image data, video files and camera real-time video streams based on user selection using a detector built based on an improved YOLOv7 multi-vehicle detection algorithm model;

[0117] Vehicle tracking module 300: used for tracking vehicles according to user selection results by a tracker built based on the improved Byte Track multi-vehicle tracking algorithm model;

[0118] Tracking display module 400: used to visualize the detection or tracking results.

[0119] This embodiment integrates the improved YOLOv7 vehicle detection algorithm and the improved Byte Track vehicle tracking algorithm, and uses the Streamlit library and OpenCV to develop a vehicle detection and tracking system that can be applied to urban road traffic. The system has good interactive functions and is more convenient to deploy overall.

[0120] The Streamlit library provides a visual Python framework for machine learning applications or data analysis, and supports generating a very rich front-end interactive interface by writing simple code. Streamlit is widely used in scenarios such as data analysis and intelligent AI, and is also very efficient, simple, and practical in the development of personal application scenarios.

[0121] As a cross-platform open source library in the field of computer vision, OpenCV can run on multiple operating systems such as Linux and Windows. The library contains a large number of function libraries for image processing and video analysis, which can be easily called by other programming languages. In addition, OpenCV also provides interfaces that are easy to use in Java, Python, etc., and integrated with deep learning frameworks (such as TensorFlow, PyTorch, etc.), and provides some pre-trained deep learning models that can quickly implement some image processing and recognition tasks, making the entire development process easier to use.

[0122] The front-end operation page of this system provides the interactive functions required by users, including parameter settings and image source settings. Users can select the images to be detected or the videos to be tracked according to the actual usage scenario. In addition, the system also supports users to call the camera for real-time vehicle detection and tracking.

[0123] Please refer to Fig.14The system operation flow chart is shown in Figure 1. When the system is running, you can enter the system page by entering the project address in the browser. The entire system is divided into two modules: vehicle detection module 200 and vehicle tracking module 300. Different functions are provided according to different user input data. When the user chooses to input image data, the entire system will only call the vehicle detection module 200, and finally feedback the detector operation results to the web page; when the user chooses video data or calls the camera to input real-time video stream, the system will provide two options: detection and tracking. When the detection function is selected, only the objects detected in each frame of the image will be fed back. When tracking is selected, each target will be assigned an ID for real-time tracking.

[0124] Embodiment 3:

[0125] (1) Data preparation:

[0126] Collect and annotate vehicle image datasets for training and testing. In this embodiment, the public dataset UA-DETRAC dataset is used. Data enhancement processing can be used to improve the robustness of the model.

[0127] (2) Improve YOLOv7 model training:

[0128] Train the YOLOv7 model using the prepared dataset.

[0129] Adjust the network structure and hyperparameters to suit the vehicle detection task.

[0130] Use cross-validation techniques to optimize model generalization ability.

[0131] The data set preprocessing results can be found in Fig.15 .

[0132] The Image and Labels folders store the corresponding data according to the training set, validation set, and test set, while the three text files train.txt, test.txt, and val.txt store the storage paths of all training set images, test set images, and validation set images respectively.

[0133] (3) Improved Byte Track algorithm integration:

[0134] The overall model is divided into a vehicle detection module and a vehicle tracking module. In the detection part, when data is input, the data will be preprocessed first. By scaling and normalizing, the image will be processed into a size of 640*640; the improved YOLOv7 will receive the processed image and identify the position and size of the vehicle, including the coordinates of the bounding box and the width and height of the box. In the tracking part, the tracker will be initialized before tracking begins. Then, for each frame, the detection results will be associated with the corresponding data according to the confidence score. After completing the data association using the Hungarian matching algorithm, the tracker will update the Kalman filter according to the matching tracking results. The specific process of the integrated detection and tracking model can be referred to Fig.11 .

[0135] (4) System integration test:

[0136] The Streamlit library and OpenCV tools are used in PyCharm software to design visual pages, and the written network model is embedded into the page control in the form of slot functions to build the system. The entire system supports web-based operation and supports mobile access in the same local area network.

[0137] After entering the system, the initial page of the web page is as follows Fig.16 As shown, the far left is the user operation area, and the right side shows the sample image. The left image in the sample image is the input data, and the right image is the system detection result display. The target category and confidence level will be displayed in the upper left corner of the target box. When the detection function is run, the final result is similar to the sample image. When the user selects video data or calls the camera, the system page will not display the input data, but will feedback the results of the system operation. If detection is selected, the label category and confidence level will be indicated in the upper left corner of each target box in the video. When the tracking function is selected, the target box will be additionally marked with ID information. The results of the web page operation can be referred to Fig.17 .

[0138] Reference Fig.18 ,The page displayed on the mobile terminal is similar to the local display result: the sample image is also displayed when entering the system for the first time. For the convenience of display, the two mobile phone screenshots are spliced.

[0139] Reference Fig.19 The figure shows the vehicle detection effect on the mobile terminal. The rest of the functions are the same as the web page. Only the detector will be called when running the detection task. Only when the user inputs video data or calls the camera and selects the tracking task will the target be assigned an ID and tracked in real time.

[0140] The above is only a preferred specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention. In addition, the technical solution and inventive concept of the present invention can also be applied to the detection and tracking of multiple targets such as herds of cattle and sheep.

Claims

1. A detection-based multi-vehicle tracking method, characterized in that: The steps include: S1: Build the YOLOv7 multi-vehicle detection algorithm model; S2: The GSConv module is used to perform lightweight processing on the YOLOv7 multi-vehicle detection algorithm model, and the VOVGSCSP module is introduced into the head network of the YOLOv7 multi-vehicle detection algorithm to improve the detection accuracy; S3: Shape-iou combined with Soft-NMS is used in the loss function of the YOLOv7 multi-vehicle detection algorithm to build an improved YOLOv7 multi-vehicle detection algorithm model; S4: Build the Byte Track multi-vehicle tracking algorithm model. For the occlusion problem in detection, the YOLOv7 multi-vehicle detection algorithm model is further improved by combining the Shape-iou and Soft-NMS methods. The Shape-iou calculates the shape cost by considering the width and height, and evaluates the shape similarity of the two bounding boxes. The Shape-iou score is calculated based on the following formula, that is, the loss function The details are as follows: in, is the shape normalized center distance; is the shape cost; It represents the intersection and union score; in, and Represent the predicted box and the true bounding box respectively; When the target is occluded, the Soft-NMS method distinguishes the detection frame of the occluded object from the repeated detection frame of the same object through secondary processing of the threshold; S5: Reconstruct the spatial state vector of the Byte Track multi-vehicle tracking algorithm model to build an improved ByteTrack multi-vehicle tracking algorithm model; the spatial state vector of the Kalman filter in the Byte Track multi-vehicle tracking algorithm model is ,in is the coordinate of the center point of the target frame, is the frame height, is the target box aspect ratio, , , and is the first-order derivative of the variable, and the spatial state vector of the Kalman filter is reconstructed as ,in is the target box width, is the first derivative of the width variable; S6: Detect and track multiple vehicles based on the improved YOLOv7 multi-vehicle detection algorithm model and the improved Byte Track multi-vehicle tracking algorithm model.

2. A detection-based multi-vehicle tracking method according to claim 1, characterized in that: In S2, the GSConv module uses group convolution and channel shuffling to perform lightweight processing of the model; the group convolution limits each convolution kernel to operate only with channels in the same group, and adjusts the number of restricted groups; the channel shuffling is used to promote information exchange and fusion between different channels.

3. A detection-based multi-vehicle tracking method according to claim 2, characterized in that: In S2, a cross-stage local network VOVGSCSP module designed based on an aggregation algorithm performs multi-level learning on features, and the VOVGSCSP module includes a GS bottleneck module, in which GSConv and Shortcut are connected, and the Shortcut implements point-by-point operation.

4. The detection-based multi-vehicle tracking method according to claim 1, characterized in that: In S6, the specific steps are as follows: S6.1: Collect video frames and perform target detection based on the improved YOLOv7 multi-vehicle detection algorithm model; S6.2: Based on the detection threshold, the detection results are divided into two categories: high confidence score detection boxes and low confidence score detection boxes; S6.3: First, the detection box with a high confidence score is first associated with the result predicted by the Kalman filter, the associated result is predicted based on the Kalman filter, and then the detection box with a low confidence score is used to perform a second data association on the remaining unmatched tracks; S6.4: Track the matched vehicles in the two association results, update the vehicle trajectory based on the Kalman filter, and update it to the prediction result before the first data association.

5. A detection-based multi-vehicle tracking method according to claim 4, characterized in that: The first data association and the second data association in S6.3 are both based on the Hungarian matching algorithm for data association; during the first data association process, a new track is created for the unmatched detection box, and during the second data association process, the tracking is considered lost for the second unmatched track.

6. A detection-based multi-vehicle tracking system applied by the method according to any one of claims 1 to 5, characterized in that: include: Data reading module (100): used to read image data, video files and camera real-time video streams as well as vehicle detection or tracking modes selected by the user; Vehicle detection module (200): used to detect image data, video files and camera real-time video streams based on user selection through a detector built based on an improved YOLOv7 multi-vehicle detection algorithm model; Vehicle tracking module (300): used to perform corresponding vehicle tracking according to the user selection result through a tracker built based on the improved Byte Track multi-vehicle tracking algorithm model; Tracking display module (400): used to visualize the detection or tracking results.

7. A detection-based multi-vehicle tracking system according to claim 6, characterized in that: In the system, only the vehicle detection function is provided for image data; and two functions, vehicle detection and vehicle tracking, are provided for video files and real-time camera video streams, wherein the vehicle detection function detects and feeds back each frame of image.

8. A detection-based multi-vehicle tracking system according to claim 6, characterized in that: In the vehicle tracking module (300), when tracking a vehicle, an ID is assigned to each target for real-time tracking.

Images