Highway pavement dynamic small target tracking detection method and system based on improved YOLOv5 and ByteTrack

Abstract

The invention discloses a road surface dynamic small target tracking detection method and system based on improved YOLOv5 and ByteTrack, and the method comprises the steps: decoding a video stream to obtain a current video frame, and carrying out the marking and desensitization of a target in the current video frame; carrying out clustering and normalization on the data set, and then carrying out enhancement processing and equalization processing on the current video frame; attention is performed on a target feature region of the data set by using the optimized YOLOv5 backbone network structure, and then parameter loss of the optimized YOLOv5 backbone network structure is calculated; and predicting a bounding box by using Kalman filtering through a ByteTrack algorithm, then performing matching between a target and a trajectory by using a Hungary algorithm, outputting a tracking image sequence, and completing dynamic target tracking. According to the invention, the problems of easy loss, error detection and the like of dynamic small target vehicles at a long distance of monitoring equipment are solved, and the traffic safety is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent traffic target detection, and in particular relates to a method and system for tracking and detecting dynamic small targets on a highway pavement based on improved YOLOv5 and ByteTrack. Background Art

[0002] With the development of computer vision technology and deep learning algorithms, object detection technology has been increasingly used in the field of intelligent transportation and has achieved remarkable results. However, there are still many challenges and problems in the field of small object detection on highways.

[0003] The biggest problem with small targets is that they are usually small in size, so the target detection algorithm has a limited range of perception for small targets. Moreover, small targets have fewer pixels, which weakens the ability to express features, causes serious loss of detail information, and reduces effective information, thus reducing detection accuracy. In addition, small targets on highways are usually in motion, and dynamic small target vehicles at long distances are prone to loss and misdetection. This is because small targets move at high speeds, are far away, and may be blocked, making it difficult for the detection algorithm to accurately capture the position and shape of small targets.

[0004] Therefore, in order to improve the detection accuracy of dynamic small targets on the road surface in the traffic anomaly event detection model, effective dynamic small target tracking and detection technology is urgently needed. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a method and system for dynamic small target tracking detection on a highway pavement based on improved YOLOv5 and ByteTrack. The YOLOv5 backbone network structure is improved by using an attention network, and then the Kalman filter and the Hungarian algorithm are used to match the small target trajectory, thereby solving the problems of easy loss and misdetection of dynamic small target vehicles at a long distance of the monitoring equipment, and improving traffic safety.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for tracking and detecting a small dynamic target on a road surface based on improved YOLOv5 and ByteTrack, comprising the following steps: The method for tracking and detecting a small dynamic target on a road surface based on improved YOLOv5 and ByteTrack is characterized in that it comprises the following steps:

[0008] Decode the video stream to be detected to obtain the current video frame image, and mark the small targets in the current video frame image;

[0009] Perform data enhancement and data equalization on the annotated images;

[0010] Based on the improved YOLOv5 backbone network, features of the image are extracted, and the improved YOLOv5 backbone network is trained using a generalized focal loss function; the improved YOLOv5 backbone network introduces an attention mechanism so that the model pays more attention to important feature areas of the image;

[0011] Based on the image that has been attended to, the bounding box is predicted using Kalman filtering to obtain the trajectory prediction box. At the same time, the high-score detection box and the low-score detection box are divided according to the degree of overlap between the target detection box of the current frame and the target detection box of the historical frame; the high-score detection box is matched with the trajectory prediction box for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the unmatched high-score detection frame waits for the second match; the low-score detection frame and the unmatched high-score detection frame are matched with the trajectory prediction frame for the second time. After the match is successful, the trajectory is updated to the current frame trajectory set;

[0012] The target trajectory set is output as a tracking image sequence to complete dynamic target tracking.

[0013] As a preferred technical solution, the small target in the current video frame image is marked as follows:

[0014] The position, shape, movement, and color features of the small target in the current video frame image are annotated. During the annotation process, the sensitive information in the image is desensitized.

[0015] As a preferred technical solution, the data enhancement and data equalization processing of the annotated image is specifically performed as follows:

[0016] (1) Clustering, i.e. clustering the data set according to the labeled size and coordinates;

[0017] (2) Normalization: According to the target distribution area, determine the size of the cropped image. According to the high-frequency target distribution area after clustering, randomly determine the center point, expand outward from the center point, and determine the cropping area; calculate the intersection over union (IoU) of all target boxes in the cropping area. 1 ; Count the consistency between the number of target boxes in a single cropped image and the number of target boxes in the original image. If the number is inconsistent, it is necessary to additionally calculate the number of target boxes based on the intersection over union (IoU). 1 Add a crop center point and reset the distance of the center point to a random area from the cropped area;

[0018] (3) Use random erasing to remove part of the image area; if erasing is not necessary according to random probability, jump to step (4); when the overlap between the generated mask area and the target frame is high, repeat step (3); when the overlap between the generated mask area and the target frame is low, remove the mask area and jump to step (4);

[0019] (4) Use color jittering to randomly adjust the image feature attributes of the dataset, including brightness, contrast, hue, and saturation;

[0020] (5) using improved Mosaic enhancement to fuse multiple images that have been clustered, normalized, and enhanced to form an enhanced feature map; the improved Mosaic does not include the CutOut part;

[0021] (6) Determine and compare the target box size in the enhanced feature map finally generated with the set threshold. If the target box size is smaller than the set threshold, repeat steps (1)-(5);

[0022] (7) Balancing processing, including resampling, downsampling, and dynamic adjustment of the category weights of the augmented data set.

[0023] As a preferred technical solution, an improved network structure network is provided, including an input terminal Input, an improved YOLOv5 backbone network Backbone, a network Neck and a prediction terminal Prediction;

[0024] The improved YOLOv5 backbone network includes a SE attention module, which is placed after the CBL and is used to focus on important feature areas.

[0025] As a preferred technical solution, the feature extraction of the image based on the improved YOLOv5 backbone network is specifically as follows:

[0026] Compress the feature map dimension of the data set and obtain the feature map channel information;

[0027] Perform convolution operation on the original feature map, that is, first use the Non-Local module to extract global context information, and then pass it through the convolution layer and activation function;

[0028] Multiply the obtained channel information and the feature information after the convolution operation to obtain a new feature map with the same dimension as the original one.

[0029] As a preferred technical solution, the generalized focal loss function is as follows:

[0030] GFL(P yl ,P yr )=-|y-(y l P yl +y r P yr )| β ((y r -y)log(P yl )+(yy l )log(Pyr ))

[0031] P yl >=0,P yr >=0,P yl +P yr =1

[0032]

[0033] Among them, y is the true label category, yl and yr are two variables estimated by the model, and their probability values ​​are P yl and P yr , represents the adjustment factor, and parameter β represents the hyperparameter.

[0034] As a preferred technical solution, the method of dividing the high-score detection frame and the low-score detection frame according to the degree of overlap between the target detection frame of the current frame and the target detection frame of the historical frame also includes the following steps:

[0035] Calculate the intersection over union (IoU) between the target detection box and the target detection box of the historical frame 2 value;

[0036] Set the confidence range and set IoU 2 The target detection box whose value exceeds the confidence range is classified as a high-score detection box; 2 The target detection box whose value falls within the set confidence range is classified as a low-score detection box; the IoU 2 Object detection boxes with values ​​smaller than the confidence range are removed.

[0037] As a preferred technical solution, the first matching is specifically as follows:

[0038] The high-score detection frame and trajectory prediction frame are matched using the Hungarian algorithm;

[0039] If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the unmatched trajectory is temporarily stored and waits for the second match, and the number of frames is dynamically adjusted to wait for the next frame match;

[0040] If a high-score detection frame fails to match multiple times, the high-score detection frame is re-identified as a new one, and a new track and a new target ID are assigned to the high-score detection frame;

[0041] The second matching is specifically:

[0042] The low-score detection frame, the unmatched high-score detection frame and the trajectory prediction frame are matched for the second time through the Hungarian algorithm;

[0043] If the match is successful, its trajectory is updated to the current frame trajectory set, otherwise the low-score detection frame is deleted;

[0044] After the second matching is completed, the intersection over union (IoU) ratio between the target detection frame of the current frame and the target detection frame of the historical frame is calculated. 2 The value is compared with the set threshold. If it is greater than the set threshold, the unmatched trajectory prediction box is deleted or merged into the trajectory collection and waits for the next frame to match.

[0045] As a preferred technical solution, an adaptive adjustment module is introduced in the first matching or the second matching process to capture the target details and edge information in motion, and adjust the parameters of the matching algorithm based on this information to improve the matching accuracy; at the same time, the Hungarian algorithm is used to find the optimal matching solution.

[0046] In a second aspect, the present invention further provides a road surface dynamic small target tracking and detection system based on improved YOLOv5 and ByteTrack, which is applied to the road surface dynamic small target tracking and detection method based on improved YOLOv5 and ByteTrack, including a preprocessing module, a feature enhancement module, an attention module, a trajectory matching module and a task execution module;

[0047] A preprocessing module is used to decode the video stream to be detected to obtain the current video frame image and mark the small targets in the current video frame image;

[0048] The feature enhancement module is used to cluster and normalize the data set, and then perform enhancement and equalization processing on the current video frame;

[0049] The attention module is used to improve the YOLOv5 backbone network structure with the attention network to obtain an optimized YOLOv5 backbone network structure; the optimized YOLOv5 backbone network structure is used to pay attention to the target feature area of ​​the data set, and then the parameter loss of the optimized YOLOv5 backbone network structure is calculated;

[0050] The trajectory matching module is used to predict the bounding box based on the image after attention using Kalman filtering to obtain the trajectory prediction box, and at the same time divide the high-score detection box and the low-score detection box according to the overlap between the target detection box of the current frame and the target detection box of the historical frame; the high-score detection box is matched with the trajectory prediction box for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the high-score detection frame that has not been matched successfully waits for the second match; the low-score detection frame and the high-score detection frame that have not been matched successfully are matched with the trajectory prediction frame for the second time, and the trajectory is updated to the current frame trajectory set after the match is successful;

[0051] The task execution module is used to output the target trajectory set into a tracking image sequence to complete dynamic target tracking.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] (1) The present invention introduces the attention mechanism SENet into the backbone network of YOLOv5, so that the model can better extract image features and improve the detection accuracy of small targets. At the same time, the ByteTrack algorithm is used to track the moving small targets, which can accurately restore the target motion trajectory and further improve the accuracy of target detection.

[0054] (2) The present invention increases the diversity and richness of data through image enhancement technology, and improves the robustness of the model to changes such as occlusion and illumination. At the same time, the improved Mosaic enhancement technology is used to remove the CutOut part in the original Mosaic enhancement, and four pictures are selected for Mixup fusion, which further enhances the robustness of the model to image transformation.

[0055] (3) The present invention achieves sample number balance through methods such as data set resampling, downsampling and dynamic adjustment of category weights, so that the model pays more attention to categories with a small number, improves the accuracy of target detection and reduces the false detection rate.

[0056] (4) The present invention analyzes the motion characteristics of the target in continuous frames in the ByteTrack algorithm and dynamically adjusts the number of frames, so that the model can track the target more efficiently. At the same time, an adaptive adjustment module is used to capture the details and edge information of the target in motion, further improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a flow chart of a method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to an embodiment of the present invention;

[0059] Figure 2 Schematic diagram of the data enhancement process according to an embodiment of the present invention;

[0060] Figure 3 A structural diagram of the improved YOLOv5 backbone network according to an embodiment of the present invention;

[0061] Figure 4 The present invention is a schematic diagram of the structure of a dynamic small target tracking and detection system on a highway surface based on improved YOLOv5 and ByteTrack. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0063] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0064] See also Figure 1 The present embodiment provides a method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack, comprising the following steps:

[0065] S1. Decode the video stream to be detected to obtain the current video frame image, and mark the small targets in the current video frame image;

[0066] S11, obtaining a video stream to be detected, decoding the video stream to be detected to obtain a current video frame image, wherein the current video frame image includes a small moving target.

[0067] Specifically, the OpenCV library is used to decode the video stream and obtain the current video frame image. During the decoding process, the spatial and temporal correlation of adjacent images is used to effectively remove redundant information, and motion compensation technology is used for inter-frame prediction. Discrete cosine / sine transform and other methods are used for encoding in areas where video images change slowly. At the same time, deblocking filtering and pixel adaptive compensation technology are used to reduce the impact of distortion behaviors such as block effect, ringing effect, and color deviation on video quality, so as to achieve efficient decoding of video information.

[0068] S12. Video data annotation is to locate and track objects in a series of images in frames, and to annotate the position, shape, movement, color and other related features of small targets in the images.

[0069] The position, shape, movement, color and other related features of small targets in the image are annotated. It provides a large amount of data for tracking algorithms to use, thereby realizing the detection, tracking, and identification of active targets in the scene, as well as further behavior analysis and event detection, playing a very important bridging role between vision and text. The annotation process desensitizes sensitive information involved in videos and pictures, such as camera guns, shooting locations, timestamps, license plates, etc., to ensure data security and privacy.

[0070] S2. Perform data enhancement and data balancing on the annotated images. Among them, feature enhancement is performed on the images to increase the diversity and richness of the data. Please refer to Figure 2 , including the following steps:

[0071] S21, clustering processing, determine the size of the cropped image and the cropped area, and randomly determine the center point. Specifically, the height and aspect ratio of the annotated small target annotation box and the coordinate information data are input into the GMM as a sample set for clustering processing, and the data set is divided into three categories: Large, Medium and Small according to the aspect ratio.

[0072] S22, normalization processing, according to the distribution area of ​​small targets, determine the size of the cropped image and the approximate area of ​​cropping; according to the distribution area of ​​high-frequency small targets after clustering, randomly determine the center point, and then expand it to the left, right, top and bottom respectively, so as to determine the basic cropping area of ​​640*640, and calculate the intersection and union ratio (IoU) of all target boxes in the cropping area 1 , as follows:

[0073]

[0074] Among them, area represents the area of ​​the box, ∩ represents the intersection of two boxes, and ∪ represents the union of two boxes. 1 The minimum value is smaller than the set cropping area of ​​640*640, indicating that part of the target is cut off outside the cropping area. This method destroys the characteristics of the original image and the cropping area needs to be re-established. The number of repetitions is 50.

[0075] Count whether the number of target boxes in a single cropped image is consistent with the number of target boxes in the original image. If not, it is necessary to add an additional cropping center point and reset the distance of the center point to a random area (640*640) from the cropped area. At this time, the center point does not need to be set to a high-frequency area; obtain the image size that meets the YOLO v5 network input.

[0076] S23, random erasing, random erasing is used to remove some areas in the image to enhance the robustness of the model to occlusion transformation and prevent overfitting.

[0077] Specifically, a random probability of 0.5 is used to determine whether the image needs to be erased. If not, jump to step (4); for all target frames of a single image, a mask area Mask is randomly generated. The generated area is limited to the high-frequency area after clustering in the image cropping step, and is filled with the image mean; two parameters of scale and aspect ratio are randomly generated to determine the width w and height h of the occluded area, but the width and height of the occluded area must be smaller than the width and height of the original image; the upper left corner of the original image is taken as the vertex, and the difference between the width and height of the occluded area of ​​the original image is taken as the width and height to obtain a rectangle M. A point is randomly generated in M ​​as the upper left corner point of the occluded area, and the width w and height h of the occluded area are combined with the above determination to obtain the mask Mask area; the IoS of all target frames and occluded areas is calculated. If the minimum value is greater than D, D is a pre-set threshold, then the above step (3) is repeated (except for the process of determining erasure), and the maximum number of repetitions is 50 times; if the minimum value is still greater than D, the mask area is eliminated and the next step of calculation is performed.

[0078]

[0079] Among them, C represents the randomly generated mask area Mask, and G represents the target area.

[0080] S24, color jitter, i.e., adjusting the brightness, contrast, hue and saturation of the image to enhance the robustness of the model to changes in lighting, etc. Specifically, the following steps are included:

[0081] Use a random probability of 0.5 to determine whether the brightness and contrast of the image need to be adjusted. If necessary, randomly generate brightness parameters and randomly generate contrast parameters;

[0082] Use the random function to generate random probability to determine whether the hue and saturation of the image need to be adjusted. If necessary, first convert the image from RGB space to HSV space, randomly generate hue parameters from HSV space, randomly generate saturation parameters, and finally generate RGB space.

[0083] S25, improve the Mosaic enhancement, remove the CutOut part in the original Mosaic enhancement, and Mixup the partial areas of multiple pictures into one picture. Specifically, select four pictures obtained in the above steps, Mixup the 1 / 4 area of ​​each picture, and finally combine the four pictures into one picture.

[0084] It should be noted that the function of the CutOut part in the Mosaic enhancement module is to randomly cut off some areas of the sample and fill them with 0 pixel values, thereby increasing the diversity of the data and the generalization ability of the model. However, when CutOut is running, it will randomly block certain areas of the image. During model training, learning these blocked areas may introduce some noise and unnecessary patterns, causing the model to overfit the noise and outliers in the training data.

[0085] Compared with the CutOut part, Mixup can improve the generalization ability of the model, reduce the memory of wrong labels, increase the robustness of adversarial samples, and stabilize the training process. On the one hand, Mixup can train a neural network on a convex combination of paired samples and their labels, and generate new training samples by fusing two or more samples and their labels, which can increase the diversity of data and enable the model to learn more features and patterns between different samples and reduce the memory of wrong labels during the training process; on the other hand, since Mixup fusion increases the diversity of data during the training process and makes the model more robust to various input changes, when the model faces malicious disturbances or attacks, since it has been exposed to various input changes during the training process, the model can better resist these attacks and maintain good performance, thereby improving the model's resistance to malicious disturbances or attacks. Therefore, this embodiment will use Mixup to replace the CutOut part, thereby enhancing the enhancement effect of image features.

[0086] S26, judging whether the size of the target frame contained in the finally generated picture is smaller than a specified threshold, if so, repeating steps S21-S25, otherwise, performing a separate operation on the picture. Specifically, the specified threshold may be 1% of the picture size.

[0087] S27 balancing processing includes resampling, downsampling and dynamic adjustment of the weight of the category of the augmented data set, specifically, including the following steps:

[0088] S271, resampling, that is, increasing the number of categories with fewer numbers by repeated sampling or synthesizing new samples to achieve sample balance;

[0089] S272, downsampling, that is, by randomly deleting some samples, the number of samples of the categories with a large number is balanced;

[0090] S273. Dynamically adjust the category weights of the data set, that is, during training, balance the number of samples in the data set by adjusting the weights of different categories, and assign lower weights to categories with a larger number so that the model pays more attention to categories with a smaller number.

[0091] In this embodiment, the improved YOLOv5 backbone network is included in an improved network structure, such as Figure 3 As shown. The improved network structure includes an input terminal Input, an improved YOLOv5 backbone network Backbone, a network Neck and a prediction terminal Prediction. Further, the improved YOLOv5 backbone network Backbone is as step S3.

[0092] S3. Feature extraction is performed on the image based on the improved YOLOv5 backbone network, and the improved YOLOv5 backbone network is trained using the generalized focal loss function; the improved YOLOv5 backbone network introduces an attention mechanism to make the model pay more attention to important feature areas of the image; in the YOLOv5 backbone network structure, this embodiment embeds 3 SENets, which are respectively used before the upsampled network layer, so that the model pays more attention to important feature areas of the image.

[0093] In this embodiment, the improved YOLOv5 backbone network structure is mainly used to pay attention to the target feature area of ​​the data set, specifically, including the following steps:

[0094] S31, compress the width and height dimensions of the feature map. In order to ensure the consistency of the number of channels during the compression process, the number of channels of the feature map is not increased or decreased. Assuming that the input feature map is represented by P, the compression process can be expressed by the following formula. After compression, the feature map channel information is obtained as follows:

[0095]

[0096] Among them, F sq (·) indicates the compression operation performed on the feature map, P(i,j): represents the channel information of the (i,j) position of the feature map matrix P, W represents the width of the feature map, and H represents the height of the feature map.

[0097] S32, perform convolution operation on the original feature map. First, through the Non-Local module, in order to simplify the calculation, extract the global context information, through the convolution layer with a convolution kernel size of 1*1, and add an activation function during the convolution process;

[0098] S33. Multiply the obtained channel information and the feature information after the convolution operation to obtain a new feature map with the same dimension as the original one.

[0099] In order to further optimize the accuracy of model training and target detection, this embodiment calculates the generalized focal loss, namely the Generalized Focal Loss function (GFL), as shown in the following formula:

[0100] GFL(P yl ,P yr )=-y-(yl P yl +y r P yr ) β ((y r -y)log(P yl )+(yy l )log(P yr ))

[0101] P yl >=0,P yr >=0,P yl +P yr =1

[0102]

[0103] Among them, y is the true label category, yl and yr are two variables estimated by the model, and their probability values ​​are P yl and P yr , using absolute distance As an adjustment factor, the parameter β is set to 1.5 to allow the model to achieve higher target detection accuracy.

[0104] S4. Based on the image after attention, a Kalman filter is used to predict the bounding box to obtain a trajectory prediction box. At the same time, a high-score detection box and a low-score detection box are divided according to the degree of overlap between the target detection box of the current frame and the target detection box of the historical frame; the high-score detection box is matched with the trajectory prediction box for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the unmatched high-score detection frame waits for the second match; the low-score detection frame and the unmatched high-score detection frame are matched with the trajectory prediction box for the second time. After the match is successful, the trajectory is updated to the current frame trajectory set;

[0105] In step S3, this embodiment uses the ByteTrack algorithm to analyze the motion characteristics of the target in continuous frames and dynamically adjusts the number of frames so that the model can track the target more efficiently. At the same time, in order to further improve the accuracy of target detection, this embodiment also uses an adaptive adjustment module to capture the details and edge information of the target in motion.

[0106] The so-called ByteTrack algorithm is a tracking algorithm based on target detection. It uses Kalman filtering to predict bounding boxes, and then uses the Hungarian algorithm to match targets and tracks. The step "matching targets and tracks using the Hungarian algorithm" is performed by calculating the IoU value between the target detection box based on the current frame and the target detection box in the historical frame, and using the Hungarian algorithm to associate them.

[0107] Specifically, see Figure 1 In this embodiment, the confidence range is first set as the threshold range of the intersection-and-union ratio between the target detection frame of the current frame and the target detection frame of the historical frame, so as to control the matching degree between the prediction result of the model and the real target, thereby affecting the performance and accuracy of the model. Then, the target trajectory is associated using the Hungarian algorithm to achieve tracking of the same moving target in different frames in the video, including the following steps:

[0108] S41. Calculate the IoU between the target detection frame based on the current frame and the target detection frame based on the historical frame 2 Value, set thresholds E and F, greater than or equal to E is divided into high-score detection frames, greater than or equal to F and less than E is divided into low-score detection frames, and less than F is discarded.

[0109] S42, if IoU 2 If the value is higher than the threshold E of the high-score detection frame, it will be matched with the target trajectory frame predicted by the Kalman filter through the Hungarian algorithm for the first time; if the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set; if the match is not successful, the unmatched trajectory will be temporarily stored and wait for the second match, and the number of frames will be dynamically adjusted to wait for the next frame to match;

[0110] S43, if IoU 2 If the value is lower than the threshold E of the high-score detection frame and greater than the threshold F of the low-score detection frame, it will be matched with the temporarily stored unmatched trajectory; if the match is successful, its trajectory will be updated to the current frame trajectory set; if the match is still unsuccessful, the low-score detection frame will be deleted, and it will be determined whether the score of the unmatched trajectory is higher than the tracking threshold, and it will be deleted or merged into the trajectory collection and wait for the next frame to match;

[0111] S44. If the final predicted target detection box still fails to match successfully, it is considered to be newly appeared, and a new track and a new target ID are assigned to it.

[0112] In addition, in the first or second matching involved in steps S31-S34, an adaptive adjustment module is introduced to capture the target details and edge information in motion, and adjust the parameters of the matching algorithm based on this information to improve the matching accuracy. At the same time, the Hungarian algorithm is used to find the optimal matching solution to minimize the possibility of mismatching.

[0113] S5. Output the target trajectory set as a tracking image sequence to complete dynamic target tracking.

[0114] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0115] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0116] Based on the same idea as the road surface dynamic small target tracking detection method based on improved YOLOv5 and ByteTrack in the above-mentioned embodiment, the present invention also provides a road surface dynamic small target tracking detection system based on improved YOLOv5 and ByteTrack, which can be used to execute the above-mentioned road surface dynamic small target tracking detection method based on improved YOLOv5 and ByteTrack. For ease of explanation, in the structural schematic diagram of the embodiment of the road surface dynamic small target tracking detection system based on improved YOLOv5 and ByteTrack, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than the illustrated ones, or combine certain components, or arrange the components differently.

[0117] See also Figure 4 In another embodiment of the present application, a highway pavement dynamic small target tracking and detection system 10 based on improved YOLOv5 and ByteTrack is provided, the system comprising a preprocessing module 11, a feature enhancement module 12, an attention module 13, a trajectory matching module 14 and a task execution module 15;

[0118] The preprocessing module 11 is used to decode the video stream to be detected to obtain a data set of the current video frame, and then mark the target in the current video frame;

[0119] A feature enhancement module 12 is used to perform data enhancement and data equalization processing on the annotated image;

[0120] The attention module 13 is used to improve the YOLOv5 backbone network structure by using the attention network to obtain an optimized YOLOv5 backbone network structure; the optimized YOLOv5 backbone network structure is used to pay attention to the target feature area of ​​the data set, and then the parameter loss of the optimized YOLOv5 backbone network structure is calculated;

[0121] The trajectory matching module 14 is used to predict the bounding box using Kalman filtering based on the image after attention, obtain the trajectory prediction box, and divide the high-score detection box and the low-score detection box according to the overlap degree between the target detection box of the current frame and the target detection box of the historical frame; the high-score detection box is matched with the trajectory prediction box for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the high-score detection frame that has not been matched successfully waits for the second match; the low-score detection frame and the high-score detection frame that have not been matched successfully are matched with the trajectory prediction box for the second time, and the trajectory is updated to the current frame trajectory set after the match is successful;

[0122] The task execution module 15 is used to output the target trajectory set as a tracking image sequence to complete dynamic target tracking.

[0123] It should be noted that the highway pavement dynamic small target tracking and detection system based on improved YOLOv5 and ByteTrack of the present invention corresponds one to one with the highway pavement dynamic small target tracking and detection method based on improved YOLOv5 and ByteTrack of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the highway pavement dynamic small target tracking and detection method based on improved YOLOv5 and ByteTrack are applicable to the embodiment of the highway pavement dynamic small target tracking and detection method based on improved YOLOv5 and ByteTrack. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. It is hereby declared.

[0124] In addition, in the implementation of the highway pavement dynamic small target tracking and detection system based on improved YOLOv5 and ByteTrack in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the highway pavement dynamic small target tracking and detection system based on improved YOLOv5 and ByteTrack is divided into different program modules to complete all or part of the functions described above.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A road surface dynamic small target tracking and detection method based on improved YOLOv5 and ByteTrack, characterized in that: The steps include: Decode the video stream to be detected to obtain the current video frame image, and mark the small targets in the current video frame image; Perform data enhancement and data equalization on the annotated images; Based on the improved YOLOv5 backbone network, features of the image are extracted, and the improved YOLOv5 backbone network is trained using a generalized focal loss function; the improved YOLOv5 backbone network introduces an attention mechanism so that the model pays more attention to important feature areas of the image; Based on the image that has been attended to, the bounding box is predicted using Kalman filtering to obtain the trajectory prediction box. At the same time, the high-score detection box and the low-score detection box are divided according to the degree of overlap between the target detection box of the current frame and the target detection box of the historical frame; the high-score detection box is matched with the trajectory prediction box for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the unmatched high-score detection frame waits for the second match; the low-score detection frame and the unmatched high-score detection frame are matched with the trajectory prediction frame for the second time. After the match is successful, the trajectory is updated to the current frame trajectory set; The target trajectory set is output as a tracking image sequence to complete dynamic target tracking.

2. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 1, characterized in that: The small target in the current video frame image is marked as follows: The position, shape, movement, and color features of the small target in the current video frame image are annotated. During the annotation process, the sensitive information in the image is desensitized.

3. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 1, characterized in that: The data enhancement and data balancing processing of the labeled image is specifically as follows: (1) Clustering, i.e. clustering the data set according to the labeled size and coordinates; (2) Normalization: According to the target distribution area, determine the size of the cropped image. According to the high-frequency target distribution area after clustering, randomly determine the center point, expand outward from the center point, and determine the cropping area. Calculate the intersection-over-union (IoU1) of all target boxes in the cropped area; count the consistency of the number of target boxes in a single cropped image with the number of target boxes in the original image. If the number is inconsistent, it is necessary to add a cropping center point based on the intersection-over-union (IoU1) and reset the distance of the center point to a random area from the cropped area; (3) Use random erasing to remove part of the image area; if erasing is not necessary according to random probability, jump to step (4); when the overlap between the generated mask area and the target frame is high, repeat step (3); when the overlap between the generated mask area and the target frame is low, remove the mask area and jump to step (4); (4) Use color jittering to randomly adjust the image feature attributes of the dataset, including brightness, contrast, hue, and saturation; (5) Use improved Mosaic enhancement to fuse multiple images that have been clustered, normalized, and enhanced to form an enhanced feature map; The improved Mosaic does not include the CutOut part; (6) Determine the target box size in the enhanced feature map finally generated and compare it with the set threshold. If the target box size is smaller than the set threshold, repeat steps (1)-(5); (7) Balancing processing, including resampling, downsampling, and dynamic adjustment of the category weights of the augmented data set.

4. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 1, characterized in that: Including an improved network structure network, including the input end Input, the improved YOLOv5 backbone network Backbone, the network Neck and the prediction end Prediction; The improved YOLOv5 backbone network includes a SE attention module, which is placed after the CBL and is used to focus on important feature areas.

5. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 4 is characterized in that: The feature extraction of the image based on the improved YOLOv5 backbone network is specifically as follows: Compress the feature map dimension of the data set and obtain the feature map channel information; Perform convolution operation on the original feature map, that is, first use the Non-Local module to extract global context information, and then pass it through the convolution layer and activation function; Multiply the obtained channel information and the feature information after the convolution operation to obtain a new feature map with the same dimension as the original one.

6. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 1, characterized in that: The generalized focal loss function is as follows: GFL(P yl ,P yr )=-and-(and l P yl +y r P yr ) β ((and r -y)log(P yl )+(yy l )log(P yr )) P yl >=0,P yr >=0,P yl +P yr =1 Among them, y is the true label category, yl and yr are two variables estimated by the model, and their probability values ​​are P yl and P yr , represents the adjustment factor, and parameter β represents the hyperparameter.

7. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 1, characterized in that: The method further comprises the following steps: dividing the high-score detection frame and the low-score detection frame according to the overlap degree between the target detection frame of the current frame and the target detection frame of the historical frame. Calculate the intersection over union (IoU2) value between the target detection frame and the target detection frame of the historical frame; Set a confidence range, classify the target detection frame whose IoU2 value exceeds the confidence range as a high-score detection frame; classify the target detection frame whose IoU2 value falls within the set confidence range as a low-score detection frame; remove the target detection frame whose IoU2 value is less than the confidence range.

8. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 1, characterized in that: The first match is specifically: Match the high-score detection frame and trajectory prediction frame through the Hungarian algorithm; If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the unmatched trajectory is temporarily stored and waits for the second match, and the number of frames is dynamically adjusted to wait for the next frame match; If a high-score detection frame fails to match multiple times, the high-score detection frame is re-identified as a new one, and a new track and a new target ID are assigned to the high-score detection frame; The second matching is specifically: The low-score detection frame, the unmatched high-score detection frame and the trajectory prediction frame are matched for the second time through the Hungarian algorithm; If the match is successful, its trajectory is updated to the current frame trajectory set, otherwise the low-score detection frame is deleted; After the second matching is completed, the intersection-over-union (IoU2) value between the target detection frame of the current frame and the target detection frame of the historical frame is compared with the set threshold. If it is greater than the set threshold, the unmatched trajectory prediction frame is deleted or merged into the trajectory collection and waits for the next frame matching.

9. The method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack according to claim 1, characterized in that: During the first matching or the second matching process, an adaptive adjustment module is introduced to capture the target details and edge information in motion, and the parameters of the matching algorithm are adjusted according to this information to improve the matching accuracy; at the same time, the Hungarian algorithm is used to find the optimal matching solution.

10. A road surface dynamic small target tracking and detection system based on improved YOLOv5 and ByteTrack, characterized in that: A method for tracking and detecting dynamic small targets on a road surface based on improved YOLOv5 and ByteTrack, applied to any one of claims 1 to 9, comprising a preprocessing module, a feature enhancement module, an attention module, a trajectory matching module and a task execution module; A preprocessing module is used to decode the video stream to be detected to obtain the current video frame image and mark the small targets in the current video frame image; The feature enhancement module is used to cluster and normalize the data set, and then perform enhancement and equalization processing on the current video frame; The attention module is used to improve the YOLOv5 backbone network structure with the attention network to obtain an optimized YOLOv5 backbone network structure; the optimized YOLOv5 backbone network structure is used to pay attention to the target feature area of ​​the data set, and then the parameter loss of the optimized YOLOv5 backbone network structure is calculated; The trajectory matching module is used to predict the bounding box based on the image after attention using Kalman filtering to obtain the trajectory prediction box, and at the same time divide the high-score detection box and the low-score detection box according to the overlap between the target detection box of the current frame and the target detection box of the historical frame; the high-score detection box is matched with the trajectory prediction box for the first time. If the match is successful, it means that the target is tracked successfully, and its trajectory is updated to the current frame trajectory set. Otherwise, the high-score detection frame that has not been matched successfully waits for the second match; the low-score detection frame and the high-score detection frame that have not been matched successfully are matched with the trajectory prediction frame for the second time, and the trajectory is updated to the current frame trajectory set after the match is successful; The task execution module is used to output the target trajectory set into a tracking image sequence to complete dynamic target tracking.

Images