Monocular distance measurement method based on YOLO and abnormal jump filter
By introducing RedeCa module and Bot-Sort tracker in YOLOv5 network and using an abnormal jump filter of Kalman filter algorithm, the accuracy and robustness of monocular ranging in complex scenarios are solved, and the problem of Efficient target detection and distance estimation in the field of autonomous driving.
Patent Information
- Application Number
- CN202410952996.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing monocular ranging method has poor results when dealing with complex scenarios, and has huge computing resources, making it difficult to achieve high accuracy and robust target detection and distance estimation in the field of autonomous driving.
Using a monocular ranging method based on YOLO and anomaly jump filter, the target detector is built by improving the YOLOv5 network, and combining the Bot-Sort tracker and the Kalman filtering algorithm to enhance the accuracy and robustness of feature extraction and distance estimation. .
In road traffic scenarios, the accuracy and real-time nature of target detection are improved, the robustness of multi-target tracking is enhanced, and the reliability and accuracy of ranging are improved by eliminating abnormalities in detection frame size mutations.
Smart Images

Figure CN118968123A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection and recognition, and in particular to a monocular ranging method based on YOLO and an abnormal jump filter. Background Art
[0002] In the field of autonomous driving, how to enable a vehicle to achieve comprehensive perception of its surrounding environment is a challenging problem. This comprehensive perception includes not only the detection of various surrounding targets, but also the distance between these targets and the vehicle. Currently, the use of visible light cameras is a common and cost-effective way to solve this problem. Through photosensitive elements, visible light cameras can capture color images of the real world, and researchers can conduct in-depth analysis of color images to complete key perception tasks such as target detection and distance estimation.
[0003] As one of the core issues in the field of computer vision, object detection aims to locate and classify objects in images. Deep convolutional neural networks (DCNNs) provide strong support for the development of object detection technology with their excellent feature expression capabilities. At present, the main architectures of object detection can be divided into two categories: object detection methods based on region proposals and object detection methods based on regression / classification. They each have their own characteristics and solve different technical challenges.
[0004] How to obtain the distance of the target from the camera has become a key research topic in the field of computer vision since it was first proposed. In the initial methods, monocular depth estimation mainly relies on shape from shading (SFS) and depth from focus / defocus (DFF / DFD). SFS technology infers the shape based on the changes in illumination on the surface of the object, while DFF / DFD technology estimates the depth based on the focus and defocus information in the image and the calibration parameters of the camera. The basic assumption is that the focused part of the image is the clearest and the defocused part gradually becomes blurred. Although these methods are effective in specific scenarios, they usually require additional assumptions about the scene and do not work well when dealing with scenes with a large depth of field.
[0005] With the development of machine learning technology, researchers began to frame the single-pixel depth estimation problem as a probabilistic learning problem based on random fields. For example, Markov Random Field (MRF) is used to optimize the depth map, and Conditional Random Field (CRF) is used to model the posterior probability to estimate the depth. The increased computing power of graphics processing units (GPUs) and the availability of large-scale image datasets have provided impetus for deep learning, thus opening up new research directions in the field of depth estimation.
[0006] Deep learning techniques usually require real depth data as training labels. These techniques directly map pixels to depth values by constructing convolutional neural networks (CNNs). Liu et al. first applied CNN to the field of depth estimation, using a multi-scale network strategy. The global coarse estimation network provides preliminary distance estimates, the local fine network optimizes details, and introduces a scale-invariant loss function to handle errors. Subsequent studies have further improved the estimation accuracy by improving the network structure and integrating auxiliary information. Other studies do not rely on a large amount of labeled depth data, but use video sequences or stereo image pairs to automatically derive disparity maps and depth information through deep learning algorithms. For example, Garg, R. et al. process the left view through a convolutional neural network to generate a disparity map, and then reconstruct the left view by inverse mapping in combination with the right view, using the reconstruction error as a supervisory signal for training. Godard, C. et al. process the left and right views simultaneously to optimize the generation of disparity maps, and can obtain accurate depth estimates even if only monocular images are used in the test phase. In addition, Zhou, T. et al. use the previous and next frames of the video sequence to derive pose and depth, demonstrating the effectiveness of combining multiple network models for view reconstruction. Although these methods can achieve good performance, they consume huge computing resources.
[0007] Therefore, a monocular ranging method based on YOLO and abnormal jump filter is proposed. Summary of the invention
[0008] The purpose of the present invention is to provide a monocular ranging method based on YOLO and abnormal jump filter, which improves the accuracy and robustness of monocular ranging by fusing target detection and depth estimation.
[0009] To achieve the above object, the present invention provides the following solutions:
[0010] The monocular ranging method based on YOLO and abnormal jump filter includes:
[0011] Obtain a video to be tested, input the video to be tested into a preset target detector, output the target in each frame in the form of a detection frame, and add the target to a target list manager, wherein the target detector is obtained by training a training set, the training set includes a number of pictures, and the target detector is constructed by improving the YOLOv5 network;
[0012] Extracting target trajectory data from the target list manager using a Bot-Sort tracker, and obtaining trajectory distance data based on the trajectory data;
[0013] The trajectory distance data is processed by using an abnormal jump filter to obtain final trajectory data of the target, and the monocular ranging of the target is completed based on the final trajectory data.
[0014] Optionally, constructing the target detector by improving the YOLOv5 network includes:
[0015] The convolutional layers in the backbone network of the YOLOv5 network and the convolutional layers of the CBS module in the backbone network are replaced with RedeCa modules to construct a target detector based on YOLOv5-RedeCa.
[0016] Optionally, the RedeCa module includes: a residual depth-separable convolution block layer, a depth-separable convolution layer, a batch normalization layer, and a DeCa Block layer connected in sequence; the residual depth-separable convolution block layer and the DeCa Block layer both adopt a branch structure.
[0017] Optionally, one branch of the residual depthwise separable convolution block layer uses depthwise separable convolution for feature extraction, and the other branch uses residual edges to retain the original edge feature information, and then adds them together and passes through a SiLU activation function to output enhanced features.
[0018] Optionally, one branch of the DeCa Block layer uses an attention mechanism to assign weights according to the importance of information, and the other branch uses a residual edge to retain the original feature information, and then multiplies them and passes through a SiLU activation function to output key information.
[0019] Optionally, extracting the target's trajectory data from the target list manager using a Bot-Sort tracker, and acquiring the trajectory distance data based on the trajectory data includes:
[0020] When the target detector detects a target, an identifier is assigned to the target and added to the target list manager;
[0021] The Bot-Sort tracker is used to match the target with the tracked target. If the match fails, the target is created as a new tracking entry; if the match succeeds, the target is added to the tracking entry of the corresponding tracked target, and the trajectory data of the corresponding tracked target is updated, thereby obtaining the trajectory distance data.
[0022] Optionally, during the detection process, when the tracked target is not detected within a preset time, the tracking entry of the tracked target is deleted.
[0023] Optionally, the abnormal jump filter is constructed based on a fusion Kalman filter algorithm on a time series, and the Kalman filter algorithm is used to smooth the trajectory distance data and eliminate detection frames with abnormal mutations in size.
[0024] The beneficial effects of the present invention are:
[0025] The present invention designs a RedeCa module in the target detector to enhance the feature extraction capability and more accurately frame the target in road traffic scenarios; at the same time, it uses deep separable convolution to optimize the network architecture to ensure real-time performance.
[0026] The present invention introduces a multi-frame association mechanism Bot-Sort tracker to improve the robustness of the algorithm when facing target occlusion or temporary loss, and achieve simultaneous tracking and ranging of multiple different targets.
[0027] The present invention designs an abnormal jump filter to enhance the accuracy and robustness of distance estimation. The filter eliminates abnormal situations caused by sudden changes in the size of the detection box in a certain frame, smoothes the distance changes of the target, and thus improves the reliability of ranging. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0029] Figure 1 It is a flow chart of a monocular ranging method based on YOLO and abnormal jump filter according to an embodiment of the present invention;
[0030] Figure 2 This is a schematic diagram of the structure of a RedeCa module according to an embodiment of the present invention;
[0031] Figure 3 A schematic diagram of the YOLOv5-RedeCa target detection network structure according to an embodiment of the present invention;
[0032] Figure 4 A comparison diagram of the distance measurement results of the method according to an embodiment of the present invention and other methods;
[0033] Figure 5 It is a comparison diagram of ranging results with and without abnormal jump filter in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0036] This embodiment provides a monocular ranging method based on YOLO and abnormal jump filter, such as Figure 1 As shown, including:
[0037] Obtain the video to be tested, input the video to be tested into a preset target detector, output the target in each frame in the form of a detection frame, and add the target to the target list manager, wherein the target detector is obtained by training a training set, the training set includes several pictures, and the target detector is constructed by improving the YOLOv5 network;
[0038] Extract the target's trajectory data from the target list manager using the Bot-Sort tracker, and obtain the trajectory distance data based on the trajectory data;
[0039] The trajectory distance data is processed using an abnormal jump filter to obtain the final trajectory data of the target, and the monocular ranging of the target is completed based on the final trajectory data.
[0040] Furthermore, the target detector is constructed by improving the YOLOv5 network including:
[0041] like Figure 3 As shown in FIG. 5 , the convolutional layers in the backbone network of the YOLOv5 network and the convolutional layers of the CBS module in the backbone network are replaced with the RedeCa module to construct a target detector based on YOLOv5-RedeCa.
[0042] The RedeCa module includes: a residual depthwise separable convolutional block layer, a depthwise separable convolutional layer, a batch normalization layer, and a DeCa Block layer connected in sequence; both the residual depthwise separable convolutional block layer and the DeCa Block layer adopt a branch structure.
[0043] One branch of the residual depthwise separable convolution block layer uses depthwise separable convolution for feature extraction, and the other branch uses residual edges to retain the original edge feature information, which is then summed and passed through the SiLU activation function to output the enhanced features.
[0044] One branch of the DeCa Block layer uses the attention mechanism to assign weights according to the importance of information, and the other branch uses the residual edge to retain the original feature information, and then multiplies it through the SiLU activation function to output the key information.
[0045] Specifically, in this embodiment, the COCO dataset and the VOC PASCAL dataset are used to train, verify and test the target detector. The COCO dataset contains 163,957 images in total, including 118,287 images in the training set, 5,000 images in the verification set, and 40,670 images in the test set; the VOC PASCAL dataset contains 22,531 images in total, including 11,540 images in the training set and 10,991 images in the verification set.
[0046] Build an object detector based on YOLOv5-RedeCa, use the RedeCa module in the backbone network, and use depthwise separable convolution to simplify the computational complexity of the entire object detection network.
[0047] The convolutional layers in the backbone network are replaced by the RedeCa module, and the original CBS module is improved by the RedeCa module to become the CBS_RedeCa module, which is used to replace the original CBS module.
[0048] like Figure 2 As shown in the figure, the RedeCa module is based on the residual structure and sequentially integrates the residual depth separable convolution block, the depth separable convolution block and the ECA module to enhance the feature extraction capability. The feature information is input into the Redeca module and first enters the Res-Depthwise Separable Conv, which processes the input feature information in branches. One branch uses the depth separable convolution for effective feature extraction, and the other branch uses the residual edge to retain the original edge feature information as much as possible. Finally, the sum is added and the SiLU activation function is used to enhance the feature data. Its mathematical definition is as follows:
[0049]
[0050] where F * It is the feature map of the previous layer input.
[0051] Next, the feature information is input into the depthwise separable convolution to extract the effective feature information again.
[0052] After that, Batch Norm regularization is performed. When the feature information is input into the DeCa Block, this part also performs branch processing on the input feature information. One branch uses the ECA attention mechanism to assign a larger weight to the key information part, and the other branch uses the residual edge to retain the original feature information as much as possible, and then multiplies them to obtain the information that needs to be focused on, and then passes through the SiLU activation function. Its mathematical definition is as follows:
[0053]
[0054] F RedeCa =SiLU(F DWConv ×σ(Conv(AdaptAvgPool(F DWConv )))+F * )
[0055] The loss function of the YOLOv5-RedeCa target detector in this embodiment is:
[0056]
[0057]
[0058]
[0059] Among them, K, S 2 , B are the output feature map, cell and the number of anchors on each cell. x is the weight of the corresponding item. box , α obj , α cls The values are 0.05, 0.7, and 0.3 respectively. The weight used to balance the output feature map of each scale. The default value is [4.0, 1.0, 0.4]. Indicates whether the k-th output feature map, the i-th cell, and the j-th anchor box are positive samples. If they are positive samples, it is 1, otherwise it is 0. p , t gt are the prediction vector and the ground-truth vector. b, b gt Represent the center point of the predicted box and the center point of the ground truth box respectively. α is a balancing factor that weighs the loss caused by the aspect ratio and the loss caused by the IoU part. υ is the normalized difference between the aspect ratio of the predicted box and the true box. (w G ,h G ), (w P ,hP ) are the width and height of the target box and the predicted box respectively. o is the confidence score of the target in the predicted box. p IoU It is the iou value between the predicted box and the corresponding target box. represents the binary cross entropy loss. w obj , w cls Represents the weight of the positive sample.
[0060] Further, extracting the target's trajectory data from the target list manager using the Bot-Sort tracker, and obtaining the trajectory distance data based on the trajectory data includes:
[0061] When the target detector detects a target, it assigns an identifier to the target and adds it to the target list manager;
[0062] The Bot-Sort tracker is used to match the target with the tracked target. If the match fails, the target is created as a new tracking entry. If the match succeeds, the target is added to the tracking entry of the corresponding tracked target, and the trajectory data of the corresponding tracked target is updated to obtain the trajectory distance data.
[0063] During the detection process, if the tracked target is not detected within a preset time, the tracking entry of the tracked target is deleted.
[0064] Specifically, this embodiment sets the following logic in the target identification and maintenance of the target list manager: First, for newly detected targets, a unique identifier is assigned to each target through an increasing sequence to distinguish different tracking objects. Then, the Bot-Sort algorithm is used to compare the newly discovered targets with the tracked targets to determine whether these targets are newly appeared or already tracked objects.
[0065] For successfully matched targets, their status information is updated in the tracking list while maintaining their original unique identifiers. Newly appeared targets will be created as new tracking entries and assigned a new unique identifier. If a tracked target is not detected in the current frame, it may be due to occlusion or leaving the field of view, and its entry and ID will be temporarily retained. If the target does not appear again within the set time, it will be removed from the tracking list.
[0066] Connecting the Bot-Sort tracker can effectively identify and analyze each target in the video frame and determine which ones are new tracking objects. The algorithm and workflow of the Bot-Sort tracker are shown in Table 1:
[0067] Table 1
[0068]
[0069]
[0070] Furthermore, the abnormal jump filter is constructed based on the fusion of Kalman filter algorithm on the time series. The Kalman filter algorithm is used to smooth the trajectory distance data and eliminate the detection boxes with abnormal mutations in size.
[0071] Specifically, the abnormal jump filter of Kalman filter is integrated on the time series to improve the accuracy and robustness of distance estimation. The abnormal jump filter based on Kalman filter algorithm can smooth the change of target distance by analyzing historical data and eliminate the abnormality caused by the sudden change of single frame detection frame size. Its state prediction formula and observation update formula are as follows:
[0072] State prediction formula:
[0073]
[0074] in, represents the predicted state, F k is the state transfer matrix, B k is the control input model, u k is the control vector, P kk-1 is the forecast covariance, Q k is the process noise covariance.
[0075] Observation update formula:
[0076]
[0077] Among them, K k is the Kalman gain, H k is the observation matrix, z k is the actual observation, R k is the observation noise covariance and I is the identity matrix.
[0078] After testing the YOLOv5-RedeCa-based object detector proposed in this embodiment on the COCO dataset and the VOC PASCAL dataset, it can be found that the accuracy, mean average precision (mAP) and recall rate on the COCO dataset reached 71.6%, 61.5% and 57.9%; the mean average precision (mAP) and recall rate on the VOC PASCAL dataset reached 77.5% and 73.4%. The object detector of this embodiment has excellent performance.
[0079] The method proposed in this embodiment is experimentally compared with other methods, and the experimental environment is shown in Table 2:
[0080] Table 2
[0081]
[0082] (1) Comparison of ranging effects of overall methods:
[0083] like Figure 4 As shown in the figure, the standard deviation of this method is 0.25. The standard deviation of Faster-RCNN is 0.33. The standard deviation of Mask R-CNN is 0.37. The standard deviation of SSD is 0.40. The standard deviation of this method is significantly lower than that of other methods, showing that this method has stronger processing ability for outliers and better robustness.
[0084] (2) Comparison of ranging effects with and without abnormal jump filters:
[0085] like Figure 5 As shown in the figure, the standard deviation of the monocular ranging method using the abnormal jump filter is 0.72, while the standard deviation of the monocular ranging method without the abnormal jump filter is 1.29. The stability of the monocular ranging method using the abnormal jump filter is significantly improved compared with the method without the abnormal jump filter.
[0086] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A monocular ranging method based on YOLO and abnormal jump filter, characterized in that: include: Obtain a video to be tested, input the video to be tested into a preset target detector, output the target in each frame in the form of a detection frame, and add the target to a target list manager, wherein the target detector is obtained by training a training set, the training set includes a number of pictures, and the target detector is constructed by improving the YOLOv5 network; Extracting target trajectory data from the target list manager using a Bot-Sort tracker, and obtaining trajectory distance data based on the trajectory data; The trajectory distance data is processed by using an abnormal jump filter to obtain final trajectory data of the target, and the monocular ranging of the target is completed based on the final trajectory data.
2. The monocular distance measurement method based on YOLO and abnormal jump filter according to claim 1, characterized in that: The target detector is constructed by improving the YOLOv5 network including: The convolutional layers in the backbone network of the YOLOv5 network and the convolutional layers of the CBS module in the backbone network are replaced with RedeCa modules to construct a target detector based on YOLOv5-RedeCa.
3. The monocular distance measurement method based on YOLO and abnormal jump filter according to claim 2, characterized in that: The RedeCa module includes: a residual depth-separable convolution block layer, a depth-separable convolution layer, a batch normalization layer, and a DeCa Block layer connected in sequence; the residual depth-separable convolution block layer and the DeCa Block layer both adopt a branch structure.
4. The monocular ranging method based on YOLO and abnormal jump filter according to claim 3 is characterized in that: One branch of the residual depthwise separable convolution block layer uses depthwise separable convolution for feature extraction, and the other branch uses residual edges to retain the original edge feature information, which is then summed and passed through a SiLU activation function to output enhanced features.
5. The monocular distance measurement method based on YOLO and abnormal jump filter according to claim 3 is characterized in that: One branch of the DeCa Block layer uses the attention mechanism to assign weights according to the importance of information, and the other branch uses the residual edge to retain the original feature information, and then multiplies it through the SiLU activation function to output the key information.
6. The monocular distance measurement method based on YOLO and abnormal jump filter according to claim 1, characterized in that: Extracting the target's trajectory data from the target list manager using the Bot-Sort tracker, and acquiring the trajectory distance data based on the trajectory data includes: When the target detector detects a target, an identifier is assigned to the target and added to the target list manager; The Bot-Sort tracker is used to match the target with the tracked target. If the match fails, the target is created as a new tracking entry; if the match succeeds, the target is added to the tracking entry of the corresponding tracked target, and the trajectory data of the corresponding tracked target is updated, thereby obtaining the trajectory distance data.
7. The monocular distance measurement method based on YOLO and abnormal jump filter according to claim 6, characterized in that: During the detection process, when the tracked target is not detected within a preset time, the tracking entry of the tracked target is deleted.
8. The monocular distance measurement method based on YOLO and abnormal jump filter according to claim 1, characterized in that: The abnormal jump filter is constructed based on the fusion of the Kalman filter algorithm on the time series, and the Kalman filter algorithm is used to smooth the trajectory distance data and eliminate the detection frame with abnormal mutation in size.
Citation Information
Patent Citations
Special target abnormal state detection and tracking method based on improved YOLOv5 network
CN114821032A
Moving target detection and tracking method for robot navigation
CN117333833A
Target grounding point detection and distance measurement method based on YOLO
CN117611800A
Training method and electronic device
US20230394811A1
Cited By
Monocular three-dimensional target detection method based on group type dynamic query distribution algorithm
CN121962765A
