A vehicle surrounding obstacle warning method based on monocular depth estimation

Abstract

The present invention discloses a vehicle-surrounding obstacle warning method based on monocular depth estimation, comprising: (1) acquiring image data, generating pixel labels, depth map labels, and 3D target labels required for training in the image data to form a training data set; (2) establishing a 3D target detection model based on monocular depth estimation; (3) training the 3D target detection model using the training data set; (4) using the 3D target detection model obtained through training optimization to detect obstacles in consecutive frames during obstacle warning; (5) constructing a tracking model, using the Hungarian maximum matching algorithm to track corresponding obstacles in consecutive frames; (6) establishing a Kalman filter model related to the spatial position and velocity of the obstacle, and finally obtaining the spatial position information of the tracked obstacle through the filtering algorithm and judging whether there is a collision risk. By using the present invention, the accuracy of vehicle obstacle warning can be improved while saving costs.

Description

Technical Field

[0001] The present invention belongs to the field of 3D target detection in computer vision, and in particular relates to a vehicle surrounding obstacle warning method based on monocular depth estimation. Background Art

[0002] The increasing intelligence of vehicles has made assisted driving a must-have feature for mid-range and high-end models. To ensure driving safety, obstacle detection and driver warning are core features of assisted driving. The obstacle warning system calculates the relative distance and speed between the obstacle and the vehicle to determine whether an accident is imminent and promptly alerts the driver to avoid danger.

[0003] To obtain information about the state of obstacles ahead of the vehicle, Kunsoo Hu et al., in their paper "An experimental investigation of a CW / CA system for automobiles using hardware-in-the-loop simulations" presented at the 1999 American Control Conference, used a sampling discretization method to obtain the discrete millimeter-wave radar measurement system equations. They then used a second-order Kalman filter to optimize the system state, including the target vehicle's distance and relative speed. However, this method's detection accuracy was limited. To improve the detection accuracy of millimeter-wave radar, Anselm Haselhoff et al., in their paper "Radar-Vision Fusion with an Application to Car-Following using an Improved AdaBoost Detection Algorithm" presented at the 2007 IEEE Intelligent Transportation Systems Conference, proposed a method for obstacle detection that uses millimeter-wave radar and vision fusion. This approach first uses millimeter-wave radar to pre-detect 3D candidate obstacles. This information is then used to determine regions of interest in the image. Finally, an AdaBoost classifier is used to validate the millimeter-wave radar's detection results. However, this approach relies heavily on the detection results of millimeter-wave radar. If a threat target is missed in the millimeter-wave radar candidate area, subsequent operations will fail to detect the target again. To address this issue, W Tao et al. proposed a three-stage fusion strategy for monocular image and millimeter-wave radar for obstacle detection in their 2011 paper "Integrating Millimeter Wave Radar with a Monocular Vision Sensor for On-Road Obstacle Detection Applications" published in Sensors. This method first calibrates the coordinate systems of the millimeter-wave radar and camera, then locks the millimeter-wave radar detection area for obstacle detection, and then uses the corresponding image area to verify the detected target. Furthermore, Xin Liu et al. proposed a cross-validation method for obstacle detection in their 2011 paper "On-road vehicle detection fusing radar and vision" published in the IEEE International Conference on Vehicular Electronics and Safety.This method uses a special shadow segmentation method to first detect the image, and then verify and match it with the results detected by the millimeter-wave radar obtained in the same frame. For the millimeter-wave radar objects that are not matched, visual data is used again for verification.

[0004] In addition to using monocular camera fusion, Shunguang Wu et al., in their article "Collision Sensing by Stereo Vision and Radar Sensor Fusion" published in the 2009 IEEE Transactions on Intelligent Transportation Systems, proposed using depth camera and millimeter-wave radar fusion to detect obstacles. This method first uses depth vision to fit the closest point of the threatening obstacle's outline, then fuses this point with the millimeter-wave radar detection results. Finally, rigid body constraints are used to track the closest point of the fused obstacle outline to determine the threatening obstacle's spatial position and motion state.

[0005] Compared to millimeter-wave radar, lidar has a longer detection range and greater accuracy. To achieve real-time performance in acquiring information about surrounding obstacles, Alex H. Lang et al., in their paper "PointPillars: Fast Encoders for Object Detection from Point Clouds," published at the 2019 IEEE Conference on Computer Vision and Pattern Recognition, proposed encoding point cloud features into a bird's-eye view pseudo-image and using convolution to detect objects in 3D space. To improve accuracy while maintaining real-time performance, Zetong Yang et al., in their paper "3DSSD: Point-Based 3D Single-Stage Object Detector," removed the FP module from common point cloud feature learning methods by fusing F-FPS and D-FPS downsampling. This approach reduces the computational complexity of feature extraction while maintaining accuracy, achieving impressive results in 3D object detection. However, lidar is expensive and impractical for obstacle warning in assisted driving.

[0006] In summary, due to cost considerations, the LiDAR and sensor fusion solution is not very suitable for the mass production of vehicles with assisted driving functions. Using only millimeter-wave radar cannot achieve good obstacle detection effects due to the equipment limitations of its sensors. Summary of the Invention

[0007] The present invention provides a method for warning of obstacles around a vehicle based on monocular depth estimation, which can not only save actual production costs but also achieve the accuracy required for the actual application of the vehicle obstacle warning function.

[0008] A method for warning of obstacles around a vehicle based on monocular depth estimation comprises the following steps:

[0009] (1) acquiring image data, wherein the image data includes camera calibration parameters and point cloud data in the same frame as the image; generating pixel labels, depth map labels, and 3D object labels required for training from the image data to form a training data set;

[0010] (2) Establish a 3D target detection model based on monocular depth estimation;

[0011] (3) Using the training data set to train and test the 3D object detection model, and finally obtaining the trained and optimized 3D object detection model;

[0012] (4) During the obstacle warning process, the 3D target detection model obtained through training and optimization is used to detect obstacles in consecutive frames;

[0013] (5) Construct a tracking model and use the Hungarian maximum matching algorithm to track corresponding obstacles in consecutive frames;

[0014] (6) Establish a Kalman filter model related to the spatial position and speed of the obstacle. Through the filtering algorithm, the spatial position information of the tracked obstacle is finally obtained and used as the distance benchmark to determine whether there is a collision risk.

[0015] The vehicle obstacle warning method of this invention is a 3D object detection method based on monocular depth estimation. Using a monocular camera as a sensor not only saves costs, but also, with the development of monocular depth estimation methods, the depth estimated using monocular estimation has extremely low error at close ranges, thus ensuring a high degree of confidence in the spatial position of detected obstacles.

[0016] In step (1), the camera calibration parameters and the point cloud data in the same frame as the image are used to calculate the depth z value of the pixel corresponding to the point cloud data in the camera coordinate system, and the z value is used as the true pixel depth value; wherein, the depth value of the pixel that is not matched to the point cloud is set to 0 by default, thereby obtaining the depth map label of the monocular image.

[0017] In step (2), in the 3D target detection model, DenseNet121 is used as the backbone for image feature extraction, and the BTS depth estimation model is used to predict the depth value of each pixel based on the extracted image features; at the same time, an interest pixel proposal module is used to generate an interest pixel set based on the extracted image features; finally, a simplified single-stage 3D detection head is used, with the pseudo laser points generated by the interest pixels as input, to output the 3D spatial position, size and category of the obstacle obtained by regression.

[0018] In step (3), the process of training the 3D object detection model using the training dataset is as follows:

[0019] (3-1) Randomly shuffle the training dataset, and then perform data augmentation on the images, pixel labels, 3D labels, and depth map labels by randomly flipping them horizontally by 50%;

[0020] (3-2) The training dataset is fed into the 3D object detection network with a preset BatchSize number of images. The depth value of each pixel is predicted by the network depth regression head corresponding to the BTS depth estimation model, and the region of interest module corresponding to the pixel of interest proposal module generates the pixels of interest that are most likely to be obstacles.

[0021] (3-3) The above-mentioned pixel points of interest and their depth values ​​are used as input and converted into their corresponding spatial coordinate points through camera calibration parameters; the generated spatial coordinate points are input into the 3D regression head corresponding to the 3D object detection head to regress the spatial position and size of the obstacle and predict its category. In the process of training the 3D object detection head, the Euclidean distance between the pixel points with true depth value and the predicted depth and the Euclidean distance between the pixel category of interest and its predicted value are minimized as much as possible. At the same time, the Euclidean distance between the spatial position, size and category of the obstacle and the predicted value is minimized as much as possible;

[0022] (3-4) Repeat steps (3-1) to (3-3) until the training is completed after the preset number of training times.

[0023] In step (3-2), the objective function of the network deep regression head training is the scale-invariant loss function in log space, and the formula is:

[0024]

[0025] Among them, T represents the number of pixels with true depth value, λ is a hyperparameter, and its value is set to 0.5, g i It represents the Euclidean distance between the depth prediction value and the true value in log space. The specific calculation formula is as follows:

[0026]

[0027] in, with d i They represent the estimated depth value and the true depth value respectively. Since there are many true depth pixels in the scene, the final loss function of the network depth regression head is defined as:

[0028]

[0029] Among them, α is used as the loss weight control and is set to 10 during training.

[0030] When training the region of interest module, a pixel category cross entropy loss function is set to constrain the network, and its loss function is defined as:

[0031]

[0032] Among them, y represents the pixel category, and its value is 0 and 1, representing background points and obstacle pixels respectively. Represents the pixel category prediction value.

[0033] In step (3-3), the training objective function of the 3D regression head includes a classification loss function and a regression loss function, and the formula is:

[0034]

[0035]

[0036] Among them, L c is the classification loss, P i is the probability of predicting the i-th category, K represents the number of predicted categories. In this method, there are mainly two categories: car and others, so K is set to 2, y i Indicates the category of the class; L r is the regression loss of the target spatial position. The loss function uses the SmoothL1Loss loss function, β is a hyperparameter, set to 0.1, μ i and are the true value and the predicted value.

[0037] In step (6), in the established Kalman filter model, the observed quantities are x, y, z, h, w, l and θ, where x, y, z correspond to the spatial position of the obstacle, h, w, l correspond to the size of the obstacle, and θ corresponds to the direction of the obstacle; the predicted quantity is x p 、y p 、z p 、h p 、w p 、l p ,θ p 、υx 、υ y and υ z , that is, the spatial position, size, orientation and speed in three directions of the obstacle after filtering;

[0038] When establishing the Kalman filter model, it is assumed that both the prediction noise and the observation noise obey the normal distribution. The specific settings are as follows:

[0039]

[0040]

[0041]

[0042]

[0043] Among them, Q represents the prediction noise covariance matrix in the Kalman filter model, K represents the observation noise covariance matrix, F represents the state transfer matrix, and H represents the observation matrix.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The method for warning of obstacles around a vehicle based on monocular depth estimation proposed in the present invention can accurately detect the spatial position of obstacles in a relatively close range.

[0046] 2. The present invention uses a monocular camera to detect the spatial position of surrounding obstacles, thereby reducing the cost of vehicle mass production.

[0047] 3. The present invention can obtain more accurate obstacle location information through the maximum matching algorithm and Kalman filter algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the method of the present invention;

[0049] Figure 2 This is a network framework diagram of the 3D object detection model in the method of the present invention;

[0050] Figure 3 The results of detecting vehicle obstacles using a monocular camera on different roads in the KITTI dataset. DETAILED DESCRIPTION

[0051] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0052] like Figure 1As shown, the present invention mainly divides the obstacle warning function into two modules, the obstacle detection module (3D target detection model) and the obstacle tracking module. Based on the detection results, the obstacle tracking module tracks the obstacle and finally obtains the obstacle's ID and distance information.

[0053] The network framework of the 3D target detection model based on monocular depth estimation is as follows Figure 2 As shown in Figure 1, this network is an end-to-end trained network model. During training, the input data for this model is the image, the camera calibration parameters corresponding to the image, and the point cloud data of the same frame as the image. The basic training steps are as follows:

[0054] 1. Generate labels required for network model training from raw data, including pixel labels, image depth map labels, and 3D target labels. The pixel label indicates whether the pixel is a pixel of interest. The present invention generates pixel labels through 2D frame labels, wherein the label of the pixel within the 2D frame is set to 1, indicating the pixel of interest, and the pixel outside the 2D frame is set to 0. The image depth map label is obtained through laser point cloud data. The laser point cloud data is first converted into coordinates in the camera coordinate system through camera calibration external parameters, and then converted into pixel coordinates according to the camera internal parameters, so as to obtain the depth value of the pixel corresponding to the laser point, and the depth map value of the pixel that does not match the laser point is set to 0.

[0055] 2. Initialize the network model parameters and feed the image data into the image feature extraction backbone network to extract feature maps with scales downsampled by 2, 4, 8, and 16 times respectively. These four feature maps are then fed into the depth estimation module and the pixel of interest proposal module respectively.

[0056] 3. In the depth estimation module, the BTS depth estimation model, described in the paper "From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation," is used to estimate depth values. After obtaining pixel-wise estimated depth values, the model is optimized using labels. This depth information is then used as input to the 3D object detection head for further calculations.

[0057] 4. In the pixel proposal module, the feature maps of each scale are first upsampled and spliced ​​to the original image size to obtain a feature map with multi-scale information. Using this feature map as input, a prediction head is used to predict the pixel category, and the first 4096 pixels with higher scores are regarded as pixels of interest. Finally, the pixels of interest and their features are sent to the 3D object detection head for the next step of calculation.

[0058] 5. The 3D object detection head receives the pixel of interest, its depth map, and its features. It converts the pixel of interest into a point in the world coordinate system using camera calibration intrinsic and extrinsic parameters. It then downsamples, groups, and learns features based on the point with the longest feature distance to obtain 256 candidate points and their features. Finally, a detection head uses these 256 candidate points and their features as input to predict the 3D bounding box and its class.

[0059] 6. Loop through the training dataset several times to eventually obtain a converged monocular obstacle detection network.

[0060] After obtaining the detection results, the obstacle needs to be tracked. The present invention uses the Hungarian algorithm to perform maximum obstacle matching between consecutive frames. At the same time, in order to obtain stable tracking results, a Kalman filter is used to reduce the prediction error.

[0061] Figure 3 In the figure, the detection effect of vehicle obstacles in three road scenes using the monocular 3D target is demonstrated. It can be clearly seen that the method of the present invention can comprehensively detect the vehicles around the vehicle and can accurately predict the distance between the surrounding vehicles and the vehicle.

[0062] This example was trained and tested on the KITTI dataset, a large public dataset. The monocular object detection network was trained on the KITTI 3D Object Detection Evaluation 2017 dataset, which consists of 3,712 training and 3,769 validation images, respectively. Detection and tracking experiments were conducted on three scenes from the KITTI Object Tracking Evaluation dataset.

[0063] The evaluation criteria used in this paper are precision and recall. This paper uses the target detection algorithm to detect and track objects on three road scenes in KITTI. The results are shown in Table 1.

[0064] Table 1

[0065]

[0066] As can be seen from Table 1, the proposed method fully demonstrates the effectiveness of the present invention in obstacle detection and tracking.

[0067] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vehicle surrounding obstacle warning method based on monocular depth estimation, characterized in that: The following steps are involved: (1) acquiring image data, wherein the image data includes camera calibration parameters and point cloud data in the same frame as the image; generating pixel labels, depth map labels, and 3D object labels required for training from the image data to form a training data set; (2) Establishing a 3D object detection model based on monocular depth estimation; in the 3D object detection model, DenseNet121 is used as the backbone for image feature extraction, and the BTS depth estimation model is used to predict the depth value of each pixel based on the extracted image features; at the same time, a pixel of interest proposal module is used to generate a set of pixels of interest based on the extracted image features; finally, a simplified single-stage 3D detection head is used, with the pseudo laser points generated by the pixels of interest as input, to output the 3D spatial position, size, and category of the obstacle obtained by regression; (3) Use the training data set to train and test the 3D object detection model, and finally obtain the trained and optimized 3D object detection model; the specific process is: (3-1) Randomly shuffle the training dataset, and then perform data augmentation on the images, pixel labels, 3D labels, and depth map labels by randomly flipping them horizontally by 50%; (3-2) The training dataset is input into the 3D object detection network with a preset BatchSize number of images. The depth value of each pixel is predicted by the network depth regression head corresponding to the BTS depth estimation model, and the interest pixel of the obstacle is generated by the region of interest module corresponding to the interest pixel proposal module. The objective function of the network deep regression head training is the scale-invariant loss function in log space, which is formulated as: Among them, T represents the number of pixels with true depth value, λ is a hyperparameter, and its value is set to 0.5, g i It represents the Euclidean distance between the depth prediction value and the true value in log space. The specific calculation formula is as follows: in, with d i They represent the estimated depth value and the true depth value respectively. Since there are many true depth pixels in the scene, the final loss function of the network depth regression head is defined as: Among them, α is the loss weight control amount and is set to 10 during training; (3-3) The above-mentioned pixel of interest and its depth value are used as input and converted into its corresponding spatial coordinate point through the camera calibration parameters; the generated spatial coordinate point is input into the 3D regression head corresponding to the 3D object detection head to regress the spatial position and size of the obstacle and predict its category; (3-4) Repeat steps (3-1) to (3-3) until the training reaches the preset number of times; (4) During the obstacle warning process, the 3D target detection model obtained through training and optimization is used to detect obstacles in consecutive frames; (5) Construct a tracking model and use the Hungarian maximum matching algorithm to track corresponding obstacles in consecutive frames; (6) Establish a Kalman filter model related to the spatial position and speed of the obstacle. Through the filtering algorithm, the spatial position information of the tracked obstacle is finally obtained and used as the distance benchmark to determine whether there is a collision risk.

2. The vehicle surrounding obstacle warning method based on monocular depth estimation according to claim 1, characterized in that: In step (1), the camera calibration parameters and the point cloud data in the same frame as the image are used to calculate the depth z value of the pixel corresponding to the point cloud data in the camera coordinate system, and the z value is used as the true pixel depth value; wherein, the depth value of the pixel that is not matched to the point cloud is set to 0 by default, thereby obtaining the depth map label of the monocular image.

3. The vehicle surrounding obstacle warning method based on monocular depth estimation according to claim 1, characterized in that: In step (3-2), the pixel category cross entropy loss function is set to constrain the network during the training of the region of interest module. The loss function is defined as: Among them, y represents the pixel category, and its value is 0 and 1, representing background points and obstacle pixels respectively. Represents the pixel category prediction value.

4. The vehicle surrounding obstacle warning method based on monocular depth estimation according to claim 1, characterized in that: In step (3-3), the training objective function of the 3D regression head includes a classification loss function and a regression loss function, and the formula is: Among them, L c is the classification loss, P i is the probability of predicting the i-th category, K represents the number of predicted categories. This method includes two categories: car and others, so K is set to 2, y i Indicates the category of the class; L r is the regression loss of the target spatial position. The loss function uses the SmoothL1Loss loss function, β is a hyperparameter, set to 0.1, μ i and are the true value and the predicted value.

5. The vehicle surrounding obstacle warning method based on monocular depth estimation according to claim 1, characterized in that: In step (6), in the established Kalman filter model, the observed quantities are x, y, z, h, w, l and θ, where x, y, z correspond to the spatial position of the obstacle, h, w, l correspond to the size of the obstacle, and θ corresponds to the direction of the obstacle; the predicted quantity is x p 、y p 、z p 、h p 、w p 、l p ,θ p 、v x 、v y and v z , that is, the spatial position, size, orientation and speed in three directions of the obstacle after filtering; When establishing the Kalman filter model, it is assumed that both the prediction noise and the observation noise obey the normal distribution. The specific settings are as follows: Among them, Q represents the prediction noise covariance matrix in the Kalman filter model, K represents the observation noise covariance matrix, F represents the state transfer matrix, and H represents the observation matrix.

Images