A forward vehicle distance detection method based on cascaded SSD and monocular depth estimation

Abstract

The present invention discloses a forward vehicle distance detection method based on cascade SSD and monocular depth estimation, comprising the following steps: S1, obtaining forward vehicle training samples and depth estimation data sets; S2, building a backbone network; S3, building a target detection network; S4, building a depth estimation subnetwork; S5, training a forward vehicle distance detection network based on a cascade SSD network and monocular depth estimation; S6, selection and distance detection based on K-means clustering feature points. The present invention adopts an extended receptive field module based on dilated convolution to expand the receptive field, and combines an online feature selection strategy to improve the accuracy of small-scale vehicle target detection; in view of the problem of inaccurate vehicle positioning, a cascade SSD network is proposed, and the problem of feature misalignment caused by the anchor frame mechanism is alleviated by introducing deformable convolution; in view of the problem of inaccurate selection of monocular depth estimation feature points, a selection and ranging method based on K-means clustering feature points is proposed, which effectively solves the problem of large lateral and longitudinal ranging errors.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle distance detection, and in particular to a forward vehicle distance detection method based on cascaded SSD and monocular depth estimation. Background Art

[0002] With the maturity and continuous progress of computer vision technology, it has also driven the rapid development of emerging fields such as autonomous driving, smart medical care, and smart robots. In particular, the autonomous driving industry has seen the hope of realizing unmanned driving under the increasingly mature and stable trend of computer vision technology. One of the key technologies is to detect the distance of the target in front of the vehicle on the premise of accurately identifying the target in front of the vehicle, which has also become one of the main research hotspots in the field of computer vision. According to incomplete statistics, more than 1.25 million people die in road traffic accidents every year in the world, the number of injured people is as high as 50 million, and the economic loss reaches 1.85 trillion US dollars. The main cause of the accident is the driver's inattention or fatigue driving. Given that highway traffic accidents are often caused by rear-end collisions, if the distance between the vehicle and the vehicle in front can be accurately detected in real time, it can play a good preventive role in similar traffic accidents such as rear-end collisions. Therefore, the distance detection between vehicles plays an important role in ensuring people's travel safety to the greatest extent.

[0003] According to the different detection equipment and detection methods, forward vehicle distance detection methods are divided into the following categories: millimeter wave radar ranging, laser ranging, ultrasonic ranging, visual ranging, etc. Among these methods, although ultrasonic, millimeter wave radar, laser radar, etc. are far superior to visual ranging in ranging accuracy, they all have certain defects. In comparison, visual ranging has the advantages of low cost and easy popularization. Binocular visual ranging obtains depth information through feature point matching. It is time-consuming and difficult to meet the real-time requirements of detection. Monocular vision is more affordable and has a simpler model structure, making it a more advantageous research hotspot.

[0004] Existing distance measurement algorithms based on monocular vision can be roughly divided into four categories: the first category is the method based on the pinhole forming model, which realizes the conversion from the pixel coordinate system to the world coordinate system through coordinate system transformation, which requires the actual height or width of the measured object to complete; the second category is the method based on data regression modeling, which uses the reverse method of "first distance measurement, then modeling" to complete the distance measurement of the vehicle in front, such as "A vehicle distance measurement method based on monocular vision" (publication number: CN104899554A) applied by Northeastern University. The invention uses the distance measurement method of pinhole imaging at close distances and the data fitting method at long distances, which reduces the error rate, but the problem is that the generalization ability is too poor when changing scenes. The third category is to combine the calibration parameters of the camera to build an imaging geometric model to measure the distance, such as "A front vehicle distance detection method" (publication number: CN106153000A) applied by Hefei University of Technology. The invention uses a geometric model to complete the distance measurement of the front vehicle, with fast calculation speed and simple implementation, but it is greatly affected by camera parameters and pitch angle. The fourth category is the forward distance detection method based on monocular depth estimation, which obtains distance information from the picture through monocular depth estimation, such as the "A monocular vision forward vehicle distance detection method based on depth estimation" applied by Chongqing University (application number: 202110633046.7). This invention can make good use of monocular depth estimation to complete the ranging task, but lacks real-time detection of vehicle detection, and there are certain constraints in the application.

[0005] Real-time ranging of forward vehicle targets is an important part of the automatic driving system, and the first step is to detect the forward vehicle. In view of the different angles of the forward vehicle, the obvious difference in size with the distance, and the low pixel, the present invention selects the SSD neural network with a faster detection speed as the basis for improvement while meeting the real-time detection, and uses it as a sub-network of the forward vehicle distance detection network. First, in view of the problem that the SSD network is not able to express the features extracted by small targets in a sufficient manner, the present invention proposes a cascaded SSD network, and uses the ASPP network cascade method to enrich the low-level network features in the initial inspection network. The cascaded SSD network uses deformable convolution as a connection to alleviate the feature misalignment problem caused by the anchor frame mechanism, and reduce the impact of negative samples on network training, thereby improving the accuracy of target detection. During the imaging process, the camera projects the three-dimensional image onto the two-dimensional image plane and loses the depth information. The depth information needs to be restored, and the distance information from each pixel to the camera plane is obtained. Then, a ranging feature point is selected to complete the ranging of the forward vehicle. This process can establish a monocular depth estimation subnetwork that extracts global pixel information and connect it in parallel with the target detection subnetwork to achieve distance detection of the forward vehicle target. Summary of the invention

[0006] Starting from the actual environment of highways and urban roads, the present invention studies the problem that the SSD network has poor effect on the extraction of forward small-scale vehicle features, and the deficiencies of the above-mentioned traditional ranging algorithms in forward vehicle ranging, and designs a forward vehicle distance detection network based on cascaded SSD and monocular depth estimation. Aiming at the problem that conv4_3 in the original SSD network does not extract shallow features well, resulting in poor small target detection, an extended receptive field module based on dilated convolution is used to expand the receptive field, and combined with an online feature selection strategy, the accuracy of small-scale vehicle target detection is improved; for the problem of inaccurate vehicle positioning, a cascaded SSD network is proposed, and the problem of feature misalignment caused by the anchor frame mechanism is alleviated by introducing deformable convolution; for the problem of inaccurate feature point selection for monocular depth estimation, a method for selecting and ranging feature points based on K-means clustering is proposed, which can effectively solve the problem of large errors in lateral and longitudinal ranging.

[0007] The technical method provided by the present invention mainly includes three parts: sample training, detection and ranging, which are specifically the following steps:

[0008] Step 1: Obtain forward vehicle training samples and depth estimation datasets

[0009] Download the KITTI dataset, which contains vehicle training images, annotation files, camera and radar calibration parameters for each image, and point cloud files.

[0010] Perform data cleaning on the KITTI dataset, filter out and remove incorrectly labeled images;

[0011] K-means clustering is used to select the appropriate number of anchor boxes and aspect ratio;

[0012] Calculate the depth information of the statistical data set to prepare for the subsequent optimization of the depth estimation model;

[0013] Randomly use 80% of the data set as a training set and 20% as a test set;

[0014] Step 2: Building the backbone network

[0015] Select VGG-16 as the basis of the backbone network for feature extraction, and make modifications on this basis;

[0016] The training image is downsampled to the resolution of Conv4-3, and then passed through three 3×3 convolutional layers to obtain the low-level feature map δ0 of the image target details; in order to obtain high-level features, the fc7 layer features are upsampled to the resolution of Conv4-3 to obtain the feature map δ1, and then Conv5-3 is upsampled by deconvolution to obtain the feature map δ2. Finally, the three feature maps δ0, δ1, and δ2 are fused with the channel dimension of Conv4-3 to achieve feature enhancement and obtain the fused feature map ψ0;

[0017] Take ψ0 as the input of the cascaded ASPP, and set the hole rate of ASPP to 2 and 3 respectively. Then extract the feature layers ψ1 and ψ2 after each ASPP, superimpose ψ0, ψ1, and ψ2 on the channel, and make them consistent with the number of channels of the original Conv4-3 to obtain the feature map ψ.

[0018] Step 3: Building the target detection subnetwork

[0019] The object detection sub-network is improved based on the SSD network. First, the number and size of anchor boxes in the detection network are changed to the size obtained in step 1 above.

[0020] Based on the feature map ψ obtained in step 2 above, three additional convolutional layers are added. Each convolutional layer consists of a 1×1 convolution with a stride of 2 and a 3×3 convolution. The feature maps of four scales are used to construct a feature pyramid as the initial inspection network of the target detection network to classify the background and foreground.

[0021] Based on the initial inspection network, deformable convolution is used for the feature maps of each scale to alleviate the feature misalignment problem caused by the anchor box mechanism and improve the regression accuracy of the detection box.

[0022] Step 4: Building the depth estimation subnetwork

[0023] Based on the feature map ψ obtained in step 2 above, an additional 3 convolutional layers are added. Each convolutional layer consists of a 1×1 convolution with a stride of 2 and a 3×3 convolution to obtain the feature layer σ0.

[0024] Then, based on the feature layer σ0, upsampling is performed through the identity mapping and the deconvolution layer, and the deconvolution image size is reduced to 1 / 4 of the original image;

[0025] Step 5: Training of forward vehicle distance detection network based on cascaded SSD network and monocular depth estimation

[0026] Design the overall loss function, including target detection loss function, monocular estimation network loss function and perception loss function;

[0027] The network input image size is set to 300*300, the initial learning rate is set to 0.003, the learning rate is adjusted to polynomial decay, and the batch-size is set to 16;

[0028] The model training was first iterated 500,000 times, then the depth estimation network branch was suppressed, only the target detection branch was iterated 150,000 times, and then the entire network was iterated 200,000 times;

[0029] Step 6: Selection of feature points and distance detection based on K-means clustering

[0030] Using the original image as input, the trained model is used to obtain the vehicle detection confidence and detection box in the image, as well as the monocular depth image of the image.

[0031] Get the coordinates of the detection box (x1, y1, x2, y2), and the coordinates of the center point are expressed as:

[0032]

[0033] The horizontal coordinate of the center point is used to divide the vehicle into two parts, the lower part, which contains most of the information of the rear or front of the vehicle and is also the closest point between vehicles. This is used as the depth estimation cluster set;

[0034] The K-means clustering algorithm is used to cluster the set obtained in step 2), mainly into two categories: foreground and background. Finally, the center point of the foreground cluster is used as the ranging feature point and the distance information is output.

[0035] Beneficial effects:

[0036] Aiming at the problem that conv4_3 in the original SSD network does not extract shallow features sufficiently, resulting in poor small target detection effect, the invention adopts an extended receptive field module based on hole convolution to expand the receptive field, and combines it with an online feature selection strategy to improve the accuracy of small-scale vehicle target detection; aiming at the problem of inaccurate vehicle positioning, a cascaded SSD network is proposed, and the feature misalignment problem caused by the anchor frame mechanism is alleviated by introducing deformable convolution; aiming at the problem of inaccurate feature point selection for monocular depth estimation, a feature point selection and ranging method based on K-means clustering is proposed, which can effectively solve the problem of large lateral and longitudinal ranging errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the forward vehicle distance detection method based on the cascaded SSD network and monocular depth estimation of the present invention. DETAILED DESCRIPTION

[0038] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the above detection network schematic diagram. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment provides a forward vehicle distance detection method based on cascaded SSD and monocular depth estimation, including the following steps:

[0041] Step 1: Obtain the forward vehicle training sample and depth estimation dataset, which mainly includes the following five parts:

[0042] 1) Download the KITTI dataset from the official website of the KITTI dataset. The dataset contains vehicle training images, annotation files, camera and radar calibration parameters and point cloud files corresponding to each image. The sparse 3D point cloud data of the radar novice needs to be projected from the lidar coordinate system to the pixel coordinate system, and then the linear interpolation algorithm is used to obtain the depth map;

[0043] 2) Clean the KITTI dataset and filter out and remove images with incorrect annotations. For example, images with serious occlusion between vehicles are still annotated, resulting in serious overlap of label frames and incorrect annotation of vehicle labels.

[0044] 3) Based on the k-means clustering method proposed in YOLO v2, this paper sets the Anchor shape through the following steps:

[0045] 1. Treat all target boxes as sets and set the number of cluster categories;

[0046] 2. Randomly select k target boxes as the original “center box”;

[0047] 3. Calculate the "IoU distance" between each non-"center box" target box and k "center boxes", and divide the target box into the "center box" category set with the smallest "IoU distance" In , i is the index of the “center box”. The “IoU distance” is defined as:

[0048]

[0049] Among them, area(C) and area(G) represent the areas of the two boxes, and “IoU distance” also represents the degree of overlap between the two boxes;

[0050] 4. For any Calculate the median width and height of all target boxes it contains as the new "center box".

[0051] 5. Determine whether the "IoU distance" between the new "center box" and the original "center box" of each category is less than the set threshold or whether the maximum number of iterations has been reached. If the conditions are met, proceed to the next step, otherwise repeat steps 3 to 5;

[0052] 6. Calculate the maximum IoU of each target box and the "center box" to get the average IoU;

[0053] 7. Traverse according to k = [2, 3, ..., 11, 12], repeat 2 to 6, and get 11 groups of k and average IoU. With k as the horizontal coordinate and average IoU as the vertical coordinate, find the inflection point that best balances speed and accuracy, and determine the number of categories k;

[0054] 4) By analyzing the depth information of the statistical data set, it can be found that the KITTI data set has a long-tail data distribution problem similar to that of target detection, which can guide the network to optimize during the training process;

[0055] 5) The KITTI dataset has about 7,500 images. 80% of the dataset is randomly used as a training set and 20% as a test set.

[0056] Step 2: Building a backbone network mainly includes the following three parts:

[0057] 1) Select VGG-16 as the basis of the backbone network for feature extraction and make modifications on this basis;

[0058] 2) The original training image is downsampled to the resolution size of Conv4-3 (the resolution size obtained when the input is 300*300 is 38*38), and then a convolution layer group including 3 3*3 convolution layers with a step size of 1 is used to obtain the low-level feature map δ0 of the image target details; in order to obtain high-level features, the fc7 layer features are upsampled to the resolution size of Conv4-3 to obtain the feature map δ1; considering the redundancy of the fc7 layer in information, 128 1024×1×1 convolution kernels are added for online feature screening to achieve model pruning, and then Conv5-3 is deconvolved and upsampled to the resolution size of Conv4-3 to obtain the feature map δ2; the semantic information of the receptive field is supplemented, and finally the three feature layers δ0, δ1, and δ2 are fused with Conv4-3 to achieve feature enhancement and obtain the fused feature map ψ0;

[0059] 3) Take ψ0 as the input of the cascaded ASPP, and set the hole rate of ASPP to 2 and 3 respectively to enhance the receptive field of the feature layer and improve the detection robustness of small-scale or incomplete-looking targets. Then extract the feature layers ψ1 and ψ2 after each ASPP, superimpose ψ0, ψ1, and ψ2 on the channel, and make them consistent with the number of channels of the original Conv4-3 to obtain the feature map ψ.

[0060] Step 3: The construction of the target detection sub-network mainly includes the following three parts:

[0061] 1) The target detection sub-network is improved based on the SSD network. First, the number and size of anchor boxes in the detection network are changed to the size obtained in the above step 1;

[0062] 2) Based on the feature map ψ obtained in step 2 above, three additional convolutional layers are added. Each convolutional layer consists of a 1×1 convolution with a step size of 2 and a 3×3 convolution. The feature maps of four scales are used to construct a feature pyramid as the initial inspection network of the target detection network to classify the background and foreground. The final target detection network feature layer resolution size is shown in Table 1:

[0063] Table 1 Feature layer resolution

[0064] Resolution 38×38 19×19 10×10 5×5

[0065] 3) Based on the initial inspection network, deformable convolution is used for the feature map of each scale, and a learning-based method is used to obtain the offset of the convolution sampling point to complete the deformable convolution calculation, which can be used to alleviate the feature misalignment problem caused by the anchor frame mechanism and improve the regression accuracy of the detection frame. The initial inspection network outputs four variables: (dx, dy, dh, dw). Among them, (dx, dy) corresponds to the offset of the spatial position, and (dh, dw) represents the offset on the scale. A feature alignment module FAM can be designed. The offset required by FAM is obtained by the convolution of dx and dy. The whole process of FAM is:

[0066] Δp i =f(dx i ,dy i )

[0067] FM i+1 =Deformable(FM i ,Δp i )

[0068] Among them, FM represents the feature map, f represents the convolution operation, and Deformable represents deformable convolution.

[0069] Step 4: Building the depth estimation subnetwork mainly includes the following two parts:

[0070] 1) Based on the feature map ψ obtained in step 2 above, three additional convolutional layers are added. Each convolutional layer consists of a 1×1 convolution with a step size of 2 and a 3×3 convolution. Feature layers with resolutions of 38*38, 19*19, 10*10, and 5*5 are obtained in turn to extract depth estimation features. Finally, the resolution of the feature layer σ0 is 5*5.

[0071] 2) Then, based on the feature layer σ0, the corresponding size feature layers in 1) are connected through identity mapping, and the deconvolution layer is used for upsampling. The feature layer resolutions obtained are 10*10, 19*19, 38*38, and 75*75, respectively. The final feature layer resolution is 1 / 4 of the original image size, and then the 75*75 feature layer is used as the detection network for depth estimation;

[0072] Step 5: Training of the forward vehicle distance detection network based on the cascaded SSD network and monocular depth estimation mainly includes the following three parts:

[0073] 1) Design the overall loss function L total , including the target detection loss function L detect , monocular estimation network loss function L depth And the perceptual loss function L DA , β is a hyperparameter, and its corresponding expression is:

[0074] L total =L detect +β(L depth +L DA )

[0075] Among them, the target detection loss function uses the SSD network loss function:

[0076]

[0077] L conf represents the confidence loss, L loc represents the position loss, the balance coefficient λ is set to 1, and N is set to the total number of positive samples in a batch to scale the loss;

[0078] Monocular estimation network loss function L depth It is expressed as:

[0079]

[0080] D represents the predicted depth image, D * represents the original depth image, i represents the pixel index, and in order to retain similar local features, the horizontal and vertical gradients are calculated;

[0081] Perceptual loss function L DA It is used to optimize the long-tail distribution problem of the KITTI dataset, which is specifically expressed as follows:

[0082]

[0083] Among them, N represents the number of pixels, i represents the pixel index, l represents the distance metric, and d i represents the predicted depth value, Indicates the true depth value, α D Represents the depth-aware attention term, which is used to guide the network to focus on the long-tail depth distribution. λ D Is a regular term, used to prevent the gradient from disappearing, and the calculation formula is:

[0084]

[0085] 2) The network input image size is set to 300*300, the initial learning rate is set to 0.003, the learning rate is adjusted to polynomial decay, and the batch-size is set to 16;

[0086] 3) The model training was first iterated 500,000 times, then the depth estimation network branch was suppressed, only the target detection branch was iterated 150,000 times, and then the entire network was iterated 200,000 times;

[0087] Step 6: Selection of feature points and distance detection based on K-means clustering

[0088] 1) Using the original image as input, the trained model is used to obtain the vehicle detection confidence and detection box in the image, as well as the monocular depth image of the image;

[0089] 2) Get the coordinates of the detection box (x1, y1, x2, y2). The coordinates of the center point of the detection box are expressed as:

[0090]

[0091] The midpoint horizontal coordinate is used to divide the vehicle into two parts, the lower part, which contains most of the information of the rear or front of the vehicle and is also the closest point between vehicles. This is used as the depth estimation cluster set;

[0092] The K-means clustering algorithm is used to cluster the set obtained in step 2), mainly into two categories: foreground and background. Finally, the center point of the foreground cluster is used as the ranging feature point and the distance information is output.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A forward vehicle distance detection method based on cascaded SSD and monocular depth estimation, characterized in that: The following steps are involved: S1, obtain forward vehicle training samples and depth estimation datasets; S2. Build a backbone network; S21, select VGG-16 as the basis of the backbone network for feature extraction, and make modifications on this basis; S22, downsample the training image to the resolution of Conv4-3, and then pass through three 3×3 convolutional layers to obtain the low-level feature map δ0 of the image target details; then upsample the fc7 layer features to the resolution of Conv4-3 to obtain the feature map δ1, and then upsample Conv5-3 through deconvolution to obtain the feature map δ2, and finally fuse the three feature maps δ0, δ1, and δ2 with Conv4-3 in the channel dimension to achieve feature enhancement and obtain the fused feature map ψ0; S23, use ψ0 as the input of the cascaded ASPP, set the hole rates of ASPP to 2 and 3 respectively, extract the feature layers ψ1 and ψ2 after each ASPP, superimpose ψ0, ψ1, and ψ2 on the channel, and make them consistent with the number of channels of the original Conv4-3, and obtain the feature map ψ; S3, build the target detection network, that is, the cascaded SSD network; S31, the target detection sub-network is improved based on the SSD network. First, the number and size of the anchor boxes in the detection network are changed to the size obtained in S13; S32, based on the feature map ψ obtained in S23, add three additional groups of convolutional layers, each of which consists of a 1×1 convolution with a step size of 2 and a 3×3 convolution. The feature maps of the four scales are used to construct a feature pyramid as the initial inspection network of the target detection network to classify the background and foreground; S33. Based on the initial inspection network, deformable convolution is used for the feature maps of each scale to alleviate the feature misalignment problem caused by the anchor frame mechanism; S4, build a depth estimation sub-network; S41, based on the feature map ψ obtained in S23, through an additional 3 layers of convolutional layers, each group of convolutional layers consists of a 1×1 convolution with a stride of 2 and a 3×3 convolution, to obtain the feature layer σ0; S42, based on the feature layer σ0, up-sampling is performed through the identity mapping and the deconvolution layer, so that the size of the deconvolution image is reduced to 1 / 4 of the original image; S5, training of the forward vehicle distance detection network based on the cascaded SSD network and monocular depth estimation; S6. Selection of feature points and distance detection based on K-means clustering.

2. The forward vehicle distance detection method based on cascaded SSD and monocular depth estimation according to claim 1 is characterized in that: The S1 comprises the following steps: S11. Download the KITTI dataset, which contains vehicle training images, annotation files, camera and radar calibration parameters corresponding to each image, and point cloud files; S12. Clean the KITTI dataset, filter out the images with incorrect annotations and remove them; S13, use K-means clustering to select the appropriate number and aspect ratio of anchor boxes; S14. Count the depth information of the KITTI dataset to prepare for the subsequent optimization of the depth estimation model; S15. Randomly use 80% of the KITTI dataset as a training set and 20% as a test set.

3. The forward vehicle distance detection method based on cascaded SSD and monocular depth estimation according to claim 2 is characterized in that: The S5 comprises the following steps: S51. Design the overall loss function, including target detection loss function, monocular estimation network loss function and perception loss function; S52, set the network input image size to 300*300, the initial learning rate to 0.003, the learning rate to polynomial decay, and the batch-size to 16; S53. Iterate the model training 500,000 times, then suppress the depth estimation network branch, iterate the target detection branch 150,000 times, and then iterate the entire network 200,000 times.

4. The forward vehicle distance detection method based on cascaded SSD and monocular depth estimation according to claim 3 is characterized in that: The S6 comprises the following steps: S61, using the original image as input, obtaining the vehicle detection confidence and detection frame in the image and the monocular depth image of the image through the trained model; S62. Get the coordinates of the detection frame (x1, y1, x2, y2). The coordinates of the center point are expressed as: The horizontal coordinate of the center point is used to divide the vehicle into two parts, the lower part, which contains most of the information of the rear or front of the vehicle and is also the closest point between vehicles. This is used as the depth estimation cluster set; S63. Use the K-means clustering algorithm to cluster the set obtained in S62, mainly clustering it into two categories: foreground and background. Finally, use the center point of the foreground cluster as the distance measurement feature point and output the distance information.

Images