A binocular 3D target detection method

Abstract

The application discloses a binocular 3D target detection method, which comprises the following steps: left and right view features are cascaded on each disparity level to construct a disparity cost volume, and the disparity cost volume is mapped into a depth cost volume by using the inverse ratio relationship between the disparity value and the depth value; a binocular depth estimation network with 3D foreground prior embedding is constructed by a cost aggregation module, a 3D foreground segmentation module and a depth estimation module, the binocular depth estimation network is used to introduce the foreground prior knowledge contained in the 3D target detection label into the binocular depth estimation process, and a depth map suitable for the 3D target detection task is obtained; a 3D foreground segmentation mask is constructed by using the 3D target detection label, the 3D foreground segmentation mask is used to supervise the 3D foreground segmentation result, a network optimization loss function is constructed, a dynamic weight average strategy is adopted, the weight coefficients corresponding to different loss items are determined according to the change degree of the loss items, and the binocular 3D target detection network is trained. The application improves the prediction quality of the foreground region pseudo point cloud and obtains accurate binocular 3D target detection results.

Description

Technical Field

[0001] This invention relates to the fields of target detection and computer vision, and in particular to a binocular 3D target detection method. Background Technology

[0002] As a core algorithm in fields such as autonomous driving and robotic perception, 3D object detection plays a crucial role in detecting pedestrians, vehicles, and obstacles. Based on the different types of data collected by sensors, existing 3D object detection methods can be categorized into point cloud-based, monocular image-based, and binocular image-based methods. Point clouds are acquired using LiDAR, which is costly; monocular images are acquired by a single camera, and their object detection performance is limited due to a lack of depth information. In contrast, binocular images are acquired by binocular cameras, offering lower acquisition costs while containing richer scene depth information. Therefore, researching 3D object detection methods based on binocular images is of significant importance and practical application value.

[0003] In recent years, binocular 3D object detection methods typically rely on binocular stereo matching or depth estimation algorithms to predict depth maps, convert these depth maps into pseudo-point clouds, and then use point cloud-based 3D object detection algorithms to obtain the detection results. With advancements in binocular stereo matching, binocular depth estimation, and point cloud-based 3D object detection algorithms, the accuracy of binocular 3D object detection methods has continuously improved. For example, Wang et al. used binocular stereo matching to predict depth maps, converted them into pseudo-point clouds, and then used point cloud-based 3D object detection algorithms to obtain the final detection results. Pon et al. predicted the depth map of the target region through stereo matching of the foreground region and converted it into a pseudo-point cloud for 3D object detection. In these binocular 3D object detection methods, the quality of the pseudo-point cloud has a significant impact on the final detection performance. To improve the quality of the pseudo-point cloud, researchers have conducted further studies. Considering the distribution differences between the foreground and background in binocular images, Li et al. proposed a method that estimates the depth maps of the foreground and background separately and uses confidence scores to filter high-quality pseudo-point clouds, effectively alleviating the artifact problem of pseudo-point clouds and improving the accuracy of binocular 3D object detection. Garg et al. designed a continuous parallax network to obtain continuous stereo matching results, thereby obtaining accurate depth maps and effectively improving the accuracy of binocular 3D object detection.

[0004] However, the aforementioned work separates the depth estimation process from the 3D object detection task, and the predicted depth map is difficult to fully match the requirements of the 3D object detection task. Summary of the Invention

[0005] This invention provides a binocular 3D object detection method. The method incorporates foreground location information indicated by 3D object detection labels, which helps the network obtain more accurate depth maps of the foreground region, thereby improving the performance of binocular 3D object detection. By constructing a binocular depth estimation network with 3D foreground prior embedding, the foreground prior of the 3D object detection labels is introduced into the binocular depth estimation process, improving the prediction quality of pseudo-point clouds in the foreground region and thus obtaining more accurate binocular 3D object detection results. See the description below for details:

[0006] A binocular 3D target detection method, the method comprising:

[0007] The left and right view features are cascaded at each disparity level to construct the disparity cost volume. The disparity cost volume is mapped to the depth cost volume by utilizing the inverse relationship between disparity values ​​and depth values.

[0008] A stereo depth estimation network with 3D foreground prior embedding is constructed, consisting of a cost aggregation module, a 3D foreground segmentation module, and a depth estimation module. The stereo depth estimation network is used to introduce the foreground prior knowledge contained in the 3D object detection label into the stereo depth estimation process to obtain a depth map adapted to the 3D object detection task.

[0009] A 3D foreground segmentation mask is constructed using 3D object detection labels to supervise the 3D foreground segmentation results. A network optimization loss function is constructed, and a dynamic weight averaging strategy is adopted to determine the corresponding weight coefficients based on the degree of change of different loss terms. Simultaneously, a stereo 3D object detection network is trained.

[0010] The 3D foreground segmentation module is as follows:

[0011] Each optimized depth cost volume is projected onto the camera coordinate system to construct its corresponding 3D cost volume. P-layer 3D convolution is used to extract discriminative information from the 3D cost volume. A foreground segmentation head containing Q-layer 3D convolution is used to predict the 3D foreground segmentation result, while constraining the 3D foreground segmentation prediction result.

[0012] The prediction of 3D foreground segmentation results using a foreground segmentation head containing Q-layer 3D convolutions is as follows:

[0013] For the first foreground segmentation head, the input is the first 3D cost volume. For the ith foreground segmentation head, the input is the sum of the ith 3D cost volume and the (i-1)th 3D cost volume.

[0014] Preferably, the trained binocular 3D object detection network is:

[0015] The first stage input consists of left and right view images, which are used to train the binocular view feature extraction module, cost aggregation module, 3D foreground segmentation module, and depth estimation module until convergence.

[0016] The second stage trains the point cloud 3D object detection module until convergence, with the input being the downsampled pseudo point cloud.

[0017] The beneficial effects of the technical solution provided by this invention are:

[0018] 1. This invention introduces prior knowledge of the foreground into the depth estimation process by designing a 3D foreground segmentation module, thereby improving the depth estimation accuracy of the foreground region and obtaining more accurate binocular 3D target detection results.

[0019] 2. This invention utilizes 3D object detection labels to construct a 3D foreground segmentation mask for supervising the 3D foreground segmentation module. No additional labels are required. By dynamically adjusting the weights of different loss terms, a balanced optimization between different tasks is achieved.

[0020] 3. This invention constructs a 3D cost volume by projecting the depth cost volume from the pixel coordinate system to the camera coordinate system, which effectively eliminates the influence of the inherent spatial distortion of the pixel coordinate system on 3D foreground segmentation, helps to perceive distant foregrounds, and thus improves the effect of embedding prior knowledge of distant foregrounds.

[0021] 4. This invention constructs a residual prediction structure for the 3D foreground segmentation module and the depth estimation module. It reduces the learning difficulty through a multi-level prediction method and strengthens the constraint effect of the loss function by supervising the prediction results of each level, thereby improving the prediction performance.

[0022] 5. In view of the problem of the imbalance between the number of foreground and background voxels in 3D foreground segmentation, this invention constructs a loss function based on focal loss to enhance the focus on foreground voxels, thereby promoting the embedding of foreground prior information. Attached Figure Description

[0023] Figure 1 A flowchart of a binocular 3D target detection method;

[0024] Figure 2 A schematic diagram of a binocular depth estimation network structure for 3D foreground prior embedding. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0026] To address the problems existing in the background art, the embodiments of the present invention consider that 3D object detection labels contain information such as foreground position and category, and effectively utilize the foreground prior knowledge contained in the 3D object detection labels, which helps to obtain a more accurate depth map of the foreground region. Therefore, introducing the foreground prior knowledge contained in the 3D object detection labels into the binocular depth estimation process helps to obtain a depth map that is more suitable for the 3D object detection task, thereby effectively improving the accuracy of binocular 3D object detection.

[0027] Therefore, this invention proposes a binocular 3D target detection method. By designing a binocular depth estimation network with 3D foreground prior embedding, the quality of the depth map of the foreground region is improved by utilizing the foreground prior, thereby obtaining more accurate binocular 3D target detection results.

[0028] This invention provides a binocular 3D target detection method, see [link to relevant documentation]. Figure 1 and Figure 2 The method includes:

[0029] I. Constructing a Deep Cost System

[0030] First, a binocular view feature extraction module is constructed, which consists of two convolutional neural networks with identical structures and shared parameters. The left and right view images are used as input to the constructed binocular view feature extraction module to obtain the corresponding left and right view features F. l With F r .

[0031] Subsequently, the acquired left and right view features F l With F r The disparity cost volume is constructed by cascading at each disparity level. Then, the disparity cost volume is mapped to the depth cost volume using the inverse relationship between the disparity value and the depth value.

[0032] The construction technique of the parallax cost body is well known to those skilled in the art, and will not be described in detail in the embodiments of the present invention.

[0033] II. Constructing a Binocular Depth Estimation Network with 3D Foreground Prior Embedding

[0034] (1) Constructing a cost aggregation module

[0035] Furthermore, a cost aggregation module is constructed to optimize the acquired depth cost volume. This module is implemented using a 3D convolutional neural network, specifically consisting of N 3D convolutional layers and M consecutive 3D hourglass networks, with each 3D hourglass network outputting an optimized depth cost volume. For each optimized depth cost volume, the corresponding 3D foreground segmentation and depth map are subsequently predicted by a 3D foreground segmentation module and a depth estimation module, respectively.

[0036] The term 3D hourglass network is well-known to those skilled in the art, and will not be elaborated upon in the embodiments of the present invention.

[0037] Furthermore, the optimized depth cost volume, compared to the depth cost volume described in the first part, is simply optimized using a well-known 3D hourglass network (similar to feature extraction). The data structure remains unchanged, both being tensors of (C, H, W, D) dimensions. Here, C is the number of channels, H is the height, W is the width, and D is the depth.

[0038] (2) Constructing a 3D foreground segmentation module

[0039] Subsequently, a 3D foreground segmentation module is constructed to predict the corresponding 3D foreground segmentation result based on each optimized depth cost volume, in order to learn foreground discriminative information and thus improve the accuracy of foreground depth estimation. The 3D foreground segmentation module designed in this embodiment is used to predict the position of foreground voxels in the camera coordinate system. Considering that the optimized depth cost volume is in the pixel coordinate system, its geometry is distorted relative to the normal 3D space in the camera coordinate system, resulting in insufficient attention to distant foreground voxels by the 3D foreground segmentation module.

[0040] To address the aforementioned issues, each optimized depth cost volume is first projected onto the camera coordinate system to construct its corresponding 3D cost volume. Taking the m-th optimized depth cost volume as an example, the projection method is shown in equations (1)-(3):

[0041]

[0042]

[0043]

[0044] Among them, f x and f y c represents the horizontal and vertical focal lengths of the camera, respectively. u and c v These represent the x and y coordinates of the image center point, respectively. and This represents the coordinates of the m-th optimized depth cost volume. and This represents the coordinates of its corresponding 3D cost volume. After the above transformation, the m-th optimized depth cost volume is located in... Features at coordinates will be projected onto the 3D cost volume. At the coordinates.

[0045] After obtaining the 3D cost volume, this embodiment of the invention employs P-layer 3D convolution to extract discriminative information from the 3D cost volume. Subsequently, a foreground segmentation head containing Q-layer 3D convolution is used to predict the 3D foreground segmentation result. The foreground segmentation head is constructed using a residual connection method.

[0046] Specifically, for the first foreground segmentation head, the input is the first 3D cost volume; for the ith foreground segmentation head, the input is the sum of the ith 3D cost volume and the (i-1)th 3D cost volume. By constraining the 3D foreground segmentation prediction results, it is helpful to embed the foreground prior knowledge in the 3D object detection label into the depth cost volume, enriching its foreground discriminative information and contributing to the accurate depth estimation of the foreground region in the subsequent process.

[0047] (3) Constructing a depth estimation module

[0048] After obtaining the depth cost volume with embedded foreground prior knowledge, a depth estimation module is constructed to obtain the depth map. Specifically, the first optimized depth cost volume directly predicts the depth map through a depth estimation head composed of K layers of 3D convolutions. Subsequent optimized depth cost volumes predict the offset relative to the previous depth map through the depth estimation head. The sum of the offset and the previous depth map is the depth map of the optimized depth cost volume. The depth map obtained by the last optimized depth cost volume is used as the final predicted depth map.

[0049] III. Constructing a Point Cloud 3D Target Detection Module

[0050] After obtaining the depth map prediction results, this embodiment of the invention projects the depth map into a three-dimensional space to generate a pseudo point cloud for subsequent 3D target detection.

[0051] The operation of projecting the depth map into three-dimensional space is shown in formulas (4)-(6):

[0052] z=D(h,w) (4)

[0053]

[0054]

[0055] Where D represents the depth map, w and h represent the horizontal and vertical coordinates of each pixel on the depth map, respectively, and x, y, and z represent the pseudo-point cloud coordinates of each pixel on the depth map projected into the three-dimensional space.

[0056] The amount of data in a pseudo-point cloud is several times that of a real radar-acquired point cloud. Directly feeding it as input into a point cloud 3D target detector would result in a huge computational burden. Therefore, the pseudo-point cloud is first downsampled before being fed into a pre-defined point cloud 3D target detector to obtain the final 3D target detection result.

[0057] IV. Constructing the Network Optimization Loss Function

[0058] In the 3D foreground segmentation module, this embodiment of the invention utilizes 3D object detection labels to construct a 3D foreground segmentation mask to supervise the 3D foreground segmentation results. Specifically, the set of foreground voxels contained within the ground truth 3D bounding box is calculated. Voxels located within the ground truth 3D bounding box are defined as foreground voxels and assigned a label of 1, while the remaining voxels are defined as background voxels and assigned a label of 0. Considering that foreground voxels occupy a relatively small proportion and there is an imbalance in the number of foreground and background voxels, a classification loss function to address class imbalance is adopted, expressed by the formula:

[0059]

[0060] Where FL(*) represents focal loss, seg′ i Let seg represent the prediction result of the i-th foreground segmentation head, where seg represents the 3D foreground segmentation mask, and α represents the prediction result of the i-th foreground segmentation head. i These represent weighting coefficients. In this embodiment of the invention, α1, α2, and α3 are set to 0.5, 0.7, and 1.0, respectively.

[0061] In the depth estimation module, this embodiment of the invention utilizes a real depth map constructed from real point cloud data to supervise the training of the module. Consistent with 3D foreground segmentation, the prediction results of different depth estimation heads are weighted by preset weight coefficients, expressed by the following formula:

[0062]

[0063] Where SmoothL1(*) represents the smoothing L1 loss function, dep′ j This represents the prediction result of the j-th depth estimation head, where dep represents the true depth map, and β... j This represents the weighting coefficients. In this embodiment of the invention, β1, β2, and β3 are set to 0.5, 0.7, and 1.0, respectively.

[0064] To dynamically adjust the weights of each loss term during training, this embodiment of the invention employs a dynamic weight averaging strategy, determining the corresponding weight coefficient based on the degree of change of different loss terms. Specifically, loss terms with small changes in the loss function are assigned larger weight coefficients to encourage learning, while those with large changes receive smaller weight coefficients to slow down learning.

[0065] The formula for the weighting coefficient is as follows:

[0066]

[0067] Where, r k(t-1) represents the degree of loss change of the k-th loss term in the (t-1)-th iteration, and T represents the temperature coefficient, which is set to 2.0 in this embodiment of the invention. ω k (t) represents the weight coefficient of the k-th loss term in the t-th iteration.

[0068] Based on this, the loss function of the binocular depth estimation network with 3D foreground prior embedding is:

[0069] L=ω0(t)L dep +ω1(t)L seg (10)

[0070] Wherein, ω0(t) and ω1(t) are obtained from formula (9), which are the loss weights of the depth estimation module and the loss weights of the 3D foreground segmentation module, respectively.

[0071] In the 3D point cloud object detection module, the loss function is:

[0072]

[0073] Among them, L cls L represents the classification loss. bbox Let D represent the regression loss, and let D represent the set of all candidate boxes. pos Let D represent the set of positive samples.

[0074] V. Training a binocular 3D target detection network

[0075] The binocular 3D object detection network proposed in this embodiment includes: a binocular view feature extraction module, a cost aggregation module, a 3D foreground segmentation module, a depth estimation module, and a point cloud 3D object detection module. The network training is divided into two stages. The first stage trains the binocular view feature extraction module, cost aggregation module, 3D foreground segmentation module, and depth estimation module until convergence. The input for this stage is the left and right view images, and the training process is constrained by the loss function shown in formula (10). The second stage trains the point cloud 3D object detection module until convergence. The input for this stage is the downsampled pseudo-point cloud, and the training process is constrained by the loss function shown in formula (11). During the inference stage, the left and right view images are used as input to the binocular 3D object detection network to obtain the corresponding 3D object detection results in an end-to-end manner.

[0076] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A binocular 3D target detection method, characterized in that, The method includes: The left and right view features are cascaded at each disparity level to construct the disparity cost volume. The disparity cost volume is mapped to the depth cost volume by utilizing the inverse relationship between disparity values ​​and depth values. A stereo depth estimation network with 3D foreground prior embedding is constructed, consisting of a cost aggregation module, a 3D foreground segmentation module, and a depth estimation module. The stereo depth estimation network is used to introduce the foreground prior knowledge contained in the 3D object detection label into the stereo depth estimation process to obtain a depth map adapted to the 3D object detection task. A 3D foreground segmentation mask is constructed using 3D object detection labels to supervise the 3D foreground segmentation results. A network optimization loss function is constructed, and a dynamic weight averaging strategy is adopted to determine the corresponding weight coefficients based on the degree of change of different loss terms. Simultaneously, a stereo 3D object detection network is trained.

2. The binocular 3D target detection method according to claim 1, characterized in that, The 3D foreground segmentation module is: Each optimized depth cost volume is projected onto the camera coordinate system to construct its corresponding 3D cost volume. Layer 3D convolution is used to extract discriminative information of the 3D cost volume, utilizing layers containing The 3D convolutional foreground segmentation head predicts the 3D foreground segmentation result and simultaneously constrains the 3D foreground segmentation prediction result.

3. The binocular 3D target detection method according to claim 2, characterized in that, The projection is: ； ； ； in, and These represent the camera's horizontal and vertical focal lengths, respectively. and These represent the x and y coordinates of the image center point, respectively. , and Indicates the first The coordinates of an optimized depth cost volume. , and This represents the coordinates of the corresponding 3D cost body.

4. The binocular 3D target detection method according to claim 2, characterized in that, The use includes The 3D foreground segmentation result predicted by the 3D convolutional foreground segmentation head is as follows: For the first foreground segmentation head, the input is the first 3D cost volume; for the second... The nth foreground segmentation head, input is the nth... The 3D cost body and the first The sum of the costs of each 3D body.

5. The binocular 3D target detection method according to claim 1, characterized in that, The network optimization loss function is: ； in, Represents focal loss. Indicates the first Prediction results for each foreground segmentation head. This represents a 3D foreground segmentation mask. Indicates the weighting coefficient. The number of 3D hourglass networks.

6. The binocular 3D target detection method according to claim 5, characterized in that, The weighting coefficients corresponding to different loss terms are determined as follows: ； in, Indicates the first The loss item is in the first The degree of change in loss in each iteration Indicates the temperature coefficient. Indicates the first The loss item is in the first The weighting coefficients for the next iteration; The loss function of the stereo depth estimation network with 3D foreground prior embedding is: ； in, This is for the depth estimation module loss.

7. The binocular 3D target detection method according to claim 5, characterized in that, The trained binocular 3D object detection network is: The first stage input consists of left and right view images, which are used to train the binocular view feature extraction module, cost aggregation module, 3D foreground segmentation module, and depth estimation module until convergence. The second stage trains the point cloud 3D object detection module until convergence, with the input being the downsampled pseudo point cloud.

Images