An automatic driving 3D target detection method and device based on an object imaging method

Abstract

The application relates to an automatic driving 3D target detection method and device based on an object imaging method, wherein the automatic driving 3D target detection method based on the object imaging method comprises the following steps: acquiring an RGB image in a driving scene; performing depth estimation on the RGB image to obtain a corresponding depth map; performing feature extraction and image reconstruction on the depth map and the RGB image respectively to obtain an image containing only an object of interest; performing mask processing on the depth map by taking the image containing only the object of interest as a mask, and combining coordinate information of the image containing only the object of interest to generate corresponding point cloud data; and completing decision selection of automatic driving according to the image containing only the object of interest and the point cloud data. The automatic driving 3D target detection method based on the object imaging method has the advantages that the problems of excessive image redundant data, large calculation amount and poor real-time performance in the prior art target detection method can be effectively solved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to the field of 3D target detection for autonomous driving based on object imaging methods. Background Technology

[0002] Autonomous driving, as an integrated technology, is considered to have the potential to revolutionize the industry and drastically change human lifestyles. In modern autonomous driving systems, the perception system is an indispensable component, designed to accurately estimate the state of the surrounding environment and provide reliable observations for prediction and planning. 3D object detection can intelligently predict the location, size, and category of key 3D objects near autonomous vehicles, while 2D object detection algorithms based on RGB images face relatively greater challenges in prediction; therefore, research has shifted towards 3D object detection.

[0003] Current mainstream 3D object detection algorithms are mainly based on point cloud and multimodal fusion, and their limitations are as follows:

[0004] The algorithm is highly dependent on point cloud data, and its performance is greatly limited by the quality of the point cloud data. The configuration of the radar has a significant impact on the acquisition of point cloud data. Low-line-count radars can only collect sparse point clouds, which will miss a lot of key information in the scene, resulting in a significant drop in algorithm performance.

[0005] Point clouds are sparse and irregular 3D representations, containing mostly useless point cloud data. Specially designed models are needed for feature extraction, which is computationally intensive. Redundant point cloud data makes it difficult for algorithms to achieve high efficiency and meet the requirements of real-time inference.

[0006] Image-based methods require extracting 3D information of objects from 2D image data, lack sufficient geometric constraints, have low prediction performance, and are difficult to predict, still lagging far behind LiDAR methods.

[0007] For multimodal fusion algorithms, the complexity of LiDAR-based and camera-based detection systems means that combining the two modalities inevitably leads to additional computational overhead and inference time delay.

[0008] All algorithms directly predict the target's position and shape information from the data, lacking intuitive visualization effects. Summary of the Invention

[0009] Based on this, the purpose of this invention is to propose a method and device for 3D target detection in autonomous driving based on object imaging, combining deep learning technology and object of interest recognition methods. It utilizes object imaging technology and leverages the powerful feature extraction capabilities of deep neural networks to learn the feature information of objects of interest, such as vehicles on a road. The deep neural network is trained using a self-designed loss function and a reconstructed road dataset. Based on object imaging technology, useless data other than the target of interest can be filtered out, achieving rapid target detection and reducing the computational load of subsequent operations. The RGB image after object imaging processing achieves target detection while simultaneously filtering the background, effectively solving the problems of existing technologies such as reliance on expensive equipment like radar, excessive redundant data, high computational load, and poor real-time performance.

[0010] This application is implemented through the following technical solution.

[0011] On the one hand, this application provides a 3D target detection method for autonomous driving based on object imaging, which includes the following steps:

[0012] Acquire RGB images from the driving scene;

[0013] Depth estimation is performed on the RGB image to obtain the corresponding depth map;

[0014] Feature extraction and image reconstruction are performed on the depth map and the RGB image respectively to obtain an image containing only the object of interest;

[0015] The depth map is masked using an image containing only the object of interest as a mask, and the corresponding point cloud data is generated by combining the coordinate information of the image containing only the object of interest.

[0016] The autonomous driving decision is made based on the image containing only the object of interest and the point cloud data.

[0017] Furthermore, feature extraction and image reconstruction are performed on the depth map and the RGB image respectively to obtain an image containing only the object of interest, specifically including:

[0018] The depth map and the RGB image are respectively subjected to feature extraction and downsampling compression to obtain the depth map branch feature map and the RGB image branch feature map;

[0019] The depth map branch feature map and the RGB image branch feature map are fused across modalities to obtain an image containing only the object of interest.

[0020] Furthermore, features are extracted from the RGB image using the following formula:

[0021] y = σ(W*f + b)

[0022] Where y represents the feature map after feature extraction through the convolutional layer, σ represents the non-linear activation function, and W represents the convolutional kernel R. n×n ,* represents the convolution operation between the convolution kernel and the image, and f represents the input image R. m×n , b represents linear bias.

[0023] Furthermore, the feature filter is based on a preset dataset {X,Y} and a loss function. The dataset {X,Y} obtained through training and optimization consists of contour data of various objects of interest.

[0024] Furthermore, the loss function Designed for mask-weighted smoothness L1 ,in:

[0025]

[0026] l i,j =smooth L1 (y i, -x i, )

[0027] Where y i, To output the image; then, l is processed using the following formula. i,j Perform weighted processing:

[0028]

[0029] Where α is an adjustable weighting factor.

[0030] Furthermore, the depth map is masked using an image containing only the object of interest as a mask, which is achieved through the following formula:

[0031]

[0032] Where a i, These are matrix elements representing the three RGB channels;

[0033]

[0034] p i, These are elements in the depth map.

[0035] On the other hand, this application also provides an autonomous driving 3D target detection device based on an object imaging method, which includes: an image acquisition module for acquiring RGB images in a driving scene;

[0036] Depth extraction module: Performs depth estimation on the RGB image to obtain the corresponding depth map;

[0037] Target object imaging module: performs feature extraction and image reconstruction on the depth map and the RGB image respectively to obtain an image containing only the object of interest;

[0038] Depth map masking module: The depth map is masked using an image containing only the object of interest as a mask, and the corresponding point cloud data is generated by combining the coordinate information of the image containing only the object of interest.

[0039] Driving decision module: Makes autonomous driving decisions based on the image containing only the object of interest and the point cloud data.

[0040] Furthermore, the target object imaging module includes:

[0041] Feature map extraction submodule: Performs feature extraction and downsampling compression on the depth map and the RGB image respectively to obtain depth map branch feature map and RGB image branch feature map;

[0042] Modal fusion submodule: used to perform cross-modal fusion of the depth map branch feature map and the RGB image branch feature map to obtain an image containing only the object of interest.

[0043] Furthermore, the feature map extraction submodule extracts features from the RGB image using the following formula:

[0044] y = σ(W*f + b)

[0045] Where y represents the feature map after feature extraction through the convolutional layer, σ represents the non-linear activation function, and W represents the convolutional kernel R. n×n ,* represents the convolution operation between the convolution kernel and the image, and f represents the input image R. m×n , b represents linear bias.

[0046] Furthermore, the image acquisition module is provided with multiple components.

[0047] This application provides a method and apparatus for 3D target detection in autonomous driving based on object imaging. By combining object imaging technology with a deep learning network, it can acquire specific object images from RGB images and filter background data to obtain RGB images containing only the object of interest. The resulting depth map is then filtered based on the RGB image containing only the object of interest, further reducing the amount of computational data and improving image processing efficiency. Furthermore, it provides an implementation method for training a feature filter fitted by a CNN and for designing a loss function to optimize the feature filter, resulting in a more optimized feature filter and improving the accuracy of feature extraction and object of interest recognition.

[0048] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0049] Figure 1 This application provides a flowchart of an autonomous driving 3D target detection method based on object imaging.

[0050] Figure 2 This is a structural block diagram of an autonomous driving 3D target detection device based on an object imaging method provided in this application.

[0051] The attached diagrams in the instruction manual are as follows: 10: Image acquisition module; 20: Depth extraction module; 30: Target imaging module; 302: Feature map extraction submodule; 304: Modality fusion submodule; 40: Depth map masking processing module; 50: Driving decision module. Detailed Implementation

[0052] This application addresses the problems of existing 3D target detection algorithms for autonomous driving technology, which suffer from strong dependence on point cloud data, performance greatly limited by the quality of the point cloud data, significant impact of radar configuration on point cloud data acquisition, and waste of computational resources due to the inclusion of a large amount of irrelevant background data, resulting in slow target detection speed. It proposes a 3D target detection method based on a monocular camera, which effectively solves the problems of dependence on expensive equipment such as radar, excessive redundant data, high computational load, and poor real-time performance of existing solutions by leveraging object imaging technology and deep learning networks.

[0053] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart of the autonomous driving 3D target detection method based on object imaging method of this application; Figure 2 for Figure 1 The diagram shows the structural block diagram of an autonomous driving 3D target detection device based on an object imaging method. The autonomous driving 3D target detection method based on an object imaging method of this application specifically includes the following steps:

[0054] S10: Acquire the RGB image of the driving scene. Step S10 is executed by the image acquisition module 10.

[0055] RGB images of the driving scene are captured using a monocular camera, and the acquired images are input into a depth estimation network. In another implementation, multiple image acquisition modules 10 are added to perform observations and improve the prediction results of the depth estimation.

[0056] S20: Perform depth estimation on the RGB image to obtain the corresponding depth map. Step S20 is executed by the depth extraction module 20.

[0057] Specifically, depth estimation refers to estimating the distance between all pixels in an input RGB image and the camera. This solution uses the MonoDELSNet monocular depth estimation algorithm based on deep learning, proposed by Akhil Gurram, to perform the depth estimation operation. Due to current limitations in the development of depth cameras, effective ranging is limited to within ten meters. Therefore, effective depth information cannot be collected in outdoor scenes. Current outdoor driving scene depth datasets all use LiDAR for depth ranging and then generate sparse depth maps through coordinate system transformation, in which the depth values ​​of most pixels are missing.

[0058] Current monocular depth estimation algorithms, when training in outdoor scenes, only calculate the loss for effective pixels in the sparse depth ground truth (GT) map, ignoring pixels with missing depth. This deficiency is compensated for by training with a large number of samples. In contrast, MonoDELSNet uses a game engine to create virtual road scenes, thus obtaining a complete dense GT map for training. Through joint training using self-supervision of the real-world scene and supervision of the virtual scene based on SFM, it effectively overcomes this deficiency in current algorithms, exhibiting better robustness and scene adaptability in practical applications compared to other algorithms.

[0059] S30: Feature extraction and image reconstruction are performed based on the depth map and the RGB image respectively to obtain an image containing only the object of interest. Step S30 is executed by the target object imaging module 30.

[0060] Specifically, S30 includes:

[0061] S302: The depth map and the RGB image are subjected to feature extraction and downsampling compression respectively to obtain the depth map branch feature map and the RGB image branch feature map. Step S302 is executed by the feature map extraction submodule 302.

[0062] S304: Perform cross-modal fusion of the depth map branch feature map and the RGB image branch feature map to obtain an image containing only the object of interest. Step S304 is executed by the modal fusion submodule 304.

[0063] Specifically, the image segmentation network CMX is modified to complete the object imaging operation. The CMX network takes RGB images and other types of data corresponding to the same scene as RGB, such as depth maps and heat maps (we use depth maps obtained by depth estimation), as inputs and feeds them into two branches of the CMX imaging network for feature extraction, generating RGB images that filter out redundant backgrounds and only retain the specific target.

[0064] Specifically, an image contains M objects, each represented by f. i(i = 1, 2, 3...M) are used to represent this. Therefore, an RGB image f of a driving scene is represented by the following formula:

[0065]

[0066] In an N-dimensional sparse space, each object f i (i = 1, 2, 3...M) can be represented by the following formula:

[0067]

[0068] Where Ψ is the sparse basis of the N-dimensional space, and x is the sparse coefficient in the sparse space. Different coefficients represent different objects.

[0069] Combining the two equations, an RGB image f of a driving scene can be represented as:

[0070]

[0071] Compared to traditional image processing algorithms, deep learning's powerful fitting ability can help us quickly design an optimal feature filter. This is achieved by utilizing a specially designed dataset {X,Y} (in this application, the dataset consists of various vehicle contour styles) and a loss function. Supervised training is performed using inverse gradient descent to minimize the loss function. This helps the CNN network learn the optimal parameters θ for extracting only the semantic features of f0, and then uses the CNN network to fit the optimal feature filter Φ. * For example:

[0072] Specifically, the input RGB image set X, where each image can be viewed as a three-channel two-dimensional matrix, can be represented as R. m×n A matrix with m rows and n columns, where each element a i,j This represents a pixel at a corresponding location in the image. For each pixel 'a'... i,j This task requires simultaneous detection and regression operations. Based on this pixel-level objective, the CNN network used to fit the feature filter can be designed as a fully convolutional network. A fully convolutional network can be divided into two parts: 1. a backbone for feature extraction; 2. a recovery part that restores the image size and performs classification and regression. In the convolutional layers of the backbone, feature extraction and data compression are achieved by sliding convolutional kernels across the image and summing the results. Feature extraction is performed using the following formula:

[0073] y = σ(W*f + b)

[0074] y represents the feature map after feature extraction through the convolutional layer, σ represents the non-linear activation function, and R represents the convolutional kernel.n×n ,* represents the convolution operation between the convolution kernel and the image, and f represents the input image R. m×n , b represents linear bias.

[0075] The recovery part involves analyzing the category feature map R1, which contains high-level semantic information. c1×m′×n′ Upsampling expands the feature map size and combines it with R2, which contains low-level semantic pixel feature maps. c2×m′×n′ By using skip connections, channels are concatenated to summarize high-level and low-level features, and then the features are fused through convolutional layers to generate a new feature map R3. (c1+c2)×m1×n1 Repeat the operation multiple times until the feature map is restored to the original image size, thus achieving image reconstruction containing only the target object.

[0076] S40: Using the image containing only the object of interest as a mask, perform masking processing with the depth map to generate corresponding point cloud data.

[0077] The dense depth map generated after depth estimation suffers from low accuracy in many complex backgrounds due to limitations in the performance of current monocular depth estimation algorithms and datasets. Furthermore, its predictions outside the target of interest (vehicle) are highly redundant, hindering subsequent data processing and direct observation. Combining object imaging with this method can effectively address these issues. First, the output image after target detection in object imaging serves as a mask. Then, a filtering operation is performed on the obtained depth map. Specifically, the image output by object imaging is a three-channel two-dimensional matrix R. 3×m×n The matrix elements corresponding to each position and the three channels are a, respectively. 1,i,j ,a 2,i,j ,a 3,i,j We merge the color information from the three RGB channels into a single channel, with each matrix element being a. i,j =a 1,i,j +a 2,i,j +a 3,i,j Let 20 be the lower limit for a. i,j Thresholding is performed to filter out pixels that are not of interest, and then a masking operation is performed. Using the generated 0,1 matrix as a mask, the generated depth map is multiplied to perform masking. Let the dense depth map be R. m×n Its element y i,j Perform a multiplication operation. This generates a sparse depth map after masking to filter out redundant backgrounds. This depth map contains only the depth information of the pixels corresponding to the target of interest. The depth information y in the depth map... i,j By combining the coordinate information i and j, a projection operation is performed onto the real world to generate a pseudo-point cloud. Because most redundant points are filtered out, the operation of generating a pseudo-point cloud through projection requires very little computation.

[0078] S50: Make a decision on autonomous driving based on the RGB image containing only the object of interest and the point cloud data.

[0079] By combining RGB images with point cloud data, the distance and angle between each object of interest and the vehicle can be obtained. Based on the distance between the vehicles and the corresponding angle, the vehicle speed can be adjusted and the driving direction can be changed to avoid collisions. The RGB images and point cloud data are then visualized for the car owner to view.

[0080] In another embodiment, we removed the softmax operation after cross-modal fusion. In traditional 3D object construction, to normalize and ensure the sum of class probabilities equals 1, a softmax operation is often added after cross-modal fusion, resulting in pixel values ​​in the output image ranging from 0 to 1. Object imaging is based on the object of interest, directly regressing the image's pixel values, which range from 0 to 255. Therefore, by removing the softmax function, the pixel values ​​at the corresponding object of interest locations in the reconstructed image are perfectly regressed to their original values, while pixel values ​​at non-object of interest locations are regressed to near 0, thus achieving background-filtered object detection. In existing autonomous driving object detection algorithms, most object detection is achieved by searching for object regions, classifying image features within the regions, and performing fine-grained regression of region locations; pixel-level object detection algorithms are still lacking.

[0081] In another preferred embodiment, for a trained feature filter fitted by a CNN network, the parameters within the filter affect the feature extraction effect of the RGB image. In order to enable the feature filter to stably extract the object of interest, we redesigned the loss function to ensure the training stability of the feature filter.

[0082] In a sample image, the proportion of the target object of interest is often relatively small, accounting for only 10% to 20% of the total pixels. Sometimes, the target object may not even exist in the entire sample. Therefore, the training sample loss fluctuates greatly, the ratio of positive to negative samples is unbalanced, and it is prone to getting stuck in local optima, leading to training stagnation and an inability to fit the optimal filter. We designed our own loss function. Designed for mask-weighted smoothness L1 Regression loss, where smooth L1 (x) as shown in the formula: Building upon the loss function, a weighted mask is introduced to address the imbalance between positive and negative samples. First, a zero-based matrix of the same size as the original RGB image is generated, with different labels for different objects serving as indices. Let's assume element a in the matrix... i,If the corresponding row i and column j are within the target object of interest in the image, then let a i, =α, and if it is in other places, let it be a. i, = 1 - α, where α is an adjustable weighting factor, typically set to 0.8. Using this matrix as a mask, for each image y in the output... i, Perform smooth L1 Loss calculation i, =smooth L1 (y i, -x i, ), and perform weighted processing. Ensure total loss This keeps the training focused on reducing losses at the object of interest, ensuring training stability.

[0083] In summary, this application provides a method and apparatus for autonomous driving 3D target detection based on object imaging. By combining object imaging technology with a deep learning network, it can acquire specific object images from RGB images and filter background data to obtain RGB images containing only the object of interest. Further filtering of the resulting depth map based on the RGB image containing only the object of interest further reduces the amount of computational data and improves image processing efficiency. Furthermore, it provides an implementation method for training a feature filter fitted by a CNN and for designing a loss function to optimize the feature filter, resulting in a more optimized feature filter and improving the accuracy of feature extraction and object of interest recognition.

[0084] Based on the same inventive concept described above, this application also provides an electronic device, which can be a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). This device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the aforementioned autonomous driving 3D target detection method based on object imaging; the memory is used to store computer programs executable by the processor.

[0085] Based on the same inventive concept, this application also provides a computer-readable storage medium corresponding to the aforementioned embodiment of an autonomous driving 3D target detection method based on object imaging method. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the news recommendation method described in any of the above embodiments.

[0086] This application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0087] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A method for 3D target detection in autonomous driving based on object imaging, characterized in that, include: Acquire RGB images from the driving scene; Depth estimation is performed on the RGB image to obtain the corresponding depth map; The depth map and the RGB image are respectively subjected to feature extraction and downsampling compression to obtain the depth map branch feature map and the RGB image branch feature map; The depth map branch feature map and the RGB image branch feature map are fused across modalities to obtain an image containing only the object of interest; The depth map is masked using an image containing only the object of interest as a mask, and the corresponding point cloud data is generated by combining the coordinate information of the image containing only the object of interest. The masking of the depth map using an image containing only the object of interest as a mask is achieved through the following formula: in These are matrix elements representing the three RGB channels; For elements in the depth map; The autonomous driving decision is made based on the image containing only the object of interest and the point cloud data.

2. The autonomous driving 3D target detection method based on object imaging as described in claim 1, characterized in that, Feature extraction of RGB images is performed using the following formula: in, This represents the feature map after feature extraction through the convolutional layer. Represents a non-linear activation function. Represents the convolution kernel , This represents the convolution operation between the convolution kernel and the image. Represents input image , This represents a linear bias.

3. The autonomous driving 3D target detection method based on object imaging as described in claim 2, characterized in that: The convolution kernel It is based on a preset dataset and loss function The dataset obtained through training and optimization, wherein the preset dataset is... Contour data for various objects of interest.

4. The autonomous driving 3D target detection method based on object imaging as described in claim 3, characterized in that, The loss function Designed for mask-weighted calculation ,in: in To output the image; then, the following formula is used to... Perform weighted processing: in It is an adjustable weighting factor.

5. The autonomous driving 3D target detection method based on object imaging according to any one of claims 1-4, characterized in that, This is performed using an autonomous driving 3D target detection device based on an object imaging method, wherein the target detection device includes: Image acquisition module: used to acquire RGB images in driving scenarios; Depth extraction module: Performs depth estimation on the RGB image to obtain the corresponding depth map; The target object imaging module includes: Feature map extraction submodule: Performs feature extraction and downsampling compression on the depth map and the RGB image respectively to obtain depth map branch feature map and RGB image branch feature map; Modal fusion submodule: used to perform cross-modal fusion of the depth map branch feature map and the RGB image branch feature map to obtain an image containing only the object of interest; Depth map masking module: The depth map is masked using an image containing only the object of interest as a mask, and the corresponding point cloud data is generated by combining the coordinate information of the image containing only the object of interest. Driving decision module: Makes autonomous driving decisions based on the image containing only the object of interest and the point cloud data.

6. The autonomous driving 3D target detection method based on object imaging as described in claim 5, characterized in that, The feature map extraction submodule extracts features from the RGB image using the following formula: in, This represents the feature map after feature extraction through the convolutional layer. Represents a non-linear activation function. Represents the convolution kernel , This represents the convolution operation between the convolution kernel and the image. Represents input image , This represents a linear bias.

7. The autonomous driving 3D target detection method based on object imaging as described in claim 6, characterized in that, The target detection device has multiple image acquisition modules.

Images