3D target detection method and device, electronic equipment and computer program product

Abstract

The invention discloses a 3D target detection method and device, electronic equipment and a computer program product, and the method comprises the steps: obtaining laser point cloud data, and carrying out the multi-scale feature extraction through a point cloud feature extraction network, and obtaining a multi-scale point cloud feature; acquiring camera image data and extracting image features by using the image feature extraction network; fusing the multi-scale point cloud features and the image features by using a preset feature fusion strategy to obtain fused features; and detecting the fusion feature by using the target detection head network to obtain a 3D target detection result, including the position and size of the target frame and the target frame angle optimized based on the target frame angle optimization strategy. According to the method, a traditional 3D target detection model is improved, through multi-scale feature extraction of the point cloud data and a specific fusion strategy, the effectiveness of the point cloud features of the bottom layer is enhanced, the detection effect of the small target is improved, the target angle is optimized through the angle optimization strategy, and the calculation accuracy of the yaw angle of the large target is improved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a 3D target detection method, device, electronic equipment, and computer program product. Background Art

[0002] In the field of autonomous driving technology, perception technology plays a pivotal role. Perception technology relies on a variety of high-precision sensors such as lidar and cameras, which are like the "eyes" of the autonomous driving system, continuously capturing and analyzing complex information data of the surrounding environment. Through intelligent processing and analysis of these raw data, advanced algorithms can accurately identify various obstacles on the road, providing crucial prior knowledge and decision-making basis for subsequent core functions such as vehicle obstacle avoidance operations and driving route planning.

[0003] In the 3D target detection scenario, detection technology is subdivided into several targeted research directions according to the different sensors used. Among them, point cloud-based detection technology directly uses dense point cloud data generated by lidar to identify and locate three-dimensional targets in space through complex algorithm models; image-based detection technology focuses on using image information captured by high-definition cameras, combined with advanced technologies such as deep learning, to achieve accurate recognition and tracking of target objects; and multi-modal detection technology is a fusion of the previous detection methods, aiming to further improve the accuracy and robustness of target detection by effectively integrating complementary information from different modal sensors.

[0004] In the field of multimodal target detection, how to efficiently and accurately fuse sensor data of multiple modalities has become the focus and difficulty of current research. In particular, for autonomous vehicles, accurately obtaining and parsing the target's orientation (yaw angle) information is of great significance for subsequent trajectory prediction, path planning, and overall driving safety.

[0005] However, existing multimodal fusion technologies, such as pixel-level alignment, often place extremely stringent requirements on the calibration accuracy of the vehicle and the installation position of the sensor, which poses considerable challenges in practical applications. In addition, these technologies also have certain deficiencies in the accuracy of the regression angle, making it difficult to meet the needs of autonomous driving systems for high-precision target detection and positioning. Summary of the invention

[0006] The embodiments of the present application provide a 3D target detection method, device, electronic device, and computer program product to improve the accuracy of multimodal target fusion, thereby improving the accuracy of 3D target detection.

[0007] The present application embodiment adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a 3D object detection method, where the 3D object detection method includes:

[0009] Obtain lidar point cloud data, and use a point cloud feature extraction network to perform multi-scale feature extraction on the lidar point cloud data to obtain multi-scale point cloud features;

[0010] Obtain camera image data, and use an image feature extraction network to perform feature extraction on the camera image data to obtain image features;

[0011] Use a preset feature fusion strategy to fuse the multi-scale point cloud features and the image features to obtain fused features;

[0012] Use an object detection head network to perform object detection on the fused features to obtain a 3D object detection result, where the 3D object detection result includes the position and size of the 3D object bounding box and the object bounding box angle optimized based on the object bounding box angle optimization strategy.

[0013] Optionally, the using a point cloud feature extraction network to perform multi-scale feature extraction on the lidar point cloud data to obtain multi-scale point cloud features includes:

[0014] Use a point cloud data processing network to perform spatial partitioning on the lidar point cloud data to obtain multiple point cloud spaces;

[0015] Perform feature representation on each point cloud space according to the points included in each point cloud space to obtain multiple point cloud space features;

[0016] Use a point cloud feature extraction network to perform multi-scale feature convolution processing on multiple point cloud space features to obtain the multi-scale point cloud features.

[0017] Optionally, the point cloud feature extraction network includes multiple sequentially connected feature extraction modules, and the using a point cloud feature extraction network to perform multi-scale feature convolution processing on multiple point cloud space features to obtain the multi-scale point cloud features includes:

[0018] Input multiple point cloud space features into multiple sequentially connected feature extraction modules, and output the multi-scale point cloud features, where the multi-scale point cloud features include the point cloud features output by the feature extraction modules in the intermediate layer and the point cloud features output by the feature extraction module in the last layer;

[0019] Among them, the spatial dimension of the point cloud features output by the feature extraction modules in each intermediate layer is smaller than the spatial dimension of the point cloud features output by the corresponding upper-layer feature extraction module.

[0020] Optionally, the using an image feature extraction network to perform feature extraction on the camera image data to obtain image features includes:

[0021] Extract features from the camera image data using a basic feature extraction network to obtain initial image features;

[0022] Process the initial image features using a BEV feature extraction algorithm to obtain BEV image features.

[0023] Optionally, the point cloud feature extraction network includes multiple sequentially connected feature extraction modules. The multi-scale point cloud features include the point cloud features output by the feature extraction module in the middle layer and the point cloud features output by the feature extraction module in the last layer. The using a preset feature fusion strategy to fuse the multi-scale point cloud features and the image features to obtain fusion features includes:

[0024] Fuse the point cloud features output by the feature extraction module in the last layer with the image features to obtain an initial fusion feature;

[0025] Fuse the point cloud features output by the feature extraction module in the middle layer with the initial fusion feature to obtain a final fusion feature.

[0026] Optionally, the fusing the point cloud features output by the feature extraction module in the last layer with the image features to obtain an initial fusion feature includes:

[0027] Concatenate the point cloud features output by the feature extraction module in the last layer with the image features at the channel level to obtain concatenated features;

[0028] Use a convolutional layer to extract features from the concatenated features to obtain the initial fusion feature.

[0029] Optionally, the using a target detection head network to perform target detection on the fusion features to obtain 3D target detection results includes:

[0030] Input the fusion features into the angle classification branch of the target detection head network to output the angle category of the target box. The angle classification branch is used to classify the angle of the target box based on pre-divided angle categories;

[0031] Input the fusion features into the residual angle regression branch of the target detection head network to obtain a residual angle. The residual angle regression branch is used to regress the difference between the optimized target box angle and the angle corresponding to the angle category;

[0032] Determine the optimized target box angle according to the angle category of the target box and the residual angle.

[0033] In a second aspect, an embodiment of the present application further provides a 3D target detection device, where the 3D target detection device includes:

[0034] A point cloud feature extraction unit, configured to obtain lidar point cloud data, and perform multi-scale feature extraction on the lidar point cloud data by using a point cloud feature extraction network to obtain multi-scale point cloud features;

[0035] An image feature extraction unit, configured to obtain camera image data, and perform feature extraction on the camera image data by using an image feature extraction network to obtain image features;

[0036] A fusion unit, configured to fuse the multi-scale point cloud features and the image features by using a preset feature fusion strategy to obtain fused features;

[0037] A detection unit, configured to perform object detection on the fused features by using a target detection head network to obtain 3D object detection results, where the 3D object detection results include the position and size of a 3D object box and the angle of the object box optimized based on an object box angle optimization strategy.

[0038] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0039] A processor; and a memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute any one of the foregoing 3D object detection methods.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, where the computer program or instructions, when executed by a processor, implement any one of the foregoing 3D object detection methods.

[0041] The above at least one technical solution adopted by the embodiment of the present application can achieve the following beneficial effects: The 3D object detection method of the embodiment of the present application first obtains lidar point cloud data, and performs multi-scale feature extraction on the lidar point cloud data by using a point cloud feature extraction network to obtain multi-scale point cloud features; then obtains camera image data, and performs feature extraction on the camera image data by using an image feature extraction network to obtain image features; then fuses the multi-scale point cloud features and the image features by using a preset feature fusion strategy to obtain fused features; finally, performs object detection on the fused features by using a target detection head network to obtain 3D object detection results, where the 3D object detection results include the position and size of a 3D object box and the angle of the object box optimized based on an object box angle optimization strategy. The 3D object detection method of the embodiment of the present application improves the traditional 3D object detection model. By performing multi-scale feature extraction on point cloud data and then adopting a certain fusion strategy to fuse with image features, the utility of underlying point cloud features is enhanced, the detection effect of small objects is improved, and the accuracy of large object yaw angle calculation is improved by optimizing the object angle through an angle optimization strategy. Brief Description of the Drawings

[0042] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0043] Figure 1 is a schematic flowchart of a 3D object detection method in an embodiment of the present application;

[0044] Figure 2 is a schematic structural diagram of a 3D object detection device in an embodiment of the present application;

[0045] Figure 3 is a schematic structural diagram of an electronic device in an embodiment of the present application. Detailed Description of the Embodiments

[0046] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0047] The following will describe in detail the technical solutions provided by each embodiment of the present application in conjunction with the drawings.

[0048] The embodiment of the present application provides a 3D object detection method. As Figure 1 shown, a schematic flowchart of a 3D object detection method in an embodiment of the present application is provided. The 3D object detection method at least includes the following steps S110 to step S140:

[0049] Step S110, obtain lidar point cloud data, and use a point cloud feature extraction network to perform multi-scale feature extraction on the lidar point cloud data to obtain multi-scale point cloud features.

[0050] The 3D object detection method of the embodiment of the present application is a multi-modal 3D object detection method. It is defined that a lidar and a camera have been installed and calibrated on the vehicle, and a machine similar to Orin is equipped on the vehicle to run a deep learning model, which can detect various types of objects in real time at the vehicle end, including object categories such as pedestrians, cyclists, cars, trucks, etc. Of course, the 3D object detection method of the embodiment of the present application can also be applied to 3D object detection at the roadside, and those skilled in the art can flexibly expand it according to actual application requirements.

[0051] The neural network structure designed in the embodiments of this application can be obtained by improving on the network structure of the BEVFusion model. The 3D object detection model in the embodiments of this application mainly consists of a point cloud feature extraction network, an image feature extraction network, a fusion network, and an object detection head network.

[0052] When performing 3D object detection, it is necessary to obtain the point cloud data collected by the lidar, and then use the trained point cloud feature extraction network to perform multi-scale feature extraction on the lidar point cloud data to obtain multi-scale point cloud features. Here, the multi-scale feature extraction can be understood as extracting point cloud features at different levels based on the feature extraction modules at multiple levels in the point cloud feature extraction network. By extracting multi-scale features, it is possible to appropriately retain the point cloud feature information at the bottom layer of the feature extraction network, which helps to improve the detection effect of small targets in the subsequent process.

[0053] Step S120: Obtain camera image data, and use the image feature extraction network to perform feature extraction on the camera image data to obtain image features.

[0054] When performing 3D object detection, it is also necessary to obtain the image data collected by the camera, and then use the trained image feature extraction network to perform feature extraction on the camera image data to obtain image features. The specific implementation of the image feature extraction network can be flexibly determined in combination with the existing technology and will not be elaborated here.

[0055] Step S130: Use a preset feature fusion strategy to fuse the multi-scale point cloud features and the image features to obtain fused features.

[0056] After obtaining the multi-scale point cloud features and image features based on the foregoing steps, it is necessary to adopt a certain fusion strategy to perform feature fusion on the multi-scale point cloud features and image features to obtain fused features. The purpose of setting the fusion strategy is to make full use of and give play to the utility of the point cloud feature information at the bottom layer, thereby improving the detection effect of small targets.

[0057] Step S140: Use the object detection head network to perform object detection on the fused features to obtain 3D object detection results. The 3D object detection results include the position and size of the 3D object box and the angle of the object box optimized based on the object box angle optimization strategy.

[0058] Input the above-obtained fused features into the object detection head network. The object detection head network performs further feature extraction such as Transformer feature extraction, and finally outputs the center point position (x, y, z) of the 3D object box, the length, width, and height (l, w, h) of the 3D object box, and the angle of the object box optimized based on the object box angle optimization strategy.

[0059] The 3D object detection method of the embodiments of the present application improves the traditional 3D object detection model. By performing multi-scale feature extraction on point cloud data and then adopting a certain fusion strategy to fuse with image features, the utility of the underlying point cloud features is enhanced, the detection effect of small objects is improved, and the target angle is optimized through an angle optimization strategy, improving the accuracy of the yaw angle calculation of large objects.

[0060] In some embodiments of the present application, the multi-scale feature extraction of the laser point cloud data by using the point cloud feature extraction network to obtain multi-scale point cloud features includes: using the point cloud data processing network to perform spatial partitioning on the laser point cloud data to obtain multiple point cloud spaces; performing feature representation on each point cloud space according to the points included in each point cloud space to obtain multiple point cloud space features; using the point cloud feature extraction network to perform multi-scale feature convolution processing on the multiple point cloud space features to obtain the multi-scale point cloud features.

[0061] Before performing point cloud feature extraction, the embodiments of the present application may first use the point cloud feature processing network to preprocess the point cloud data. Specifically, the point cloud feature processing network can divide the point cloud space corresponding to each frame into voxels with a fixed size, which is convenient for subsequent processing and analysis. The length, width, and height of each voxel and the number of points included in each voxel can be set according to actual needs or flexibly adjusted according to the detection effect. Each point in each voxel can be represented by 4-dimensional features (x, y, z, intensity), where x, y, z are the spatial position coordinates of the point, and intensity is the reflection intensity of the point. Taking the mean of all points within each voxel can obtain the basic feature representation of the voxel. Of course, in addition to the mean method, those skilled in the art can also flexibly adopt other feature representation methods according to actual needs, such as variance, maximum and minimum values, etc., which are not specifically limited herein.

[0062] After obtaining the point cloud feature representation of each voxel space, the point cloud feature representation of each voxel space is input into the point cloud feature extraction network. The point cloud feature extraction network performs multi-scale feature convolution processing on the point cloud features of multiple voxel spaces based on sparse convolution to obtain multi-scale point cloud features.

[0063] In some embodiments of the present application, the point cloud feature extraction network includes a plurality of sequentially connected feature extraction modules. Using the point cloud feature extraction network to perform multi-scale feature convolution processing on a plurality of point cloud spatial features to obtain the multi-scale point cloud features includes: inputting a plurality of point cloud spatial features into a plurality of sequentially connected feature extraction modules and outputting the multi-scale point cloud features. The multi-scale point cloud features include the point cloud features output by the feature extraction modules in the intermediate layers and the point cloud features output by the feature extraction module in the last layer. Among them, the spatial dimension of the point cloud features output by the feature extraction modules in each intermediate layer is smaller than the spatial dimension of the point cloud features output by the corresponding upper-layer feature extraction module.

[0064] In the embodiments of the present application, a plurality of sequentially connected feature extraction modules (block) are designed in the point cloud feature extraction network. The above-mentioned plurality of point cloud spatial features obtained in the embodiments are input into the plurality of sequentially connected feature extraction modules (block) for convolution processing. Here, it can be set that the spatial dimension of the point cloud features output by each intermediate layer block is smaller than the spatial dimension of the point cloud features output by the corresponding upper-layer block, so as to improve the receptive field of the features.

[0065] After convolution processing by a plurality of blocks, finally, not only the point cloud features of the last block need to be output, but also the point cloud features of one or several intermediate blocks need to be output. The purpose of outputting the point cloud features of the intermediate layer is to strengthen the utilization of the underlying features, thereby improving the detection effect of small targets. Especially in multi-task scenarios, such as the multi-task scenarios of 3D object detection and 3D object segmentation, it is particularly important for improving the accuracy of object segmentation.

[0066] For example, in the point cloud feature extraction network in the embodiments of the present application, 4 sequentially connected feature extraction modules (block) can be designed. The sizes of the point cloud feature maps obtained by the first 3 blocks are 1 / 2 of the size of the point cloud feature map of the previous block, so as to continuously improve the receptive field of the features. Finally, the point cloud feature maps obtained by the 2nd block and the 4th block are output for subsequent fusion processing. Of course, the specific number of blocks set and which intermediate layer's point cloud features to output can be flexibly adjusted by those skilled in the art according to the detection effect of the actual scenario, and no specific limitation is made here.

[0067] In some embodiments of the present application, the number of convolutions in the last block can be reduced on the basis of the network structure of the BEVFusion model, so as to reduce the time consumption at the vehicle end and further meet the requirements of real-time detection at the vehicle end.

[0068] In some embodiments of the present application, the extracting of image features from the camera image data by using the image feature extraction network includes: extracting initial image features from the camera image data by using a basic feature extraction network; and processing the initial image features by using a BEV feature extraction algorithm to obtain BEV image features.

[0069] In the embodiments of the present application, when extracting image features, a basic feature extraction network such as a ResNet network can be first used to extract image features. ResNet is a deep convolutional neural network widely used in image recognition and feature extraction tasks. It can extract preliminary and low-level image features such as edges and textures from camera image data through multiple layers of convolution and pooling operations.

[0070] After that, a BEV (Bird's Eye View) feature extraction algorithm such as the LSS (Lift-Splat-Shoot) algorithm is used to process the above initial image features. First, the depth probability of each feature point is estimated, and the features are spread to the pixel depths corresponding to the features according to the depth probability of each feature point. Finally, BEV image features are obtained through BEV Pool. The processing of the LSS algorithm preserves the symmetry of the 3D space and is applicable to tasks such as 3D object detection and segmentation in autonomous driving.

[0071] In some embodiments of the present application, the point cloud feature extraction network includes a plurality of sequentially connected feature extraction modules. The multi-scale point cloud features include the point cloud features output by the feature extraction modules in the middle layer and the point cloud features output by the feature extraction module in the last layer. The fusing of the multi-scale point cloud features and the image features by using a preset feature fusion strategy to obtain fused features includes: fusing the point cloud features output by the feature extraction module in the last layer with the image features to obtain initial fused features; and fusing the point cloud features output by the feature extraction modules in the middle layer with the initial fused features to obtain final fused features.

[0072] When performing feature fusion, the point cloud features output by the last block of the point cloud feature extraction network can be first fused with the image features to obtain initial fused features, and then the point cloud features output by the middle block of the point cloud feature extraction network can be further fused with the initial fused features, so as to make more full use of the underlying feature information and output the final fused features.

[0073] For example, for a point cloud feature extraction network composed of 4 sequentially connected feature extraction modules (blocks), the point cloud feature map and the image feature map output by the 4th block can be first fused to obtain preliminary fused features, and then the point cloud features output by the 2nd block can be fused with the preliminary fused features to obtain the final fused features.

[0074] In some embodiments of the present application, the fusion of the point cloud features output by the feature extraction module of the last layer and the image features to obtain the initial fusion features includes: splicing the point cloud features output by the feature extraction module of the last layer and the image features at the channel level to obtain the spliced features; using a convolutional layer to perform feature extraction on the spliced features to obtain the initial fusion features.

[0075] When fusing the point cloud features output by the feature extraction module of the last layer and the image features, the point cloud features output by the feature extraction module of the last layer and the image features can be first spliced at the channel level, and further deep feature extraction can be performed using a convolutional layer to obtain the preliminary fusion features. Finally, the point cloud features output by the intermediate layer and the preliminary fusion features are spliced and further feature extracted to obtain the final fusion features.

[0076] In some embodiments of the present application, the use of the target detection head network to perform target detection on the fusion features to obtain the 3D target detection results includes: inputting the fusion features into the angle classification branch of the target detection head network to output the angle category of the target box, and the angle classification branch is used to classify the angle of the target box based on the pre-divided angle categories; inputting the fusion features into the residual angle regression branch of the target detection head network to obtain the residual angle, and the residual angle regression branch is used to regress the difference between the optimized target box angle and the angle corresponding to the angle category; determining the optimized target box angle according to the angle category of the target box and the residual angle.

[0077] In the embodiments of the present application, when using the target detection head network to perform target detection on the fusion features, the original regression of the cosine value and sine value of the angle is adjusted to the method of first classifying the target angle and then regressing the residual angle. This method can improve the accuracy of the yaw angle regression of large targets such as large vehicles.

[0078] Specifically, the target detection head network of the embodiments of the present application is composed of two branches. One branch is the angle classification branch, and the other branch is the residual angle regression branch. The angle classification branch is used to predict and output the angle category to which the target box angle belongs. Each angle category corresponds to an angle range, which is equivalent to first predicting which angle range the target box angle belongs to. In the model training stage, the angle categories can be defined and divided in advance. For example, the 360-degree angle can be divided into 24 categories, each category corresponding to a 15-degree sector, and each angle sector is respectively marked, such as marked as category 1, 2, 3,... corresponding to 0-15 degrees, 15-30 degrees, 30-45 degrees... Of course, how to specifically divide the angle categories can be flexibly set by those skilled in the art according to actual needs, and no specific limitation is made here.

[0079] The residual angle regression branch is used to regress the difference between the final target box angle and the angle corresponding to the angle category, which is equivalent to further regressing the specific residual angle on the basis of the angle category.

[0080] Finally, the final target box angle can be calculated by combining the angle category and the regressed residual angle.

[0081] For example, assume that the angle category label output by the angle classification branch is 3, and the residual angle output by the residual angle regression branch is 2 degrees. Then the angle corresponding to the angle category label 3 is 3 * 15 degrees = 45 degrees. Adding the residual angle of 2 degrees, we get the final target box angle = 45 degrees + 2 degrees = 47 degrees.

[0082] In summary, the 3D object detection method proposed in this application can better adapt to the detection conditions at the vehicle end by fusing the features of lidar point cloud and camera image at the BEV level, and reduces the layout requirements between sensors. The complementary advantages of the two sensor data can improve the accuracy and recall rate of 3D object detection to a certain extent; at the same time, the yaw angle problem of targets such as large vehicles is optimized, which significantly improves subsequent prediction and path planning.

[0083] The embodiment of this application also provides a 3D object detection device 200, as Figure 2 shown in the structural schematic diagram of a 3D object detection device in the embodiment of this application. The 3D object detection device 200 includes: a point cloud feature extraction unit 210, an image feature extraction unit 220, a fusion unit 230, and a detection unit 240, where:

[0084] The point cloud feature extraction unit 210 is used to obtain lidar point cloud data and perform multi-scale feature extraction on the lidar point cloud data by using a point cloud feature extraction network to obtain multi-scale point cloud features;

[0085] The image feature extraction unit 220 is used to obtain camera image data and perform feature extraction on the camera image data by using an image feature extraction network to obtain image features;

[0086] The fusion unit 230 is used to fuse the multi-scale point cloud features and the image features by using a preset feature fusion strategy to obtain fusion features;

[0087] The detection unit 240 is used to perform object detection on the fusion features by using an object detection head network to obtain a 3D object detection result, where the 3D object detection result includes the position and size of the 3D target box and the target box angle optimized based on the target box angle optimization strategy.

[0088] In some embodiments of the present application, the point cloud feature extraction unit 210 is specifically configured to: perform spatial partitioning on the lidar point cloud data by using a point cloud data processing network to obtain a plurality of point cloud spaces; perform feature representation on each point cloud space according to the points included in each point cloud space to obtain a plurality of point cloud space features; and perform multi-scale feature convolution processing on the plurality of point cloud space features by using a point cloud feature extraction network to obtain the multi-scale point cloud features.

[0089] In some embodiments of the present application, the point cloud feature extraction network includes a plurality of sequentially connected feature extraction modules. The point cloud feature extraction unit 210 is specifically configured to: input the plurality of point cloud space features into the plurality of sequentially connected feature extraction modules and output the multi-scale point cloud features, where the multi-scale point cloud features include the point cloud features output by the feature extraction modules in the intermediate layer and the point cloud features output by the feature extraction module in the last layer; and among them, the spatial dimension of the point cloud features output by the feature extraction modules in each intermediate layer is smaller than the spatial dimension of the point cloud features output by the corresponding upper-layer feature extraction module.

[0090] In some embodiments of the present application, the image feature extraction unit is specifically configured to: perform feature extraction on the camera image data by using a basic feature extraction network to obtain initial image features; and perform processing on the initial image features by using a BEV feature extraction algorithm to obtain BEV image features.

[0091] In some embodiments of the present application, the point cloud feature extraction network includes a plurality of sequentially connected feature extraction modules. The multi-scale point cloud features include the point cloud features output by the feature extraction modules in the intermediate layer and the point cloud features output by the feature extraction module in the last layer. The fusion unit 230 is specifically configured to: fuse the point cloud features output by the feature extraction module in the last layer with the image features to obtain initial fusion features; and fuse the point cloud features output by the feature extraction modules in the intermediate layer with the initial fusion features to obtain final fusion features.

[0092] In some embodiments of the present application, the fusion unit 230 is specifically configured to: splice the point cloud features output by the feature extraction module in the last layer and the image features at the channel level to obtain spliced features; and perform feature extraction on the spliced features by using a convolutional layer to obtain the initial fusion features.

[0093] In some embodiments of the present application, the detection unit 240 is specifically configured to: input the fusion feature into the angle classification branch of the target detection head network to output the angle category of the target box, where the angle classification branch is used to classify the angle of the target box based on pre-divided angle categories; input the fusion feature into the residual angle regression branch of the target detection head network to obtain a residual angle, where the residual angle regression branch is used to regress the difference between the optimized angle of the target box and the angle corresponding to the angle category; and determine the optimized angle of the target box according to the angle category of the target box and the residual angle.

[0094] It can be understood that the above 3D object detection device can implement each step of the 3D object detection method provided in the foregoing embodiments. The relevant explanations of the 3D object detection method are applicable to the 3D object detection device and will not be elaborated here.

[0095] Figure 3 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 3 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0096] The processor, network interface, and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 3 only a bidirectional arrow is used in [figure], but it does not mean that there is only one bus or one type of bus.

[0097] The memory is used to store a program. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0098] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a 3D object detection device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0099] Obtain the laser point cloud data, and use the point cloud feature extraction network to perform multi-scale feature extraction on the laser point cloud data to obtain multi-scale point cloud features;

[0100] Obtain the camera image data, and use the image feature extraction network to perform feature extraction on the camera image data to obtain image features;

[0101] Use a preset feature fusion strategy to fuse the multi-scale point cloud features and the image features to obtain fused features;

[0102] Use the object detection head network to perform object detection on the fused features to obtain 3D object detection results, where the 3D object detection results include the position and size of the 3D object box and the object box angle optimized based on the object box angle optimization strategy.

[0103] The above as in this application Figure 1The method executed by the 3D object detection device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor or instructions in software form. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0104] The embodiments of the present application also propose a computer program product. The computer program product stores one or more programs. The one or more programs include instructions that, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 the method executed by the 3D object detection device in the illustrated embodiment, and specifically used to execute:

[0105] Obtain laser point cloud data, and use a point cloud feature extraction network to perform multi-scale feature extraction on the laser point cloud data to obtain multi-scale point cloud features;

[0106] Obtain camera image data, and use an image feature extraction network to perform feature extraction on the camera image data to obtain image features;

[0107] Use a preset feature fusion strategy to fuse the multi-scale point cloud features and the image features to obtain fusion features;

[0108] The target detection head network is used to perform target detection on the fused features to obtain 3D target detection results, where the 3D target detection results include the position and size of the 3D target box and the target box angle optimized based on the target box angle optimization strategy.

[0109] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0111] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0113] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0114] The memory may include non-permanent memory in the form of computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0115] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0116] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as a method, system, or computer program product. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A 3D object detection method, wherein: The 3D target detection method comprises: Acquire laser point cloud data, and use a point cloud feature extraction network to perform multi-scale feature extraction on the laser point cloud data to obtain multi-scale point cloud features; Acquire camera image data, and use an image feature extraction network to perform feature extraction on the camera image data to obtain image features; Using a preset feature fusion strategy to fuse the multi-scale point cloud features and the image features to obtain a fused feature; The target detection head network is used to perform target detection on the fused features to obtain a 3D target detection result, wherein the 3D target detection result includes the position and size of the 3D target box and the target box angle optimized based on the target box angle optimization strategy.

2. The 3D object detection method according to claim 1, wherein: The point cloud feature extraction network is used to extract multi-scale features from the laser point cloud data to obtain multi-scale point cloud features, including: Using a point cloud data processing network to spatially divide the laser point cloud data to obtain a plurality of point cloud spaces; Perform feature representation on each point cloud space according to the points contained in each point cloud space to obtain multiple point cloud space features; A point cloud feature extraction network is used to perform multi-scale feature convolution processing on multiple point cloud spatial features to obtain the multi-scale point cloud features.

3. The 3D object detection method according to claim 2, wherein: The point cloud feature extraction network comprises a plurality of feature extraction modules connected in sequence, and the point cloud feature extraction network is used to perform multi-scale feature convolution processing on a plurality of point cloud spatial features to obtain the multi-scale point cloud features, including: Inputting a plurality of point cloud spatial features into a plurality of sequentially connected feature extraction modules, and outputting the multi-scale point cloud features, wherein the multi-scale point cloud features include the point cloud features output by the feature extraction modules of the middle layer and the point cloud features output by the feature extraction modules of the last layer; Among them, the spatial dimension of the point cloud features output by the feature extraction module of each intermediate layer is smaller than the spatial dimension of the point cloud features output by the feature extraction module of the corresponding previous layer.

4. The 3D object detection method according to claim 1, wherein: The image feature extraction network is used to extract features from the camera image data to obtain image features, including: Using a basic feature extraction network to extract features from the camera image data to obtain initial image features; The initial image features are processed using a BEV feature extraction algorithm to obtain BEV image features.

5. The 3D object detection method according to claim 1, wherein: The point cloud feature extraction network includes a plurality of feature extraction modules connected in sequence, the multi-scale point cloud features include point cloud features output by the feature extraction modules of the middle layer and point cloud features output by the feature extraction modules of the last layer, and the multi-scale point cloud features and the image features are fused by using a preset feature fusion strategy to obtain fused features including: The point cloud features output by the feature extraction module of the last layer are fused with the image features to obtain initial fused features; The point cloud features output by the feature extraction module of the intermediate layer are fused with the initial fusion features to obtain the final fusion features.

6. The 3D object detection method according to claim 5, wherein: The point cloud features output by the feature extraction module of the last layer are fused with the image features to obtain the initial fused features, including: Splicing the point cloud features output by the feature extraction module of the last layer with the image features at the channel level to obtain spliced ​​features; The convolutional layer is used to extract the concatenated features to obtain the initial fusion features.

7. The 3D object detection method according to claim 1, wherein: The target detection is performed on the fused features by using the target detection head network to obtain a 3D target detection result, which includes: Inputting the fusion feature into the angle classification branch of the target detection head network, and outputting the angle category of the target frame, wherein the angle classification branch is used to perform angle classification on the target frame based on the pre-divided angle categories; Inputting the fused feature into the residual angle regression branch of the target detection head network to obtain a residual angle, wherein the residual angle regression branch is used to regress the difference between the optimized target frame angle and the angle corresponding to the angle category; The optimized target frame angle is determined according to the angle category of the target frame and the residual angle.

8. A 3D object detection device, wherein: The 3D target detection device comprises: A point cloud feature extraction unit is used to obtain laser point cloud data, and perform multi-scale feature extraction on the laser point cloud data using a point cloud feature extraction network to obtain multi-scale point cloud features; An image feature extraction unit, used to obtain camera image data, and perform feature extraction on the camera image data using an image feature extraction network to obtain image features; A fusion unit, used to fuse the multi-scale point cloud feature and the image feature using a preset feature fusion strategy to obtain a fusion feature; The detection unit is used to perform target detection on the fused features using a target detection head network to obtain a 3D target detection result, wherein the 3D target detection result includes the position and size of the 3D target box and the target box angle optimized based on the target box angle optimization strategy.

9. An electronic device, comprising: processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the 3D object detection method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implements the 3D object detection method according to any one of claims 1 to 7.

Images