A method and apparatus for 3D object detection

Abstract

The embodiment of the application discloses a kind of 3D target detection method and equipment, it can be applied to the computer vision field in artificial intelligence field, the method comprises: first, the 2D information corresponding to each laser point in laser point cloud is obtained, and then laser point cloud and 2D information are input into two three-dimensional sparse convolution modules respectively, to obtain first feature and second feature, first feature and second feature are cascaded and then input into a two-dimensional convolution module, to obtain third feature (i.e. fusion feature), third feature is cascaded with first feature and second feature to obtain fourth feature (i.e. combination feature), finally, 3D target detection is realized using fourth feature.The embodiment of the application fuses the laser point cloud and 2D information at the feature level, improves the 3D target detection performance while retaining the original features of the laser point cloud, so that the 3D target detection has good robustness in complex scenarios such as night, rain and fog, etc.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a method and apparatus for 3D object detection. Background Technology

[0002] Object detection is a traditional task in computer vision. Compared to image classification, object detection requires identifying the category of an object and outputting its smallest bounding box. For example... Figure 1 The two-dimensional (2D) object detection shown on the left typically takes a 2D image as input and outputs the category of the target object and its 2D bounding box (also called a 2D object bounding box). However, in a real three-dimensional (3D) environment, objects have three-dimensional shapes, and most applications require information such as the object's length, width, height, and angles. For example, in the field of autonomous driving, with the widespread use of sensors such as LiDAR and RGB-D (red, green, blue-depth) cameras, 2D object detection can no longer meet the needs of certain scenarios. Therefore, 3D object detection has emerged, such as... Figure 1 As shown on the right, in 3D object detection, the output is the 3D bounding box of the target object (also called the 3D target box, which includes information such as length, width, height and rotation angle) and the classification category of the target object. It can be applied to a variety of application scenarios, such as the detection task of autonomous vehicles.

[0003] like Figure 2 As shown, taking the application of 3D object detection in the field of autonomous driving as an example, for 3D object detection, sensor data such as LiDAR, millimeter-wave radar, ultrasonic sensors, light detection and ranging (LiDAR), and RGB-D cameras can all be used as input data. Different types of sensors have their own advantages and disadvantages. For example, LiDAR has low resolution and sparse laser point cloud data, but its ranging capability and environmental adaptability are better; RGB image data is denser and performs better in target recognition, but it is greatly affected by the environment and completely fails at night or in rainy or foggy weather; ultrasonic sensors are good at detecting targets at close range, but cannot perform long-range ranging; millimeter-wave radar can accurately detect target speed, but cannot perform target recognition. Therefore, by fusing sensor data from multiple modalities (i.e., sensor data collected by different sensors), the redundancy and accuracy of the detection model can be improved.

[0004] However, most current 3D target detection models use pure LiDAR as input, which far exceeds the performance of multimodal 3D detection. How to more reasonably and effectively fuse multimodal data to improve the performance of 3D target detection is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method and device for 3D target detection, which obtains 2D information corresponding to each laser point in the laser point cloud based on the laser point cloud and the 2D image, and performs feature fusion of the laser point cloud and the 2D information at the feature level. This improves the performance of 3D target detection while retaining the original features of the 3D laser point cloud, enabling the 3D target detection to have good robustness in complex scenarios such as nighttime, rainy or foggy days when the camera fails.

[0006] Based on this, the embodiments of this application provide the following technical solutions:

[0007] Firstly, this application provides a method for 3D object detection, applicable to the field of computer vision within the field of artificial intelligence. The method includes: an execution device first acquires an image and a laser point cloud at a certain location using a sensor device (e.g., a camera, LiDAR, etc.) deployed on the execution device. Then, based on the acquired image and laser point cloud, it obtains two-dimensional information corresponding to the laser point cloud. After acquiring the laser point cloud and the corresponding two-dimensional information, the execution device performs convolution operations on the laser point cloud and the two-dimensional information using a first three-dimensional sparse convolution module and a second three-dimensional sparse convolution module, respectively, to obtain a first feature corresponding to the laser point cloud and a second feature corresponding to the two-dimensional information. After obtaining the first feature corresponding to the laser point cloud and the second feature corresponding to the two-dimensional information, the execution device concatenates the first and second features and inputs the concatenated first and second features into a two-dimensional convolution module for convolution operations to obtain a third feature. This third feature, because it integrates the first and second features, can also be called a fused feature. It should be noted that in this embodiment, feature cascading refers to superimposing features to obtain a new cascaded feature. For example, assuming the first feature is 1×2×3 and the second feature is also 1×2×3, where 1 represents the number of channels and 2×3 represents the size of the first / second feature, then the cascaded feature obtained after cascading the first and second features is 2×2×3, where 2 is the number of channels after cascading and 2×3 is the size of the cascaded feature. After the execution device obtains the third feature, it cascades the previously obtained first, second, and third features to obtain the fourth feature, which can also be called a combined feature. After obtaining the fourth feature, the execution device can input the fourth feature into the classification regression module, such as the classification regression head, to output the 3D bounding box and the classification category of the target object within the 3D bounding box. The target object can also be called the target object.

[0008] In the above embodiments of this application, the execution device first obtains the 2D information corresponding to each laser point in the laser point cloud (i.e., 3D information). Then, the 3D information and the 2D information are respectively input into two three-dimensional sparse convolutional modules to obtain a first feature and a second feature. The first feature and the second feature are then concatenated and input into a two-dimensional convolutional module to obtain a third feature (i.e., fused feature). The third feature is then concatenated with the first and second features to obtain a fourth feature (i.e., combined feature). Finally, the fourth feature is used to achieve 3D target detection through a classification and regression module. This application is used to obtain the 2D information corresponding to each laser point in the laser point cloud based on the laser point cloud and the 2D image, and to fuse the laser point cloud and the 2D information at the feature level. This improves the performance of 3D target detection while retaining the original features of the 3D laser point cloud, enabling the 3D target detection to have good robustness in complex scenarios such as nighttime, rainy or foggy weather where the camera is malfunctioning.

[0009] In one possible design of the first aspect, the execution device obtains the two-dimensional information corresponding to the laser point cloud based on the acquired image and the laser point cloud through the following process: First, the execution device performs semantic segmentation on the acquired image using a semantic segmentation model to obtain the semantic segmentation score of each pixel in the image. This segmentation score represents the probability that each pixel belongs to its respective classification category, and the semantic segmentation scores of each pixel in the image constitute a semantic segmentation map. Then, the execution device projects the laser point cloud onto the image and the semantic segmentation map obtained based on the image, respectively, to obtain the target RGB information and target semantic segmentation score corresponding to each laser point cloud. This target RGB information and target semantic segmentation score constitute the two-dimensional information.

[0010] In the above embodiments of this application, it is described how to obtain 2D information from an image and a laser point cloud. That is, the image is first semantically segmented to obtain a semantic segmentation map, and then the laser point cloud is projected onto the image and the semantic segmentation map respectively to obtain 2D information, which is feasible.

[0011] In one possible design of the first aspect, the input data of the sparse convolution module can be voxelized first, and then input into the sparse convolution module to perform convolution operation. Therefore, the execution device can voxelize the laser point cloud to obtain the voxelized laser point cloud, and then perform convolution operation on the voxelized laser point cloud through the first sparse convolution module to obtain the first feature.

[0012] In the above embodiments of this application, the input data of the three-dimensional sparse convolution module needs to be voxelized first, and the voxelized input data is then input into the sparse convolution module to perform convolution operation. Therefore, in this embodiment, the laser point cloud is first voxelized, and the voxelized laser point cloud is then processed by the first sparse convolution module to perform convolution operation, which is flexible.

[0013] In one possible design of the first aspect, the execution device will also perform voxelization on the two-dimensional information to obtain voxelized two-dimensional information, and then perform convolution operation on the voxelized two-dimensional information through the second sparse convolution module to obtain the second feature.

[0014] In the above embodiments of this application, not only is voxelization of the laser point cloud required, but also voxelization of the two-dimensional information is required, which provides flexibility. Furthermore, the embodiments of this application perform voxelization of both the laser point cloud and the two-dimensional information, which is easy to operate.

[0015] In one possible design of the first aspect, the execution device may first concatenate the laser point cloud and the two-dimensional information to obtain multimodal information. Then, the execution device performs voxelization on the multimodal information to obtain voxelized multimodal information. Obviously, the voxelized multimodal information includes the voxelized laser point cloud and the voxelized two-dimensional information. Finally, the execution device performs a convolution operation on the voxelized laser point cloud through a first sparse convolution module to obtain a first feature, and performs a convolution operation on the voxelized two-dimensional information through a second sparse convolution module to obtain a second feature.

[0016] In the above embodiments of this application, another voxelization method is described, namely, first concatenating the laser point cloud with the 2D information to obtain multimodal information. In this way, voxelizing the multimodal information once is equivalent to voxelizing the laser point cloud and the 2D information simultaneously, thereby saving one voxelization operation.

[0017] In one possible design of the first aspect, in some embodiments of this application, the input data of the two-dimensional convolution module can be devoxed first. Devoxing is the reverse of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution operation. Specifically, the execution device concatenates the first feature and the second feature to obtain a concatenated feature. Then, the concatenated feature is devoxed to obtain a devoxed concatenated feature. Finally, the execution device inputs the devoxed concatenated feature into the two-dimensional convolution module to perform convolution operation to obtain the third feature, i.e., the fused feature.

[0018] In the above embodiments of this application, the input data of the two-dimensional convolution module needs to be devoxed (i.e., denser representation). Devoxing is the opposite of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution operation, which provides flexibility.

[0019] The second aspect of this application also provides a training method for a 3D object detection model. The 3D object detection model includes a three-dimensional first sparse convolution module, a three-dimensional second sparse convolution module, a two-dimensional convolution module, and a classification and regression module. The method may include: First, a training device acquires an initial training set. Any training sample in this initial training set can be referred to as an initial training sample. Each initial training sample in the initial training set includes a frame of laser point cloud and two-dimensional information (which can be referred to as the initial laser point cloud and initial two-dimensional information). The initial two-dimensional information is obtained based on the initial laser point cloud and the initial image. The initial laser point cloud and the initial image correspond, i.e., they are sensor data collected simultaneously at the same location by different types of sensors. Then, the training device constructs a first training set based on the initial training set. The construction process involves extracting target objects, such as local laser points and local two-dimensional information constituting target objects like "cars," "trucks," and "pedestrians," from each initial training sample in the initial training set. The local laser points and local two-dimensional information of each target object are then randomly copied into each initial training sample to obtain the first training sample. Each first training sample constitutes the first training set described in this application. Similarly, each first training sample in the first training set also includes a frame of laser point cloud and two-dimensional information (which can be referred to as the first laser point cloud and the first two-dimensional information). After the training device constructs the first training set based on the initial training set, the first training samples in the first training set are all data-augmented training samples. These first training samples include the first laser point cloud and the first two-dimensional information. The training device then performs convolution operations on the first laser point cloud and the first two-dimensional information through the first sparse convolution module and the second sparse convolution module, respectively, to obtain the first feature corresponding to the first laser point cloud and the second feature corresponding to the first two-dimensional information. After obtaining the first feature corresponding to the first laser point cloud and the second feature corresponding to the first two-dimensional information, the training device concatenates the first feature and the second feature, and then inputs the concatenated first feature and second feature into the two-dimensional convolution module for convolution operation to obtain the third feature. This third feature, because it combines the first feature and the second feature, can also be called the fused feature. After obtaining the third feature, the training device concatenates the previously obtained first feature, second feature, and third feature to obtain the fourth feature, which can also be called the combined feature. After the training device obtains the fourth feature, it can input this fourth feature into the classification and regression module, such as the classification and regression head, to obtain the 3D predicted bounding box and the predicted classification category of the target object within the 3D predicted bounding box. The target object can also be referred to as the target object.After obtaining the 3D predicted bounding box and the predicted classification category of the target object within the 3D predicted bounding box, the training device will iteratively train the 3D target prediction model using the target loss function based on the 3D ground truth bounding box, the ground truth classification category of the target object within the 3D ground truth bounding box, the 3D predicted bounding box and the predicted classification category of the target object within the 3D predicted bounding box.

[0020] In this embodiment, it is specifically described how to use the various modules in the training device to iteratively train the various modules in the 3D object detection model, and the training process is easy to implement.

[0021] In one possible design of the second aspect, the initial two-dimensional information includes the initial RGB information corresponding to each initial laser point in the initial laser point cloud obtained by projecting the initial laser point cloud onto the initial image, and the initial semantic segmentation score corresponding to each initial laser point obtained by projecting the initial laser point cloud onto the initial semantic segmentation map. The initial semantic segmentation score is the probability that each pixel in the initial image belongs to its respective classification category when semantically segmented using a semantic segmentation model. The initial semantic segmentation scores of each pixel in the initial image constitute the initial semantic segmentation map. Similarly, the first two-dimensional information includes the first RGB information corresponding to each first laser point in the first laser point cloud obtained by projecting the first laser point cloud onto the first image, and the first semantic segmentation score corresponding to each first laser point obtained by projecting the first laser point cloud onto the first semantic segmentation map. The first semantic segmentation score is the probability that each pixel in the first image belongs to its respective classification category when semantically segmented using a semantic segmentation model. The first semantic segmentation scores of each pixel in the first image constitute the first semantic segmentation map.

[0022] In the above embodiments of this application, the nature and origin of two-dimensional information are specifically explained, demonstrating its feasibility.

[0023] In one possible design of the second aspect, the input data of the sparse convolution module can be voxelized first, and then input into the sparse convolution module to perform convolution operation. Therefore, the execution device can voxelize the first laser point cloud to obtain the voxelized first laser point cloud, and then perform convolution operation on the voxelized first laser point cloud through the first sparse convolution module to obtain the first feature.

[0024] In the above embodiments of this application, the input data of the three-dimensional sparse convolution module needs to be voxelized first, and the voxelized input data is then input into the sparse convolution module to perform convolution operation. Therefore, the embodiments of this application first voxelize the first laser point cloud, and then perform convolution operation through the first sparse convolution module to obtain the voxelized first laser point cloud, which has flexibility.

[0025] In one possible design of the second aspect, the execution device will also perform voxelization on the first two-dimensional information to obtain voxelized first two-dimensional information, and then perform convolution operation on the voxelized first two-dimensional information through the second sparse convolution module to obtain the second feature.

[0026] In the above embodiments of this application, not only is it necessary to perform voxelization processing on the first laser point cloud, but also on the first two-dimensional information, which provides flexibility. Furthermore, the embodiments of this application perform voxelization processing on the first laser point cloud and the first two-dimensional information respectively, which is easy to operate.

[0027] In one possible design of the second aspect, the execution device may first concatenate the first laser point cloud and the first two-dimensional information to obtain multimodal information. Then, the execution device performs voxelization processing on the multimodal information to obtain voxelized multimodal information. Obviously, the voxelized multimodal information includes the voxelized first laser point cloud and the voxelized first two-dimensional information. Finally, the execution device performs a convolution operation on the voxelized first laser point cloud through a first sparse convolution module to obtain a first feature, and performs a convolution operation on the voxelized first two-dimensional information through a second sparse convolution module to obtain a second feature.

[0028] In the above embodiments of this application, another voxelization method is described, namely, first concatenating the first laser point cloud with the first two-dimensional information to obtain multimodal information. In this way, voxelizing the multimodal information once is equivalent to voxelizing the first laser point cloud and the first two-dimensional information at the same time, thereby saving one voxelization operation.

[0029] In one possible design of the second aspect, in some embodiments of this application, the input data of the two-dimensional convolution module can be devoxed first. Devoxing is the reverse of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution operation. Specifically, the execution device concatenates the first feature and the second feature to obtain a concatenated feature. Then, the concatenated feature is devoxed to obtain a devoxed concatenated feature. Finally, the execution device inputs the devoxed concatenated feature into the two-dimensional convolution module to perform convolution operation to obtain the third feature, i.e., the fused feature.

[0030] In the above embodiments of this application, the input data of the two-dimensional convolution module needs to be devoxed (i.e., denser representation). Devoxing is the opposite of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution operation, which provides flexibility.

[0031] Training with all training samples in the entire training set once is called a training epoch (i.e., epoch = 1). One epoch is equivalent to training once using all training samples in the training set. The number of epochs indicates how many training epochs have been performed on the training set. Therefore, in one possible design of the second aspect, the above steps can be repeated until the number of training epochs on the first training set reaches a first preset number of epochs. For example, if the first preset number of epochs is 5, then the training device will repeat steps 1001 to 1006 a total of 5 times. The first training set in each epoch is reconstructed based on the initial training set, so that the first training set in each training epoch is not exactly the same (i.e., data augmentation is performed each time), which is used to improve the performance of the 3D detection model. When the number of training rounds on the first training set reaches the first preset number of rounds (e.g., 15 times), the first training set is no longer constructed. Instead, the initial training set is directly used as the new first training set (only data augmentation methods such as rotation and flipping can be retained). The 3D object detection model is iteratively trained according to the above steps based on the initial training set until the number of training rounds using the initial training set reaches the second preset number of rounds (e.g., 5 times).

[0032] In the above embodiments of this application, each epoch requires the construction of a first training set based on the initial training set until the number of the first preset epochs (i.e., the first preset rounds, for example, 15 times) is reached. Then, the initial training set is directly trained until the number of the second preset epochs (i.e., the second prediction rounds, for example, 5 times) is reached. The purpose of this operation is to allow the model to see more real samples, avoid performance loss caused by the large difference between the sample space generated by data augmentation and the original input sample space, and further improve the efficiency of data augmentation.

[0033] A third aspect of this application provides an execution device that has the function of implementing the method described in the first aspect or any possible implementation of the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described function.

[0034] A fourth aspect of this application provides a training device that has the function of implementing the method of the second aspect or any possible implementation of the second aspect described above. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0035] The fifth aspect of this application provides an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to call the program stored in the memory to execute the method of the first aspect of this application or any possible implementation of the first aspect.

[0036] A sixth aspect of this application provides a training device that may include a memory, a processor, and a bus system. The memory is used to store a program, and the processor is used to call the program stored in the memory to execute the method of the second aspect of this application or any possible implementation thereof.

[0037] A seventh aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the method of the first aspect or any possible implementation thereof, or enable the computer to perform the method of the second aspect or any possible implementation thereof.

[0038] The eighth aspect of this application provides a computer program product that, when run on a computer, causes the computer to execute the method of the first aspect or any possible implementation thereof, or causes the computer to execute the method of the second aspect or any possible implementation thereof.

[0039] A ninth aspect of this application provides a chip including at least one processor and at least one interface circuit coupled to the processor. The interface circuit performs transceiver functions and sends instructions to the at least one processor. The at least one processor runs a computer program or instructions, which has the function of implementing the method as described in the first aspect or any possible implementation of the first aspect, or has the function of implementing the method as described in the second aspect or any possible implementation of the second aspect. This function can be implemented in hardware, software, or a combination of hardware and software, and the hardware or software includes one or more modules corresponding to the above functions. Furthermore, the interface circuit is used to communicate with other modules outside the chip. For example, the interface circuit can send a 3D target detection model obtained by the processor on the chip to various intelligent driving agents (e.g., autonomous driving, assisted driving, etc.) for motion planning (e.g., driving behavior decision-making, global path planning, etc.). Attached Figure Description

[0040] Figure 1 A schematic diagram illustrating the difference between 2D object detection and 3D object detection provided in the embodiments of this application;

[0041] Figure 2A schematic diagram illustrating the application of multi-sensor data in autonomous vehicles;

[0042] Figure 3 This is a schematic diagram of an existing 3D object detection method.

[0043] Figure 4 This is another schematic diagram of an existing 3D object detection method;

[0044] Figure 5 A schematic diagram of a two-dimensional convolution module provided in an embodiment of this application;

[0045] Figure 6 A schematic diagram of a three-dimensional sparse convolution module provided in an embodiment of this application;

[0046] Figure 7 A schematic diagram of the main framework of artificial intelligence provided in the embodiments of this application;

[0047] Figure 8 The application flow of the 3D target detection method provided in the embodiments of this application and the model used for 3D target detection;

[0048] Figure 9 A system architecture diagram of the 3D target detection system provided in the embodiments of this application;

[0049] Figure 10 A flowchart illustrating a training method for a 3D object detection model provided in an embodiment of this application;

[0050] Figure 11 A schematic diagram of the process for obtaining two-dimensional information provided in an embodiment of this application;

[0051] Figure 12 A schematic diagram of the training device provided in this application performing voxelization on the first laser point cloud and the first two-dimensional information before performing a convolution operation;

[0052] Figure 13 Another schematic diagram of the training device provided in this application performing voxelization processing on the first laser point cloud and the first two-dimensional information before performing a convolution operation;

[0053] Figure 14 A schematic diagram illustrating an anomaly situation for data augmentation provided in an embodiment of this application;

[0054] Figure 15 This is an example of an adaptive data augmentation strategy proposed in an embodiment of this application;

[0055] Figure 16 A schematic diagram of the 3D target detection method provided in the embodiments of this application;

[0056] Figure 17 A schematic diagram of the execution device provided in this application embodiment performing voxelization processing on the laser point cloud and the two-dimensional information before performing a convolution operation;

[0057] Figure 18 Another schematic diagram of the execution device provided in this application embodiment performing voxelization processing on the laser point cloud and the two-dimensional information before performing a convolution operation;

[0058] Figure 19 A schematic diagram of the output 3D target bounding box and the classification category of the target object provided in the embodiments of this application;

[0059] Figure 20 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0060] Figure 21 A schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0061] Figure 22 A schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0062] Figure 23 A schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0063] Figure 24 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0064] Figure 25 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0065] Figure 26 Another structural schematic diagram of the execution device provided in the embodiments of this application;

[0066] Figure 27 Another structural schematic diagram of the training device provided in the embodiments of this application;

[0067] Figure 28 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0068] This application provides a method and device for 3D target detection, which obtains 2D information corresponding to each laser point in the laser point cloud based on the laser point cloud and the 2D image, and performs feature fusion of the laser point cloud and the 2D information at the feature level. This improves the performance of 3D target detection while retaining the original features of the 3D laser point cloud, enabling the 3D target detection to have good robustness in complex scenarios such as nighttime, rainy or foggy days when the camera fails.

[0069] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0070] Before introducing the embodiments of this application, we will first give a brief introduction to the common methods of 3D object detection to facilitate the understanding of the embodiments of this application.

[0071] (1) One of the common methods for 3D object detection

[0072] The laser point cloud collected by the lidar is directly used as the input data of the 3D target detection model. After the model is processed, the output is the 3D target detection box and the classification category of the target object (also known as the target object) in each 3D target detection box.

[0073] However, this 3D target detection method uses pure LiDAR as input. LiDAR has low resolution and sparse laser point cloud data, which leads to prominent missed detection problems.

[0074] (2) A common method for 3D target detection

[0075] Multiview Figure 3 The Multi-View 3D object detection network (MV3D) proposes a multi-view fusion method based on LiDAR and RGB images (also known as two-dimensional images). Specifically, it uses multimodal sensor data as input to detect target objects in three-dimensional space. Figure 3As shown, this method uses three perspectives. First, the LiDAR-acquired point cloud is converted into a bird's-eye view (BEV) perspective. The BEV reduces the original point cloud by one dimension, transforming it from a 3D point cloud into a 2D image of the bird's-eye view. The second perspective is the front view of the LiDAR. The third is the RGB image acquired by the camera. MV3D uses sensor data from these three perspectives as input data for the model. Features from the 2D image in the BEV perspective are processed by convolutional layers to regress 3D candidate boxes. These 3D candidate boxes are then projected onto feature maps from the other two perspectives and subjected to ROI pooling. The features from the three perspectives are then fused using a deep fusion module. The fused features are used for final classification and regression, outputting 3D object detection boxes and the classification category of the object within each 3D object detection box.

[0076] However, for 3D target detection, the features of the laser point cloud in all three dimensions are crucial. This method loses the z-direction (i.e., height) dimension information after converting the laser point cloud obtained by the lidar into a BEV view, and loses the y-direction dimension information when using the front view of the lidar. This means that the input data always loses one dimension of information, which greatly reduces the performance of 3D target detection.

[0077] (3) The third common method for 3D object detection

[0078] Another 3D object detection method is Frustum-PointNet (F-PointNet), such as... Figure 4As shown, this 3D object detection model consists of three modules: a frustum proposal module, a 3D instance segmentation module, and an amodal 3D box estimation module. First, a mature 2D object detector is used to extract 2D object regions from the RGB image and classify the target objects. Then, using a known camera projection matrix, the 2D object boxes are projected into 3D space to generate a frustum containing the target object in the 3D search space (i.e., the near and far planes specified by the depth sensor range). Next, all laser point clouds within the frustum are collected to form the frustum point cloud. The laser point clouds acquired by the LiDAR in the frustum (each point contains information in the x, y, and z dimensions, as well as the LiDAR reflection intensity) are processed by PointNet to perform 3D instance segmentation on each laser point. The segmented point cloud features are then processed by T-Net to generate translations for more accurate 3D object box estimation. The difference between this method and existing multimodal fusion techniques is that it uses a multi-step training approach to completely decouple the 2D object detection model from the 3D object detection model. It uses a more mature 2D object detector as the basis for generating candidate boxes, reducing the problem of missed detections caused by the sparsity of 3D point clouds.

[0079] However, this method uses the view frustum generated based on the 2D object detector as the input to the 3D detection model, and is completely dependent on the performance of the 2D object detector. Due to the limitations of the RGB camera, it can only be used in simple scenes, and will completely fail in complex scenes (such as at night, in rainy or foggy weather, or under occlusion).

[0080] Current 3D object detection methods can be categorized based on the input data, including pure LiDAR point cloud methods, pure image methods, LiDAR point cloud methods, and image fusion methods. Among these methods, pure LiDAR point cloud methods generally perform better. Common sense dictates that fusion methods should possess more information than pure LiDAR point cloud methods, so why is the detection accuracy lower? This is because most current fusion methods, like MV3D mentioned above, convert the LiDAR point cloud to a BEV viewpoint, losing one dimension of information and significantly impacting the performance of the 3D object detection model. Others, like F-pointnet, rely too heavily on 2D object bounding boxes generated from view frustums for regression, becoming completely ineffective in complex scenes such as nighttime. In short, current fusion methods based on sensor data such as LiDAR point clouds and images are flawed, failing to adequately address the correspondence between 2D and 3D information.

[0081] Since laser point clouds are sparse while images are dense, ensuring the correspondence between these two modalities is crucial. How to rationally and effectively fuse multimodal data is a pressing issue. Furthermore, 2D images captured by cameras become ineffective in complex scenes such as at night. How to make the features obtained from multimodal sensor data fusion more robust to detection performance in such complex scenes is another problem that needs to be solved. Based on this, embodiments of this application provide a 3D target detection method. This method obtains 2D information corresponding to each laser point in the laser point cloud based on the laser point cloud and a 2D image, and then fuses the laser point cloud and 2D information at the feature level. This improves the 3D target detection performance while preserving the original features of the 3D laser point cloud, enabling the 3D target detection to have good robustness in complex scenes such as at night, in rain or fog, where cameras may fail.

[0082] Since the embodiments of this application involve a great deal of knowledge related to object detection, in order to better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below. It should be understood that the explanation of the relevant concepts may be limited due to the specific circumstances of the embodiments of this application, but it does not mean that this application can only be limited to that specific circumstance. The specific circumstances of different embodiments may also differ, and no specific limitation is made here.

[0083] (1) Laser point cloud

[0084] Laser point cloud, also known as laser point cloud data, is the collection of point data on the surface of an object being measured, obtained by measuring instruments. The laser information received by laser sensors such as lidar and 3D laser scanners is presented in the form of a point cloud. If the measuring instrument is a laser sensor, then the obtained point cloud is called a laser point cloud (generally, a 32-line laser will have tens of thousands of laser points at the same time). The laser information contained in the laser point cloud can be denoted as [x, y, z, intensity]. This laser information represents the three-dimensional coordinates of the target position hit by each laser point in the laser coordinate system and the reflection intensity of the laser point, respectively.

[0085] (2) Voxel and voxelization

[0086] A voxel is short for volume pixel. A solid containing voxels can be represented through stereo rendering or by extracting polygonal isosurfaces from a given threshold contour. A voxel is the smallest unit of digital data segmentation in three-dimensional space, used in fields such as 3D imaging, scientific data, and medical imaging. Conceptually, it's similar to the smallest unit in two-dimensional space: a pixel, used in image data for two-dimensional computer graphics. Some true 3D displays use voxels to describe their resolution; for example, a display that can show 512×512×512 voxels.

[0087] Voxelization is the process of converting the geometric representation of an object into a voxel representation that is closest to that object.

[0088] (3) Convolutional Neural Networks (CNN)

[0089] A CNN (Convolutional Neural Network) is a deep neural network with convolutional structures, a deep learning architecture. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a convolutional neural network that processes the input signal through convolution. In a convolutional layer of a convolutional neural network, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units within the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, for all locations in an image, we can use the same learned image information. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0090] Convolutional kernels can be initialized as matrices of random size, and during CNN training, they can learn appropriate weights. Furthermore, sharing weights directly reduces the connections between CNN layers, thus lowering the risk of overfitting.

[0091] (4) Two-dimensional convolution module

[0092] In this embodiment, the two-dimensional convolutional module can be considered a special type of CNN. This convolutional module includes at least a preset number of convolutional layers. Specifically, the image to be recognized is input into the two-dimensional convolutional module, and features are extracted through operations such as convolution and pooling. Figure 5 As shown, an image undergoes convolution to change its receptive field, resulting in multi-channel features. These features are then pooled to flatten the vector into a one-dimensional vector, and finally, a fully connected layer outputs the category information. The two-dimensional convolutional module described in this application can be a specific two-dimensional model such as VGG, ResNet, Iception, or EfficientNet, or other two-dimensional models. The specific form of the two-dimensional convolutional module provided in this application is not limited here.

[0093] (5) 3D sparse convolution module

[0094] Unlike 2D object detection, 3D object detection typically uses 3D laser point clouds as input in addition to 2D images. Unlike images, laser point clouds are spatially sparse (unlike images where pixels are densely and regularly arranged). To facilitate processing this point cloud information, a common approach is to divide the 3D space into a certain number of voxels of the same size and analyze the data at voxel-by-voxel levels (analogous to pixels in an image). Analyzing laser point clouds in 3D space requires 3D convolution; this 3D convolution is called a three-dimensional sparse convolution module, specifically as follows... Figure 6 As shown, Figure 6 This diagram illustrates a 3D sparse convolution module. Its function is similar to a 2D convolution module, but the difference lies in the input: 3D sparse convolution takes two parts as input: coordinates and features. Since a single voxel may contain multiple laser point data points or none, and considering the KITTI dataset (a mainstream publicly available 3D detection dataset), at commonly used voxel sizes, only 0.5% of voxels contain valid laser point data. Therefore, due to the sparsity of laser point clouds, a 3D sparse convolution module is needed to improve computational speed.

[0095] Taking the application of sparse convolution modules in the field of autonomous driving as an example: After the laser point cloud collected by the LiDAR deployed on autonomous vehicles is divided into voxels, it will generate about 5k to 8k voxels and a sparsity of about 0.005. Directly using 3D convolution will consume a huge amount of computing time and memory. Sparse convolution limits the sparsity of the output by the sparsity of the input data, thereby greatly reducing the amount of computation in subsequent convolution operations.

[0096] The three-dimensional sparse convolution module described in the embodiments of this application can be a specific three-dimensional model such as second, cbgs, centerpoint, or other three-dimensional models. The specific form of the three-dimensional sparse convolution module provided in this application is not limited here.

[0097] (6) Loss Function

[0098] In training a neural network (e.g., a CNN), to ensure the network's output closely approximates the desired predicted value, we compare the network's current prediction with the target value. Based on the difference, we update the weight matrix of each layer (usually with initialization before the first update, where parameters are pre-configured for each layer). For example, if the network's prediction is too high, the weight matrix is ​​adjusted to predict a lower value, and this process continues until the network accurately predicts the target value. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the basis of the loss function or objective function. These are crucial equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0099] (7) Backpropagation algorithm

[0100] During the training of a neural network, the back propagation (BP) algorithm can be used to correct the parameters in the initial neural network model, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss. By backpropagating this error loss information, the parameters in the initial neural network model are updated, thus bringing the error loss to a convergence. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0101] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0102] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 7 , Figure 7The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0103] (1) Infrastructure

[0104] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0105] (2) Data

[0106] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0107] (3) Data processing

[0108] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0109] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0110] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0111] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0112] (4) General ability

[0113] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0114] (5) Smart Products and Industry Applications

[0115] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, autonomous driving, and safe cities.

[0116] This application can be applied to the field of computer vision within the field of artificial intelligence, specifically involving object detection. Specifically, it combines... Figure 7 In this embodiment of the application, the data acquired by the infrastructure is the image to be detected and the laser point cloud. Then, through a series of operations involved in the 3D target detection method provided in this embodiment of the application, a 3D target box and the classification category to which the target object in the 3D target box belongs are obtained.

[0117] The basic principles of the 3D target detection method provided in the embodiments of this application will be introduced below. Please refer to [link / reference needed] for details. Figure 8 , Figure 8 The application process of the 3D object detection method and the model used for 3D object detection provided in this application embodiment can be referred to as a 3D object detection model. Specifically, the 3D object detection model may include a three-dimensional first sparse convolution module 801, a three-dimensional second sparse convolution module 802, a two-dimensional convolution module 803, and a classification and regression module 804. Based on this 3D detection model, the acquired two-dimensional image is first semantically segmented to obtain a semantic segmentation map. This semantic segmentation map is composed of the semantic segmentation scores of each pixel in the image, and the semantic segmentation scores represent the probability that each pixel belongs to its respective classification category. Then, according to projection transformation, the 3D laser point cloud (also referred to as 3D information) is projected onto the image and the semantic segmentation map respectively, obtaining the RGB information and corresponding semantic segmentation score for each laser point. The RGB information and corresponding semantic segmentation score for each laser point constitute two-dimensional information (also referred to as 2D information). After voxelization processing, the 3D information and 2D information are then input into two three-dimensional sparse convolution modules (i.e., Figure 8 The first sparse convolution module 801 and the second sparse convolution module 802 shown above obtain the first feature and the second feature, respectively. The first feature and the second feature are then concatenated and input into a two-dimensional convolution module (i.e., ...). Figure 8The convolutional module 803 shown obtains a third feature (also known as a fusion feature). This third feature is then concatenated with the first and second features to obtain a fourth feature (also known as a combined feature). Finally, the fourth feature is input as shown in the diagram. Figure 8 The classification regression module 804 shown (e.g., the classification regression module 804 can specifically be a classification regression head) obtains the 3D target detection box and the classification category to which the target object within the 3D target box belongs.

[0118] To facilitate understanding this solution, we will first combine... Figure 9 The system architecture of the 3D target detection system provided in the embodiments of this application is described below. Please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a system architecture diagram of a 3D target detection system 200 provided in an embodiment of this application. Figure 9 In the 3D target detection system 200, there are execution devices 210, training devices 220, databases 230, client devices 240, data storage systems 250 and data acquisition devices 260. The execution devices 210 include a computing module 211 and an input / output (I / O) interface 212. The computing module 211 is essentially the 3D target detection model 201 provided in the embodiments of this application.

[0119] During the training phase, the data acquisition device 260 can acquire a large-scale open-source dataset (i.e., a training set) required by the user and store the dataset in the database 230. Each training data point (i.e., a training sample) in the dataset includes a 3D laser point cloud, a 2D image, and two-dimensional information corresponding to each frame of the laser point cloud. The training device 220 then iteratively trains the 3D object detection model 201 based on the dataset maintained in the database 230 to obtain a mature 3D object detection model 201, i.e., the trained 3D object detection model 201. This 3D object detection model 201 may specifically include the features described above. Figure 8 The three-dimensional first sparse convolution module 801, the three-dimensional second sparse convolution module 802, the two-dimensional convolution module 803, and the classification and regression module 804 are described. The 3D object detection model 201, trained by the training device 220, can be applied to different systems or devices.

[0120] During the inference phase, the data acquisition device 260, such as a camera or lidar mounted on a wheeled mobile device, can acquire target data (in this embodiment, the target data is a 3D laser point cloud and a 2D image) and store it in the data storage system 250. The execution device 210 can call the data, code, etc. in the data storage system 250 for processing, or it can store data, instructions, etc. in the data storage system 250. The data storage system 250 can be located in the execution device 210, or it can be an external memory relative to the execution device 210. The calculation module 211 processes the laser point cloud and target image stored in the data storage system 250 using the trained 3D target detection model 201 to obtain a 3D target bounding box and the classification category of the target object within the 3D target bounding box (i.e., obtain the detection result), and sends the detection result to an external device, such as a mobile phone or personal computer client device 240, through the I / O interface 212.

[0121] In some embodiments of this application, the "user" can also input data to the I / O interface 212 through the client equipment 240. For example, the client device 240 can be a camera and LiDAR mounted on an autonomous vehicle. Images captured by the camera and laser point clouds collected by the LiDAR are input data to the computing module 211 of the execution device 210. The 3D target detection model 201 in the computing module 211 performs 3D target detection on the input image and laser point clouds to obtain the detection result, which is then output to the camera or displayed directly on the display interface of the execution device 210 (if applicable). Furthermore, in some embodiments of this application, the client device 240 can also be integrated into the execution device 210. For example, when the execution device 210 is an autonomous vehicle, it can directly capture images and laser point clouds collected by the LiDAR through the autonomous vehicle's camera or receive images and laser point clouds sent by other devices (e.g., mobile phones). The computing module 211 within the autonomous vehicle then performs 3D target detection on the image and laser point clouds to obtain the detection result, which is then directly displayed on the mobile phone's display interface. The product form of the execution device 210 and the client device 240 is not limited here.

[0122] It is worth noting that Figure 9 This is merely a schematic diagram of a 3D target detection system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 9 In this context, the data storage system 250 is an external memory relative to the execution device 210; however, in other cases, the data storage system 250 may be placed within the execution device 210. Figure 9In this context, the client device 240 is an external device relative to the execution device 210. In other cases, the client device 240 may also be integrated into the execution device 210.

[0123] It should also be noted that in some embodiments of this application, the 3D target detection system 200 can be further divided into multiple sub-modules / sub-units to jointly implement the solution provided in the embodiments of this application, and no specific limitation is made here.

[0124] It should also be noted that the training of the 3D object detection model 201 described in the above embodiments can all be implemented on the cloud side. For example, the training set can be obtained by the cloud-side training device 220 (which can be set on one or more servers or virtual machines), and the 3D object detection model 201 can be trained based on the training data in the training set to obtain the trained 3D object detection model 201. Then, the trained 3D object detection model 201 is sent to the execution device 210 for application. For example, the method of sending the model to the execution device 210 to perform 3D object detection is exemplarily described. Figure 9 The corresponding system architecture describes a system where the training device 220 trains the 3D object detection model 201 as a whole, and the trained 3D object detection model 201 is then sent to the execution device 210 for use. The training of the 3D object detection model 201 described in the above embodiments can also be implemented entirely on the terminal side. That is, the training device 220 can be located on the terminal side. For example, a training set can be obtained by a terminal device (such as a mobile phone, smartwatch, etc.) or a wheeled mobile device (such as an autonomous vehicle, assisted driving vehicle, etc.), and the 3D object detection model 201 can be trained based on multiple training data within the training set to obtain the trained 3D object detection model 201. This trained 3D object detection model 201 can be used directly on the terminal device, or it can be sent by the terminal device to other devices for use. Specifically, this application embodiment does not limit on which device (cloud side or terminal side) the 3D object detection model 201 is trained or applied on.

[0125] It should be noted that, in some embodiments of this application, the execution device 210, which is equipped with the trained 3D target detection model 201, may also include a camera and a lidar, for collecting environmental information around the execution device, thereby obtaining the target image and target lidar to be input.

[0126] It should also be noted that, in the above embodiments of this application, the 3D object detection model 201 trained by the training device 220 can be applied to different systems or devices (i.e., execution device 210). For example, the execution device 210 can be various terminal-side devices with a display interface, such as cameras, video recorders, smart home devices, etc. The execution device 210 can also be end-side devices such as mobile phones, personal computers, computer workstations, tablet computers, smart wearable devices (such as smartwatches, smart bracelets, smart headphones, etc.), game consoles, set-top boxes, media consumption devices, etc. The execution device 210 can also be various wheeled mobile devices (wheeled construction equipment, autonomous vehicles, assisted driving vehicles, etc.). Autonomous vehicles can also be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawnmowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, and handcarts, etc. Any device capable of deploying the 3D object detection model 201 described in the embodiments of this application can be considered as the execution device 210 described in the embodiments of this application, and no specific limitation is made here.

[0127] Based on the above description, embodiments of this application provide a 3D object detection method and a training method for a 3D object detection model, which can be applied to the inference and training phases of the 3D object detection model, respectively. Since the processes of the training and inference phases differ, the two phases will be described separately below.

[0128] I. Training Phase

[0129] In this embodiment, the training phase is as described above. Figure 9 The process by which the training device 210 uses training data to train the 3D object detection model 201. Please refer to [link / reference] for details. Figure 10 , Figure 10 This is a flowchart illustrating a training method for a 3D object detection model provided in an embodiment of this application. The 3D object detection model includes a three-dimensional first sparse convolution module, a three-dimensional second sparse convolution module, a two-dimensional convolution module, and a classification and regression module. The method may include the following steps:

[0130] 1001. Construct a first training set based on the initial training set. The first training sample in the first training set is a training sample obtained by extracting the target object from each initial training sample in the initial training set and randomly copying the target object into each initial training sample. The initial training samples include the initial laser point cloud and the initial two-dimensional information. The first training sample includes the first laser point cloud and the first two-dimensional information.

[0131] First, the training device acquires an initial training set. Any training sample in this initial training set can be called an initial training sample. Each initial training sample in the initial training set includes a frame of laser point cloud and two-dimensional information (which can be called initial laser point cloud and initial two-dimensional information). The initial two-dimensional information is obtained based on the initial laser point cloud and the initial image. The initial laser point cloud and the initial image correspond to each other, that is, they are sensor data collected by different types of sensors at the same time and location.

[0132] Subsequently, the training device constructs a first training set based on the initial training set. The construction process involves extracting target objects, such as local laser points and local two-dimensional information constituting target objects like "cars," "trucks," and "pedestrians," from each initial training sample in the initial training set. The local laser points and local two-dimensional information of each target object are then randomly copied into each initial training sample to obtain the first training sample. Each of these first training samples constitutes the first training set described in this application. Similarly, each first training sample in this first training set also includes a frame of laser point cloud and a two-dimensional information (which can be referred to as the first laser point cloud and the first two-dimensional information).

[0133] Specifically, in some embodiments of this application, such as Figure 11 As shown, the initial two-dimensional information includes the initial RGB information corresponding to each initial laser point in the initial laser point cloud obtained by projecting the initial laser point cloud onto the initial image, and the initial semantic segmentation score corresponding to each initial laser point obtained by projecting the initial laser point cloud onto the initial semantic segmentation map. The initial semantic segmentation score is the probability that each pixel in the initial image belongs to its respective classification category when the initial image is semantically segmented by the semantic segmentation model. The initial semantic segmentation scores of each pixel in the initial image constitute the initial semantic segmentation map.

[0134] Similarly, the first two-dimensional information includes the first RGB information corresponding to each first laser point in the first laser point cloud obtained by projecting the first laser point cloud onto the first image, and the first semantic segmentation score corresponding to each first laser point obtained by projecting the first laser point cloud onto the first semantic segmentation map. The first semantic segmentation score is the probability that each pixel in the first image belongs to its respective classification category when the first image is semantically segmented by the semantic segmentation model. The first semantic segmentation scores of each pixel in the first image constitute the first semantic segmentation map.

[0135] It should be noted that, in this embodiment, the process of constructing the first training set based on the initial training set can also be called ground truth augmentation (gt-augmentation), a commonly used data augmentation method in 3D object detection. Its principle is to increase the number of training sample instances by randomly copying and pasting the local laser point cloud and corresponding 2D information of target objects such as "cars," "motorcycles," and "bicycles" from the initial training samples into each initial training sample, thereby solving the problem of imbalanced sample size. The process is as follows: During training, since each frame of the initial laser point cloud corresponds to each initial image (e.g., an image and a frame of laser point cloud taken from the vehicle's position at a certain moment), gt-augmentation extracts the local laser point cloud of a certain object (e.g., a bicycle) from each frame of the laser point cloud in all initial training samples, along with the local pixels of the "bicycle" in the corresponding image and the local semantic segmentation score of the "bicycle" on the corresponding semantic segmentation map (which can be considered as extraction together), and randomly pastes them into other initial training samples.

[0136] 1002. Perform a convolution operation on the first laser point cloud using the first sparse convolution module to obtain the first feature, and perform a convolution operation on the first two-dimensional information using the second sparse convolution module to obtain the second feature.

[0137] After the training device constructs the first training set based on the initial training set, the first training samples in the first training set are all data-augmented training samples. The first training samples include the first laser point cloud and the first two-dimensional information. The training device then performs convolution operations on the first laser point cloud and the first two-dimensional information through the first sparse convolution module and the second sparse convolution module, respectively, to obtain the first feature corresponding to the first laser point cloud and the second feature corresponding to the first two-dimensional information, respectively.

[0138] It should be noted that, in some embodiments of this application, the input data of the sparse convolution module can be voxelized first, and then input into the sparse convolution module to perform convolution operations. Therefore, the specific ways in which the training device performs convolution operations on the first laser point cloud and the first two-dimensional information through the first sparse convolution module and the second sparse convolution module respectively can be, but are not limited to, the following:

[0139] Method 1, please refer to the details. Figure 12 , Figure 12This diagram illustrates how a training device performs voxelization on both the first laser point cloud and the first two-dimensional information before executing a convolution operation. Specifically, the training device voxelizes the first laser point cloud to obtain a voxelized first laser point cloud, and then performs a convolution operation on the voxelized first laser point cloud using a first sparse convolution module to obtain a first feature. Similarly, the training device voxelizes the first two-dimensional information to obtain a voxelized first two-dimensional information, and then performs a convolution operation on the voxelized first two-dimensional information using a second sparse convolution module to obtain a second feature.

[0140] Method 2, please refer to the details. Figure 13 , Figure 13 This is another schematic diagram illustrating the training device performing voxelization on the first laser point cloud and the first two-dimensional information before performing a convolution operation. Specifically, the training device first concatenates the first laser point cloud and the first two-dimensional information to obtain multimodal information. Then, the training device performs voxelization on the multimodal information to obtain voxelized multimodal information. Obviously, the voxelized multimodal information includes the voxelized first laser point cloud and the voxelized first two-dimensional information. Finally, the training device performs a convolution operation on the voxelized first laser point cloud through a first sparse convolution module to obtain the first feature, and performs a convolution operation on the voxelized first two-dimensional information through a second sparse convolution module to obtain the second feature.

[0141] 1003. The first and second features are concatenated and then input into the convolution module for convolution operation to obtain the third feature.

[0142] After the training device obtains the first feature corresponding to the first laser point cloud and the second feature corresponding to the first two-dimensional information, it will concatenate the first feature and the second feature, and input the concatenated first feature and the second feature into the two-dimensional convolution module for convolution operation to obtain the third feature. Since the third feature integrates the first feature and the second feature, it can also be called the fused feature.

[0143] It should be noted that, in the embodiments of this application, the cascading of features refers to superimposing features to obtain a new cascaded feature. For example, assuming that the first feature is 1×2×3 and the second feature is also 1×2×3, where 1 represents the number of channels and 2×3 represents the size of the first / second feature, then the cascaded feature obtained by cascading the first feature and the second feature is 2×2×3, where 2 is the number of channels after cascading and 2×3 is the size of the cascaded feature.

[0144] It should be noted that, in some embodiments of this application, the input data of the two-dimensional convolution module can be devoxed first. Devoxing is the reverse of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution. Specifically, the training device concatenates the first and second features to obtain concatenated features, then devoxes the concatenated features to obtain devoxed concatenated features. Finally, the training device inputs the devoxed concatenated features into the two-dimensional convolution module to perform convolution to obtain the third feature, i.e., the fused feature.

[0145] 1004. Concatenate the first feature, the second feature, and the third feature to obtain the fourth feature.

[0146] After the training device obtains the third feature, it concatenates the previously obtained first, second, and third features to obtain the fourth feature, which can also be called a combined feature. The concatenation method is similar to the concatenation method of the first and second features mentioned above, and will not be elaborated here.

[0147] 1005. Input the fourth feature into the classification and regression module to obtain the 3D predicted target box and the predicted classification category of the target object within the 3D predicted target box.

[0148] After the training device obtains the fourth feature, it can be input into the classification and regression module, such as the classification and regression head, to obtain the 3D predicted bounding box and the predicted classification category of the target object within the 3D predicted bounding box. The target object can also be referred to as the target object. For example, using... Figure 1 The image on the right illustrates this: Two 3D predicted target boxes are obtained, with their specific dimensions and angles as shown below. Figure 1 As shown in the right figure, the target objects within these two 3D predicted bounding boxes belong to the predicted classification categories of "car" and "tricycle", respectively.

[0149] 1006. Based on the 3D ground truth bounding box, the ground truth classification category of the target object within the 3D ground truth bounding box, the 3D predicted bounding box, and the predicted classification category of the target object within the 3D predicted bounding box, the 3D target prediction model is iteratively trained using the target loss function.

[0150] After obtaining the 3D predicted bounding box and the predicted classification category of the target object within the 3D predicted bounding box, the training device will iteratively train the 3D target prediction model using the target loss function based on the 3D ground truth bounding box, the ground truth classification category of the target object within the 3D ground truth bounding box, the 3D predicted bounding box and the predicted classification category of the target object within the 3D predicted bounding box.

[0151] 1007. Repeat steps 1001 to 1006 until the training rounds of the first training set reach the first preset round.

[0152] Training all training samples in the entire training set once is called a training epoch (i.e., epoch = 1). One epoch is equivalent to training once using all training samples in the training set. The number of epochs indicates how many training epochs have been performed on the training set. Therefore, in some embodiments of this application, steps 1001 to 1006 can be repeated until the number of training epochs on the first training set reaches a first preset number of epochs. For example, if the first preset number of epochs is 5, then the training device will repeat steps 1001 to 1006 a total of 5 times. The first training set in each epoch is reconstructed based on the initial training set, so that the first training set in each training epoch is not exactly the same (i.e., data augmentation is performed each time), which is used to improve the performance of the 3D detection model.

[0153] 1008. Using the initial training set as the new first training set, repeat steps 1001 to 1006 until the training rounds for the new first training set reach the second prediction round.

[0154] It should be noted that in some embodiments of this application, an adaptive data augmentation strategy is also proposed. This is because the principle of the gt-augmentation data augmentation method is to increase the number of instances in the training samples by randomly copying and pasting the local laser point cloud and corresponding two-dimensional information occupied by target objects such as "cars," "motorcycles," and "bicycles" in the initial training samples to each initial training sample, in order to solve the problem of imbalanced sample numbers. However, there are many unreasonable situations in the pasting process, such as the pasted position overlapping with the original object, making it impossible for the model to recognize these abnormal samples, resulting in a decrease in recognition performance. Figure 14 As shown, Figure 14This diagram illustrates an anomaly in data augmentation. The motorcycle and bicycle are duplicated augmented samples, but their placement blends seamlessly with existing obstacles (e.g., walls), which negatively impacts model training. Therefore, when the first training set reaches a first preset number of training epochs (e.g., 15), the ground truth augmentation method is no longer used to construct the first training set. Instead, the initial training set is used as the new first training set (retaining only data augmentation methods like rotation and flipping). The 3D object detection model is iteratively trained using the initial training set according to steps 1001 to 1006 until the second preset number of training epochs (e.g., 5) is reached. This allows the 3D object detection model to encounter more real training samples, avoiding performance loss due to significant differences between the data augmented sample space and the original input sample space, thus further improving the efficiency of data augmentation. For example, it can be done as follows: Figure 15 As shown, Figure 15 As an example of the adaptive data augmentation strategy proposed in this application, for categories such as motorcycle and bicycle with small size (i.e., sparse initial laser point cloud) and few instances, the gt-augmentation method is no longer used to augment the data when three-quarters of the training process has been completed (assuming the full epoch = 20, then it is at epoch = 15). Instead, the model is iteratively trained using the initial training set in the remaining 5 training epochs. The adaptive data augmentation strategy described in this application can reduce the impact of such unreasonable augmentation samples, thereby improving the efficiency of data augmentation and the performance of the 3D object detection model.

[0155] II. Reasoning Stage

[0156] Based on the training phases described above, the training device will obtain a trained 3D object detection model. This trained model can then be used to perform 3D object detection on the device. For details, please refer to [link / reference needed]. Figure 16 , Figure 16 The 3D target detection method provided in this application embodiment may include the following steps:

[0157] 1601. Obtain two-dimensional information corresponding to the laser point cloud based on the acquired image and laser point cloud.

[0158] The execution device first acquires an image and laser point cloud at a certain location and time using sensor devices (such as cameras, lidar, etc.) deployed on the execution device. Then, it obtains two-dimensional information corresponding to the laser point cloud based on the acquired image and laser point cloud.

[0159] Specifically, the execution device obtains the two-dimensional information corresponding to the laser point cloud based on the acquired image and the laser point cloud through the following process: First, the execution device performs semantic segmentation on the acquired image using a semantic segmentation model to obtain the semantic segmentation score of each pixel in the image. This segmentation score represents the probability that each pixel belongs to its respective classification category, and the semantic segmentation scores of each pixel in the image constitute a semantic segmentation map. Then, the execution device projects the laser point cloud onto the image and the semantic segmentation map obtained based on the image, respectively, to obtain the target RGB information and target semantic segmentation score corresponding to each laser point cloud. This target RGB information and target semantic segmentation score constitute the two-dimensional information.

[0160] 1602. Perform a convolution operation on the laser point cloud using a three-dimensional first sparse convolution module to obtain the first feature, and then perform a convolution operation on the two-dimensional information using a three-dimensional second sparse convolution module to obtain the second feature.

[0161] After acquiring the laser point cloud and the corresponding two-dimensional information, the execution device performs convolution operations on the laser point cloud and the two-dimensional information using a first three-dimensional sparse convolution module and a second three-dimensional sparse convolution module, respectively, to obtain a first feature corresponding to the laser point cloud and a second feature corresponding to the two-dimensional information. This process is similar to the one described above. Figure 10 Step 1002 is similar and will not be repeated here.

[0162] It should be noted that, in some embodiments of this application, the input data of the sparse convolution module can be voxelized first, and then input into the sparse convolution module to perform the convolution operation. Therefore, the specific way in which the execution device performs the convolution operation on the laser point cloud and the two-dimensional information through the first sparse convolution module and the second sparse convolution module respectively can be, but is not limited to, the following:

[0163] Method 1, please refer to the details. Figure 17 , Figure 17 This diagram illustrates how the execution device performs voxelization on both the laser point cloud and the 2D information before performing a convolution operation. Specifically, the execution device voxelizes the laser point cloud to obtain a voxelized laser point cloud, and then performs a convolution operation on the voxelized laser point cloud through a first sparse convolution module to obtain a first feature. Similarly, the execution device also voxelizes the 2D information to obtain a voxelized 2D information, and then performs a convolution operation on the voxelized 2D information through a second sparse convolution module to obtain a second feature.

[0164] Method 2, please refer to the details. Figure 18 , Figure 18This is another schematic diagram illustrating the execution device performing voxelization on the laser point cloud and the 2D information before performing a convolution operation. Specifically, the execution device first concatenates the laser point cloud and the 2D information to obtain multimodal information. Then, the execution device performs voxelization on the multimodal information to obtain voxelized multimodal information. Obviously, the voxelized multimodal information includes the voxelized laser point cloud and the voxelized 2D information. Finally, the execution device performs a convolution operation on the voxelized laser point cloud through a first sparse convolution module to obtain a first feature, and performs a convolution operation on the voxelized 2D information through a second sparse convolution module to obtain a second feature.

[0165] 1603. The first and second features are concatenated and then input into a two-dimensional convolution module for convolution operation to obtain the third feature.

[0166] After the execution device obtains the first feature corresponding to the laser point cloud and the second feature corresponding to the two-dimensional information, it will concatenate the first feature and the second feature, and then input the concatenated first feature and the second feature into the two-dimensional convolution module for convolution operation to obtain the third feature. Since the third feature is a fusion of the first feature and the second feature, it can also be called a fusion feature.

[0167] It should be noted that, in the embodiments of this application, the cascading of features refers to superimposing features to obtain a new cascaded feature. For example, assuming that the first feature is 1×2×3 and the second feature is also 1×2×3, where 1 represents the number of channels and 2×3 represents the size of the first / second feature, then the cascaded feature obtained by cascading the first feature and the second feature is 2×2×3, where 2 is the number of channels after cascading and 2×3 is the size of the cascaded feature.

[0168] It should be noted that, in some embodiments of this application, the input data of the two-dimensional convolution module can be devoxed first. Devoxing is the reverse of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution operation. Specifically, the execution device concatenates the first and second features to obtain a concatenated feature. Then, the concatenated feature is devoxed to obtain a devoxed concatenated feature. Finally, the execution device inputs the devoxed concatenated feature into the two-dimensional convolution module to perform convolution operation to obtain the third feature, i.e., the fused feature.

[0169] 1604. Concatenate the first feature, the second feature, and the third feature to obtain the fourth feature.

[0170] After the execution device obtains the third feature, it cascades the previously obtained first, second, and third features to obtain the fourth feature, which can also be called a combined feature. The cascading method is similar to the cascading method of the first and second features mentioned above, and will not be elaborated here.

[0171] 1605. Input the fourth feature into the classification and regression module to obtain the 3D target box and the classification category of the target object within the 3D target box.

[0172] After the execution device obtains the fourth feature, it can input this fourth feature into the classification and regression module, such as the classification and regression head, to output the 3D bounding box and the classification category of the target object within the 3D bounding box. The target object can also be referred to as the target object. For example, using... Figure 19 For example, the resulting 3D target bounding boxes are 3 in total, and their specific dimensions and angles are as follows: Figure 19 As shown, the target objects within these three 3D target boxes belong to the categories of "car", "car", and "bicycle", respectively.

[0173] In the above embodiments of this application, the execution device first obtains the 2D information corresponding to each laser point in the laser point cloud (i.e., 3D information). Then, the 3D information and the 2D information are respectively input into two three-dimensional sparse convolutional modules to obtain a first feature and a second feature. The first feature and the second feature are then concatenated and input into a two-dimensional convolutional module to obtain a third feature (i.e., fused feature). The third feature is then concatenated with the first and second features to obtain a fourth feature (i.e., combined feature). Finally, the fourth feature is used to achieve 3D target detection through a classification and regression module. This application is used to obtain the 2D information corresponding to each laser point in the laser point cloud based on the laser point cloud and the 2D image, and to fuse the laser point cloud and the 2D information at the feature level. This improves the performance of 3D target detection while retaining the original features of the 3D laser point cloud, enabling the 3D target detection to have good robustness in complex scenarios such as nighttime, rainy or foggy weather where the camera is malfunctioning.

[0174] For ease of understanding Figure 16The 3D target detection method described in the corresponding embodiment is illustrated below with an example: First, an execution device equipped with a camera and a LiDAR acquires a frame of laser point cloud and an image at a certain moment through the camera and the LiDAR, respectively. For the image, the execution device first uses a 2D semantic segmentation model to obtain a semantic segmentation map of the image. This semantic segmentation map is composed of the semantic segmentation score corresponding to each pixel in the image. For example, the nuScenes dataset has 11 categories (i.e., car, pedestrian, bus, barrier, traffic cone, truck, trailer, motorcycle, construction vehicle, bicycle, background), then the semantic segmentation score of each pixel is the probability of belonging to these 11 categories.

[0175] Each 3D laser point in the laser point cloud acquired by the execution device is represented as (x, y, z, r) or (x, y, z, r, t), where [x, y, z] represents the position coordinates of the laser point in three-dimensional space, r represents the reflection intensity of the laser point, and t represents the time series, that is, the moment when the lidar scans the object, i.e., the moment when the lidar obtains [x, y, z]. The laser point cloud is projected onto the image and the semantic segmentation map obtained based on the image through a projection transformation (e.g., translation, rotation, etc.). The complete transformation process is shown in the formula:

[0176]

[0177] Where T is the coordinate matrix, ego←lidar represents the transformation from the lidar coordinate system to the vehicle coordinate system, and ego... tc ←ego tl This indicates the transformation from the vehicle coordinate system corresponding to the LiDAR to the vehicle coordinate system corresponding to the camera; camera←ego indicates the transformation from the vehicle coordinate system to the camera coordinate system.

[0178] After projecting the laser point cloud onto the image and semantic segmentation map, the RGB information corresponding to each laser point in the laser point cloud and the semantic segmentation score corresponding to each laser point can be obtained. The RGB information and semantic segmentation score corresponding to each laser point are used as two-dimensional information (i.e., the 2D input mentioned below). The two-dimensional information is actually finding the one-to-one correspondence between the laser point and each pixel on the image. Then, the 2D information is concatenated with the original 3D laser point cloud (i.e., the 3D input mentioned below) to obtain multimodal information (i.e., the Multimodal input mentioned below). In some embodiments, concatenation is not required. The advantage of concatenation is that only voxelization processing is required for the multimodal information. If concatenation is not performed, voxelization processing needs to be performed separately for the laser point cloud and the 2D information. The representations of the two-dimensional information, the laser point cloud, and the multimodal information are as follows:

[0179] 3D input: (x,,y,z,r,t)

[0180] 2D input: (r, g, b, car, pedestrian, bus, barrier, …, background)

[0181] Multimodal input: (x, y, z, r, t, r, g, b, car, pedestrian, bus, barrier,…, background)

[0182] Wherein, 3D input is a laser point in the laser point cloud, 2D input is the 2D information corresponding to each laser point after the laser point cloud is projected twice, [car,pedestrian,bus,barrier,...,background] is the semantic segmentation score of a pixel corresponding to 3D input, and [r,g,b] is the value of a pixel corresponding to 3D input.

[0183] Subsequently, the execution device voxelizes the 2D input and 3D input separately, or directly voxelizes the Multimodal input once. Then, the voxelized laser point cloud and the voxelized 2D information are respectively passed through two 3D sparse convolutional modules (i.e., the first and second sparse convolutional modules mentioned above). These two sparse convolutional modules only need to ensure the same number of channels; the number of convolutional layers can be different. After convolution operations by these two sparse convolutional modules, the first feature corresponding to the laser point cloud (i.e., the 3D feature mentioned below) and the second feature corresponding to the 2D information (i.e., the 2D feature mentioned below) are obtained respectively. Then, the two features are concatenated (i.e., channel concatenation) and then passed through a 2D convolutional module (the convolutional layers of this module can be preset) to obtain a third feature after fusing these two features (also called the fused feature, i.e., the Multimodal feature mentioned below). The fused feature is then concatenated with the first and second features (i.e., channel concatenation, similar to the above) to obtain the final fourth feature (also known as the combined feature, or Combined feature as described below). The representations of the first, second, third, and fourth features are as follows:

[0184] 3Dfeature = Conv3D(Input3D)

[0185] 2Dfeature = Conv3D(Input2D)

[0186] Multimodal feature=Conv2D(Concatenate(3D feature, 2D feature))

[0187] Combined feature=Concatenate(3D feature, 2D feature, Multimodalfeature)

[0188] Finally, the fourth feature (Combinated feature) is input into the classification and regression module (e.g., into the classification and regression head), thereby outputting the 3D bounding box and the classification category of the target object within the 3D bounding box. The target object can also be referred to as the target object. In this embodiment, because the fourth feature incorporates the features of the original laser point cloud (i.e., the first feature), and the features of the original laser point cloud can ensure the normal operation of the 3D target detection model even when the camera fails in conditions such as nighttime, rain, or fog, the 3D target detection model exhibits good robustness even in special scenarios.

[0189] To provide a more intuitive understanding of the beneficial effects of the embodiments of this application, the following further compares the technical effects of the embodiments of this application. To fairly compare the merits of various 3D object detection algorithms, this application verifies the effectiveness of the 3D object detection method provided in the embodiments of this application on the nuScenes and KITTI datasets. nuScenes is a large-scale autonomous driving dataset released by the autonomous driving company nuTonomy in 2019. This dataset includes not only camera and LiDAR data but also radar data. It consists of 1000 scenes, containing 1.4 million images, 400,000 LiDAR scans (to determine the distance between objects), and 1.1 million 3D bounding boxes (objects detected using a combination of RGB cameras, radar, and LiDAR). The object detection categories are 10, and there is a serious class imbalance problem, making it very practically significant. The KITTI dataset was jointly created by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America. This dataset is used to evaluate the performance of computer vision technologies such as stereo imaging, optical flow, visual odometry, 3D object detection, and 3D tracking in an in-vehicle environment. KITTI contains real-world image data collected from urban, rural, and highway scenes, with up to 15 vehicles and 30 pedestrians per image, and varying degrees of occlusion and truncation. The 3D object detection training set contains 3712 images, the validation set contains 3769, and the test set contains 7518. Table 1 below shows the detection results of the 3D object detection method provided in this application and other existing 3D object detection methods on the nuScenes dataset, while Table 2 shows the detection results of the 3D object detection method provided in this application and other existing 3D object detection methods on the KITTI dataset.

[0190] Table 1: Detection performance of the 3D object detection method in this application and other 3D object detection methods on the nuScenes dataset.

[0191]

[0192] Table 2: Detection performance of the 3D object detection method of this application and other 3D object detection methods on KITTI

[0193]

[0194] As shown in Table 1, the 3D object detection method provided in this application participated in the 2020 nuScenes detection challenge and achieved first place. The results are shown in Table 1. The map performance surpasses the second-place CenterPoint mAP by 3 points and NDS by 1.5 points, and surpasses last year's challenge champion CBGS (MEGII) map by 12 points and NDS by 5.7 points. Furthermore, the 3D object detection method proposed in this application was validated on the KITTI dataset using the SECOND model as a baseline. The detection performance is shown in Table 2, with stable improvements in both 3D object detection and BEV detection results, especially significant improvements in pedestrian and cyclist detection.

[0195] Since the 3D object detection method and the trained 3D object detection model provided in this application embodiment can be used in fields such as intelligent security, safe city, and smart terminals to perform 3D detection on various objects, the following will introduce several application scenarios that have been implemented in products.

[0196] (1) Autonomous driving scenario

[0197] Autonomous driving is a very popular research area. With economic development, the number of cars worldwide is constantly increasing, leading to more and more traffic congestion, parking difficulties, taxi shortages, and frequent accidents. Autonomous driving technology has become the latest development direction for the entire automotive industry. Applying autonomous driving technology can comprehensively improve the safety and comfort of driving, and meet higher-level market demands. Autonomous driving is a complete hardware and software interactive system. Core technologies include hardware (automotive manufacturing technology, autonomous driving chips), autonomous driving software, high-precision maps, sensor communication networks, etc. Figure 20 As shown, Figure 20 This illustration shows a top-down, layered architecture for autonomous vehicles. Each system can have defined interfaces for data transmission, ensuring data real-time performance and integrity. A brief introduction to each system follows:

[0198] a. Environmental sensing system

[0199] Environmental perception is the most fundamental part of autonomous vehicles. Whether making driving behavior decisions or global path planning, it all relies on environmental perception. Based on real-time perception of the road traffic environment, corresponding judgments, decisions, and plans are made to enable intelligent driving. The environmental perception system mainly uses various sensors to acquire relevant environmental information, thereby constructing an environmental model and representing knowledge of the traffic scene. The sensors used include cameras, single-line radar (SICK), four-line radar (IBEO), and 3D LiDAR (HDL-64E). Cameras are mainly responsible for traffic light detection, lane line detection, road sign detection, and vehicle recognition. LiDAR sensors are mainly responsible for the detection, recognition, and tracking of dynamic / static obstacles, as well as their own precise positioning. For example, 3D LiDAR typically collects external environmental information at a frequency of 10 FPS and returns a laser point cloud at each moment. Finally, the acquired real-time laser point cloud is sent to the autonomous decision-making system for further decision-making and planning.

[0200] b. Autonomous decision-making system

[0201] The autonomous decision-making system is a key component of autonomous vehicles. This system mainly consists of two core subsystems: behavior decision-making and motion planning. The behavior decision-making subsystem primarily obtains the globally optimal driving route by running a global planning layer to clarify the specific driving task. It then uses real-time road information from the environmental perception system (i.e.,...) Figure 20 Specifically, in this embodiment, the autonomous decision-making system uses real-time laser point cloud and image data sent by the environmental perception system to output information such as the position and orientation of objects around the vehicle through a trained 3D object detection model deployed on the autonomous vehicle. Finally, based on road traffic rules and driving experience, the system determines a reasonable driving behavior based on the vehicle's positioning and the position and orientation of surrounding objects, and sends the driving behavior instruction to the motion planning subsystem. The motion planning subsystem then plans a feasible driving trajectory based on safety, stability, and other indicators, according to the received driving behavior instruction and the current environmental perception information, and sends it to the control system.

[0202] c. Control System

[0203] The control system is specifically divided into two parts: the control subsystem and the execution subsystem. The control subsystem is used to convert the feasible driving trajectory generated by the autonomous decision-making system into specific execution instructions for each execution module and then transmit them to the execution subsystem. After receiving the execution instructions from the control subsystem, the execution subsystem sends them to each controlled object to perform reasonable control of the vehicle's steering, braking, throttle, gear shift, etc., so that the vehicle can drive automatically to complete the corresponding driving operation.

[0204] It should be noted that, Figure 20 The overall architecture of the autonomous vehicle shown is for illustrative purposes only. In actual applications, it may contain more or fewer systems / subsystems or modules, and each system / subsystem or module may include multiple components. No specific limitations are made here.

[0205] based on Figure 20 The autonomous vehicle shown performs real-time target detection on acquired images and laser point clouds to locate pedestrians, obstacles, and other vehicles, and then executes corresponding driving strategies. In practical applications, autonomous vehicles can understand the surrounding traffic conditions through video information / images acquired by cameras or laser point clouds acquired by LiDAR. Based on a deployed and trained 3D target detection model, it identifies target objects in front of the vehicle and draws a safe and reliable route for navigation. Compared to 2D target detection, 3D target detection can provide the position, size, and orientation of objects in the three-dimensional environment, making it a crucial part of the environmental perception module. Accurate detection of objects in the environment is essential for safe autonomous driving.

[0206] (2) Image processing scenarios (e.g., mobile phones)

[0207] With the rapid growth of the national economy, the rapid progress of society, and the continuous enhancement of national strength, people have higher demands for daily life and entertainment. The camera functions of terminal devices (such as mobile phones) are becoming increasingly sophisticated, and 2D object detection is already very mature. Research on real-time 3D object detection for the movement of everyday objects is also increasing. For example... Figure 21 As shown, the 3D target detection model provided in this application embodiment can be deployed on terminals such as mobile phones. The sparse 3D laser point cloud obtained by the depth camera of the mobile phone is fused with the image captured by the camera to perform multimodal feature fusion, thereby realizing the 3D target detection method described in this application embodiment, which can greatly improve the user experience and increase the fun of shooting.

[0208] (3) Intelligent robot interaction scenarios

[0209] Intelligent robots will eventually enter every household. For them to become human assistants, they must first be able to perceive their surroundings and perform corresponding actions. In practical applications, intelligent robots can collect images and laser point clouds of their environment, and then use a trained 3D target detection model deployed on them to perform 3D target detection on the collected images and laser point clouds to locate a specific target. For example, please refer to... Figure 22In a room, a male homeowner is doing housework and needs a bowl, so he asks his robot butler to bring it to him. Upon receiving the instruction, the robot butler first detects the surrounding environment, acquiring images and laser point clouds. Based on these images and point clouds, it uses the 3D object detection method described in this application to locate the bowl before proceeding with subsequent actions. Specifically, the process of detecting the surrounding environment and locating the bowl involves acquiring images and laser point clouds of the surrounding environment and using a deployed 3D object detection model to pinpoint the bowl's location.

[0210] (4) Mobile robot navigation

[0211] With the advancement of science and technology, mobile robots have been widely used in many industries. These are intelligent devices that can autonomously control their movement and automatically perform tasks. They can not only accept user commands and run pre-programmed routines, but also move autonomously without human intervention. Today, laser-guided robots are the most common in homes, shopping malls, restaurants, and other indoor settings, covering all scenarios. In restaurants, food delivery robots can move freely without bumping into customers, even when crowded. In shopping malls, if a robot gets lost, clicking on its large screen will guide it to its destination. At home, robot vacuums know which areas have been cleaned and which haven't.

[0212] like Figure 23 As shown, the mobile robot is equipped with the trained 3D target detection model described in this embodiment. Similar to autonomous vehicles, mobile robot navigation first requires accurate perception of objects in the three-dimensional environment. Robots are generally equipped with LiDAR and cameras. Using the 3D target detection method provided in this embodiment can further improve the performance of 3D target detection, ensuring correct and safe navigation for the robot.

[0213] (5) 3D point cloud data annotation

[0214] Compared to 2D data, 3D data has an additional dimension, and 3D point cloud data is relatively sparse, making annotation more difficult. Currently, the training of deep networks is inseparable from data. The 3D object detection model provided in this application can assist in the annotation of 3D point cloud data, reducing labor costs and improving annotation efficiency while ensuring annotation quality.

[0215] It should be noted that the trained object detection model described in this application can be applied not only to the above-mentioned... Figures 20 to 23The application scenarios described can also be applied to various sub-fields of artificial intelligence, such as image processing, computer vision, semantic analysis, etc. As long as neural networks can be used, the trained 3D object detection model provided in this application embodiment can be applied. No further examples will be given here.

[0216] Based on the corresponding embodiments described above, in order to better implement the solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 24 , Figure 24 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 2400 includes: an acquisition module 2401, a first operation module 2402, a second operation module 2403, a cascade module 2404, and a detection module 2405. The acquisition module 2401 is used to obtain two-dimensional information corresponding to the laser point cloud based on the acquired image and laser point cloud. The first operation module 2402 is used to perform a convolution operation on the laser point cloud using a three-dimensional first sparse convolution module to obtain a first feature, and then perform a convolution operation on the two-dimensional information using a three-dimensional second sparse convolution module to obtain a second feature. The second operation module 2403 is used to cascade the first feature and the second feature and then input them into a two-dimensional convolution module for convolution to obtain a third feature. The cascade module 2404 is used to cascade the first feature, the second feature, and the third feature to obtain a fourth feature. The detection module 2405 is used to input the fourth feature into a classification regression module to obtain a 3D target bounding box and the classification category to which the target object within the 3D target bounding box belongs.

[0217] In the above embodiments of this application, the execution device 2400 first obtains the 2D information corresponding to each laser point in the laser point cloud (i.e., 3D information) through the acquisition module 2401. Then, through the first operation module 2402, the 3D information and the 2D information are respectively input into two three-dimensional sparse convolution modules to obtain a first feature and a second feature. Then, through the second operation module 2403, the first feature and the second feature are concatenated and input into a two-dimensional convolution module to obtain a third feature (i.e., fused feature). The third feature is then concatenated with the first feature and the second feature to obtain a fourth feature (i.e., combined feature). Finally, the fourth feature is used to achieve 3D target detection through a classification and regression module. This application is used to obtain the 2D information corresponding to each laser point in the laser point cloud based on the laser point cloud and the 2D image, and to perform feature fusion of the laser point cloud and the 2D information at the feature level. This improves the performance of 3D target detection while retaining the original features of the 3D laser point cloud, making the 3D target detection robust in complex scenarios such as nighttime, rainy or foggy days when the camera is malfunctioning.

[0218] In one possible design, the acquisition module 2401 is specifically used to: perform semantic segmentation on the acquired image using a semantic segmentation model to obtain a semantic segmentation score for each pixel in the image, wherein the semantic segmentation score represents the probability that each pixel belongs to its respective classification category, and the semantic segmentation scores of each pixel in the image constitute a semantic segmentation map; then, project the laser point cloud onto the image to obtain the target RGB information corresponding to each laser point in the laser point cloud, and project the laser point cloud onto the semantic segmentation map to obtain the target semantic segmentation score corresponding to each laser point, wherein the target RGB information and the target semantic segmentation score constitute the two-dimensional information.

[0219] In the above embodiments of this application, it is described how to obtain 2D information from an image and a laser point cloud. That is, the image is first semantically segmented to obtain a semantic segmentation map, and then the laser point cloud is projected onto the image and the semantic segmentation map respectively to obtain 2D information, which is feasible.

[0220] In one possible design, the first operation module 2402 is specifically used to: perform voxelization on the laser point cloud to obtain a voxelized laser point cloud; and then perform a convolution operation on the voxelized laser point cloud through a three-dimensional first sparse convolution module.

[0221] In the above embodiments of this application, the input data of the three-dimensional sparse convolution module needs to be voxelized first, and the voxelized input data is then input into the sparse convolution module to perform convolution operation. Therefore, in this embodiment, the laser point cloud is first voxelized, and the voxelized laser point cloud is then processed by the first sparse convolution module to perform convolution operation, which is flexible.

[0222] In one possible design, the first operation module 2402 is further configured to: perform voxelization on the two-dimensional information to obtain voxelized two-dimensional information; and then perform convolution operation on the voxelized two-dimensional information through a three-dimensional second sparse convolution module.

[0223] In the above embodiments of this application, not only is voxelization of the laser point cloud required, but also voxelization of the two-dimensional information is required, which provides flexibility. Furthermore, the embodiments of this application perform voxelization of both the laser point cloud and the two-dimensional information, which is easy to operate.

[0224] In one possible design, the first operation module 2402 is further configured to: concatenate the laser point cloud and the two-dimensional information to obtain multimodal information; perform voxelization processing on the multimodal information to obtain voxelized multimodal information, wherein the voxelized multimodal information includes the voxelized laser point cloud and the voxelized two-dimensional information; perform a convolution operation on the voxelized laser point cloud through a three-dimensional first sparse convolution module to obtain a first feature, and perform a convolution operation on the voxelized two-dimensional information through a three-dimensional second sparse convolution module to obtain a second feature.

[0225] In the above embodiments of this application, another voxelization method is described, namely, first concatenating the laser point cloud with the 2D information to obtain multimodal information. In this way, voxelizing the multimodal information once is equivalent to voxelizing the laser point cloud and the 2D information simultaneously, thereby saving one voxelization operation.

[0226] In one possible design, the second operation module 2403 is specifically used to: concatenate the first feature and the second feature to obtain a concatenated feature; perform devoxification processing on the concatenated feature to obtain a devoxified concatenated feature; and input the devoxified concatenated feature into a two-dimensional convolution module for convolution operation.

[0227] In the above embodiments of this application, the input data of the two-dimensional convolution module needs to be devoxed (i.e., denser representation). Devoxing is the opposite of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution operation, which provides flexibility.

[0228] It should be noted that the information interaction and execution process between the modules / units in the execution device 2400 can be applied to various application scenarios in the corresponding method embodiments described above in this application. For details, please refer to the description in the aforementioned method embodiments of this application, which will not be repeated here.

[0229] The following describes another related device provided in the embodiments of this application. Please refer to [link / reference needed]. Figure 25 , Figure 25This is a schematic diagram of a training device provided in an embodiment of this application. The training device 2500 includes: a construction module 2501, a first operation module 2502, a second operation module 2503, a cascade module 2504, a prediction module 2505, and a training module 2506. The construction module 2501 is used to construct a first training set based on an initial training set. The initial training samples in the initial training set include an initial laser point cloud and initial two-dimensional information corresponding to the initial laser point cloud. The first training sample in the first training set includes a first laser point cloud and first two-dimensional information corresponding to the first laser point cloud. The first training sample is a training sample obtained by extracting a target object from each initial training sample in the initial training set and randomly copying the target object into each initial training sample. The initial training sample is any one training sample in the initial training set, and the first training sample is any one training sample in the first training set. The first operation module 2502 is used to perform a three-dimensional first sparse convolution on the initial training sample. The first laser point cloud undergoes a convolution operation to obtain a first feature, and then a second feature is obtained by performing a convolution operation on the first two-dimensional information through a three-dimensional second sparse convolution module. A second operation module 2503 concatenates the first and second features and inputs them into a two-dimensional convolution module for convolution to obtain a third feature. A concatenation module 2504 concatenates the first, second, and third features to obtain a fourth feature. A prediction module 2505 inputs the fourth feature into a classification regression module to obtain a 3D predicted bounding box and the predicted classification category of the target object within the 3D predicted bounding box. A training module 2506 iteratively trains the model using a target loss function based on the 3D ground truth bounding box, the ground truth classification category of the target object within the 3D ground truth bounding box, the 3D predicted bounding box, and the predicted classification category of the target object within the 3D predicted bounding box. The model includes the first sparse convolution module, the second sparse convolution module, the convolution module, and the classification regression module.

[0230] In this embodiment, it is specifically described how to use the various modules in the training device 2500 to iteratively train the various modules in the 3D object detection model, and the training process is easy to implement.

[0231] In one possible design, the initial two-dimensional information includes the initial RGB information corresponding to each initial laser point in the initial laser point cloud obtained by projecting the initial laser point cloud onto an initial image, and the initial semantic segmentation score corresponding to each initial laser point obtained by projecting the initial laser point cloud onto an initial semantic segmentation map. The initial semantic segmentation score is the probability that each pixel in the initial image belongs to its respective classification category, obtained by semantic segmentation of the initial image using a semantic segmentation model. The initial semantic segmentation scores of each pixel in the initial image constitute the initial semantic segmentation map. The first two-dimensional information includes the first RGB information corresponding to each first laser point in the first laser point cloud obtained by projecting the first laser point cloud onto a first image, and the first semantic segmentation score corresponding to each first laser point obtained by projecting the first laser point cloud onto a first semantic segmentation map. The first semantic segmentation score is the probability that each pixel in the first image belongs to its respective classification category, obtained by semantic segmentation of the first image using the semantic segmentation model. The first semantic segmentation score of each pixel in the first image constitutes the first semantic segmentation map.

[0232] In the above embodiments of this application, the nature and origin of two-dimensional information are specifically explained, demonstrating its feasibility.

[0233] In one possible design, the first operation module 2502 is specifically used to: perform voxelization on the first laser point cloud to obtain a voxelized first laser point cloud; and perform a convolution operation on the voxelized first laser point cloud through the first sparse convolution module.

[0234] In the above embodiments of this application, the input data of the three-dimensional sparse convolution module needs to be voxelized first, and the voxelized input data is then input into the sparse convolution module to perform convolution operation. Therefore, the embodiments of this application first voxelize the first laser point cloud, and then perform convolution operation through the first sparse convolution module to obtain the voxelized first laser point cloud, which has flexibility.

[0235] In one possible design, the first operation module 2502 is further configured to: perform voxelization on the first two-dimensional information to obtain voxelized first two-dimensional information; and perform convolution operation on the voxelized first two-dimensional information through the second sparse convolution module.

[0236] In the above embodiments of this application, not only is it necessary to perform voxelization processing on the first laser point cloud, but also on the first two-dimensional information, which provides flexibility. Furthermore, the embodiments of this application perform voxelization processing on the first laser point cloud and the first two-dimensional information respectively, which is easy to operate.

[0237] In one possible design, the first operation module 2502 is further configured to: concatenate the first laser point cloud and the first two-dimensional information to obtain multimodal information; perform voxelization processing on the multimodal information to obtain voxelized multimodal information, wherein the voxelized multimodal information includes the voxelized first laser point cloud and the voxelized first two-dimensional information; perform a convolution operation on the voxelized first laser point cloud through the first sparse convolution module to obtain a first feature, and perform a convolution operation on the voxelized first two-dimensional information through the second sparse convolution module to obtain a second feature.

[0238] In the above embodiments of this application, another voxelization method is described, namely, first concatenating the first laser point cloud with the first two-dimensional information to obtain multimodal information. In this way, voxelizing the multimodal information once is equivalent to voxelizing the first laser point cloud and the first two-dimensional information at the same time, thereby saving one voxelization operation.

[0239] In one possible design, the second operation module 2503 is specifically used to: concatenate the first feature and the second feature to obtain a concatenated feature; perform devoxification processing on the concatenated feature to obtain a devoxified concatenated feature; and input the devoxified concatenated feature into the convolution module for convolution operation.

[0240] In the above embodiments of this application, the input data of the two-dimensional convolution module needs to be devoxed (i.e., denser representation). Devoxing is the opposite of voxelization. The third feature is devoxed before being input into the two-dimensional convolution module to perform convolution operation, which provides flexibility.

[0241] Training all training samples in the entire training set once is called one training epoch (i.e., epoch = 1). One epoch is equivalent to training once using all training samples in the training set. The number of epochs indicates the number of training epochs on the training set. Therefore, in one possible design, the training module 2506 is also used to: repeat the above steps performed by the construction module 2501, the first operation module 2502, the second operation module 2503, the cascade module 2504, the prediction module 2505, and the training module 2506 until the number of training epochs on the first training set reaches a first preset number of epochs; and take the initial training set as a new first training set, repeating the above steps performed by the construction module 2501, the first operation module 2502, the second operation module 2503, the cascade module 2504, the prediction module 2505, and the training module 2506 until the number of training epochs on the new first training set reaches a second preset number of epochs.

[0242] In the above embodiments of this application, each epoch requires the construction of a first training set based on the initial training set until the number of the first preset epochs (i.e., the first preset rounds, for example, 15 times) is reached. Then, the initial training set is directly trained until the number of the second preset epochs (i.e., the second prediction rounds, for example, 5 times) is reached. The purpose of this operation is to allow the model to see more real samples, avoid performance loss caused by the large difference between the sample space generated by data augmentation and the original input sample space, and further improve the efficiency of data augmentation.

[0243] It should also be noted that the information interaction and execution process between the modules / units in the training device 2500 can be applied to various application scenarios in the corresponding method embodiments described above in this application. For details, please refer to the description in the method embodiments described above in this application, which will not be repeated here.

[0244] The following describes another execution device provided in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 26 , Figure 26 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 2600 can specifically be an end-side device, an edge device (e.g., a virtual reality (VR) device, a mobile phone, a tablet, a laptop, a smart wearable device, etc.), or a wheeled mobile device (e.g., an autonomous vehicle, a driver-assisted vehicle, an intelligent robot, etc.), and is not limited thereto. The execution device 2600 may be equipped with... Figure 24 The modules described in the corresponding embodiments are used to implement Figure 24 The corresponding embodiment executes the function of device 2400. Specifically, execution device 2600 includes: receiver 2601, transmitter 2602, processor 2603, and memory 2604 (wherein the number of processors 2603 in execution device 2600 can be one or more). Figure 25 (Taking a processor as an example), the processor 2603 may include an application processor 26031 and a communication processor 26032. In some embodiments of this application, the receiver 2601, transmitter 2602, processor 2603, and memory 2604 may be connected via a bus or other means.

[0245] Memory 2604 may include read-only memory and random access memory, and provides instructions and data to processor 2603. A portion of memory 2604 may also include non-volatile random access memory (NVRAM). Memory 2604 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0246] Processor 2603 controls the operation of execution device 2600. In specific applications, the various components of execution device 2600 are coupled together through a bus system, which may include not only a data bus but also a power bus, control bus, and status signal bus, etc. However, for clarity, all buses are referred to as the bus system in the diagram.

[0247] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 2603. Processor 2603 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of processor 2603 or by instructions in software form. Processor 2603 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 2603 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 2604. Processor 2603 reads the information from memory 2604 and, in conjunction with its hardware, completes the steps of the above method.

[0248] Receiver 2601 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of execution device 2600. Transmitter 2602 can be used to output digital or character information through the first interface; transmitter 2602 can also be used to send instructions to disk group through the first interface to modify data in disk group; transmitter 2602 may also include display devices such as display screen.

[0249] In one embodiment of this application, the processor 2603 is configured to perform the above-described... Figure 16The corresponding target detection method involves several steps. First, a sensor device (e.g., camera, LiDAR) deployed on the execution device acquires an image and a laser point cloud at a specific location and time. Then, based on the acquired image and laser point cloud, two-dimensional information corresponding to the laser point cloud is obtained. After acquiring the laser point cloud and its corresponding two-dimensional information, convolution operations are performed on the laser point cloud and the two-dimensional information using a first three-dimensional sparse convolution module and a second three-dimensional sparse convolution module, respectively, to obtain a first feature corresponding to the laser point cloud and a second feature corresponding to the two-dimensional information. After obtaining the first feature corresponding to the laser point cloud and the second feature corresponding to the two-dimensional information, the first and second features are concatenated, and the concatenated first and second features are input into a two-dimensional convolution module for convolution operations to obtain a third feature. This third feature, because it integrates the first and second features, can also be called a fused feature. After obtaining the third feature, the first, second, and third features are concatenated to obtain the fourth feature, which can also be called the combined feature. This fourth feature can then be input into the classification and regression module, such as the classification and regression head, to output the 3D bounding box and the classification category of the target object within it. The target object can also be referred to as the target object.

[0250] In another embodiment of this application, the processor 2603 is also configured to perform the above-described actions. Figures 20 to 23 For details on the various applications of the trained target detection model in different application scenarios in the corresponding embodiments, please refer to the above embodiments, which will not be repeated here.

[0251] This application also provides a training device; please refer to [link / reference]. Figure 27 , Figure 27 This is a schematic diagram of a training device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of this application. The training device 2700 can be deployed with... Figure 25 The module of the training device 2500 described in the corresponding embodiment is used to implement Figure 25In accordance with the functionality of the training device in the corresponding embodiment, specifically, the training device 2700 is implemented by one or more servers. The training device 2700 can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) 2722 (e.g., one or more) and a memory 2732, and one or more storage media 2730 (e.g., one or more mass storage devices) for storing application programs 2742 or data 2744. The memory 2732 and storage media 2730 can be temporary or persistent storage. The program stored in the storage media 2730 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the training device. Furthermore, the CPU 2722 may be configured to communicate with the storage media 2730 and execute the series of instruction operations in the storage media 2730 on the training device 2700.

[0252] The training device 2700 may also include one or more power supplies 2726, one or more wired or wireless network interfaces 2750, one or more input / output interfaces 2758, and / or one or more operating systems 2741, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0253] In the embodiments of this application, the above Figure 10 In the corresponding embodiments, the steps performed by the training device can be based on this. Figure 27 The structure shown is implemented in detail here, but will not be elaborated upon further.

[0254] This application also provides a computer program product that, when run on a computer, causes the computer to perform the steps executed by the detection device in the method described in the foregoing embodiments, or causes the computer to perform the steps executed by the training device in the method described in the foregoing embodiments.

[0255] This application also provides a computer-readable storage medium storing a program for performing signal processing. When the program is run on a computer, it causes the computer to perform the steps executed by the detection device in the method described in the foregoing embodiments, or causes the computer to perform the steps executed by the training device in the method described in the foregoing embodiments.

[0256] The detection and training devices provided in this application embodiment can specifically be chips. A chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the training device to execute the neural network optimization method described in the above-described embodiments. Alternatively, the chip within the detection device can execute the image processing method or audio processing method described in the above-described embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0257] For details, please refer to Figure 28 , Figure 28 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 200. The NPU 200 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0258] In some implementations, the arithmetic circuit 2003 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 2003 is a two-dimensional pulsating array. The arithmetic circuit 2003 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2003 is a general-purpose matrix processor.

[0259] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2008.

[0260] Unified memory 2006 is used to store input and output data. Weight data is directly transferred to weight memory 2002 via Direct Memory Access Controller (DMAC) 2005. Input data is also transferred to unified memory 2006 via DMAC.

[0261] BIU stands for Bus Interface Unit 2010, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2009.

[0262] The Bus Interface Unit (BIU) 2010 is used by the instruction fetch memory 2009 to fetch instructions from external memory, and also by the memory access controller 2005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0263] DMAC is mainly used to move input data from external memory DDR to unified memory 2006, or weight data to weight memory 2002, or input data to input memory 2001.

[0264] The vector computation unit 2007 includes multiple processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computations in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0265] In some implementations, the vector computation unit 2007 can store the processed output vector in the unified memory 2006. For example, the vector computation unit 2007 can apply linear and / or nonlinear functions to the output of the computation circuit 2003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 2007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 2003, for example, for use in subsequent layers of the neural network.

[0266] The instruction fetch buffer 2009 connected to the controller 2004 is used to store the instructions used by the controller 2004;

[0267] Unified Memory 2006, Input Memory 2001, Weight Memory 2002, and Instruction Fetch Memory 2009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0268] The operations of each layer in the first neural network, the operations of each layer in the second neural network, and the joint iterative training process of the two neural networks can be executed by the operation circuit 2003 or the vector calculation unit 2007.

[0269] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0270] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0271] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0272] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0273] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (digital video disc, DVD), or semiconductor media (e.g., solid-state disk (SSD)).

Claims

1. A method for 3D target detection, characterized in that, include: Two-dimensional information corresponding to the laser point cloud is obtained based on the acquired image and laser point cloud. The two-dimensional information includes: the target RGB information corresponding to each laser point when the laser point cloud is projected onto the image, and the target semantic segmentation score corresponding to each laser point when the laser point cloud is projected onto the semantic segmentation map. The semantic segmentation score of each pixel in the image constitutes the semantic segmentation map. The semantic segmentation score is used to represent the probability that each pixel belongs to its respective classification category. The laser point cloud is convolved using a first sparse convolution module in three dimensions to obtain a first feature, and the two-dimensional information is convolved using a second sparse convolution module in three dimensions to obtain a second feature. The first feature and the second feature are concatenated and then input into a two-dimensional convolution module for convolution operation to obtain the third feature; The first feature, the second feature, and the third feature are concatenated to obtain the fourth feature; The fourth feature is input into the classification and regression module to obtain the 3D target box and the classification category to which the target object within the 3D target box belongs.

2. The method according to claim 1, characterized in that, The step of obtaining two-dimensional information corresponding to the laser point cloud based on the acquired image and laser point cloud includes: The acquired image is semantically segmented using a semantic segmentation model to obtain the semantic segmentation score for each pixel in the image. The laser point cloud is projected onto the image to obtain the target RGB information corresponding to each laser point in the laser point cloud. The laser point cloud is then projected onto the semantic segmentation map to obtain the target semantic segmentation score corresponding to each laser point. The target RGB information and the target semantic segmentation score constitute the two-dimensional information.

3. The method according to any one of claims 1-2, characterized in that, The convolution operation performed on the laser point cloud using a three-dimensional first sparse convolution module includes: The laser point cloud is voxelized to obtain a voxelized laser point cloud. The voxelized laser point cloud is convolved using a first sparse convolution module in three dimensions.

4. The method according to any one of claims 1-2, characterized in that, The step of performing a convolution operation on the two-dimensional information using a three-dimensional second sparse convolution module includes: The two-dimensional information is voxelized to obtain voxelized two-dimensional information; The voxelized two-dimensional information is convolved using a second sparse convolution module in three dimensions.

5. The method according to any one of claims 1-2, characterized in that, The process of performing a convolution operation on the laser point cloud using a three-dimensional first sparse convolution module to obtain a first feature, and performing a convolution operation on the two-dimensional information using a three-dimensional second sparse convolution module to obtain a second feature, includes: The laser point cloud and the two-dimensional information are concatenated to obtain multimodal information; The multimodal information is voxelized to obtain voxelized multimodal information, which includes voxelized laser point cloud and voxelized two-dimensional information. The voxelized laser point cloud is convolved by a first sparse convolution module in three dimensions to obtain a first feature, and the voxelized two-dimensional information is convolved by a second sparse convolution module in three dimensions to obtain a second feature.

6. The method according to claim 4, characterized in that, The step of concatenating the first feature and the second feature and then inputting them into a two-dimensional convolution module for convolution operation includes: The first feature and the second feature are concatenated to obtain a concatenated feature; The cascaded features are devoxed to obtain devoxed cascaded features; The devoxated cascaded features are input into a two-dimensional convolution module for convolution operation.

7. A training method for a 3D object detection model, characterized in that, The model includes a three-dimensional first sparse convolution module, a three-dimensional second sparse convolution module, a two-dimensional convolution module, and a classification and regression module. The method includes: A first training set is constructed based on an initial training set. The initial training samples in the initial training set include an initial laser point cloud and initial two-dimensional information corresponding to the initial laser point cloud. The first training sample in the first training set includes a first laser point cloud and first two-dimensional information corresponding to the first laser point cloud. The first training sample is a training sample obtained by extracting target objects from each initial training sample in the initial training set and randomly copying the target objects into each initial training sample. The initial training sample can be any training sample in the initial training set. The initial two-dimensional information includes the initial RGB information corresponding to each initial laser point in the initial laser point cloud obtained by projecting the initial laser point cloud onto an initial image, and the initial RGB information obtained by projecting the initial laser point cloud onto an initial semantic segmentation map. The initial semantic segmentation score corresponding to each initial laser point is described. The initial semantic segmentation score is the probability that each pixel in the initial image belongs to its respective classification category, obtained by semantic segmentation of the initial image through a semantic segmentation model. The initial semantic segmentation scores of each pixel in the initial image constitute the initial semantic segmentation map. The first two-dimensional information includes the first RGB information corresponding to each first laser point in the first laser point cloud obtained by projecting the first laser point cloud onto the first image, and the first semantic segmentation score corresponding to each first laser point obtained by projecting the first laser point cloud onto the first semantic segmentation map. The first semantic segmentation score is the probability that each pixel in the first image belongs to its respective classification category, obtained by semantic segmentation of the first image through the semantic segmentation model. The first semantic segmentation scores of each pixel in the first image constitute the first semantic segmentation map. The first laser point cloud is convolved by the first sparse convolution module to obtain the first feature, and the first two-dimensional information is convolved by the second sparse convolution module to obtain the second feature. The first feature and the second feature are concatenated and then input into the convolution module for convolution operation to obtain the third feature; The first feature, the second feature, and the third feature are concatenated to obtain the fourth feature; The fourth feature is input into the classification and regression module to obtain the 3D predicted target box and the predicted classification category of the target object within the 3D predicted target box; The model is iteratively trained using a target loss function based on the 3D ground truth bounding box, the ground truth classification category of the target object within the 3D ground truth bounding box, the 3D predicted bounding box, and the predicted classification category of the target object within the 3D predicted bounding box.

8. The method according to claim 7, characterized in that, The step of performing a convolution operation on the first laser point cloud through the first sparse convolution module includes: The first laser point cloud is voxelized to obtain the voxelized first laser point cloud. The first sparse convolution module performs a convolution operation on the voxelized first laser point cloud.

9. The method according to any one of claims 7-8, characterized in that, The step of performing a convolution operation on the first two-dimensional information through the second sparse convolution module includes: The first two-dimensional information is voxelized to obtain the voxelized first two-dimensional information. The second sparse convolution module performs a convolution operation on the voxelized first two-dimensional information.

10. The method according to claim 7, characterized in that, The step of performing a convolution operation on the first laser point cloud using the first sparse convolution module to obtain a first feature, and performing a convolution operation on the first two-dimensional information using the second sparse convolution module to obtain a second feature includes: The first laser point cloud and the first two-dimensional information are concatenated to obtain multimodal information; The multimodal information is voxelized to obtain voxelized multimodal information, which includes a voxelized first laser point cloud and voxelized first two-dimensional information. The first feature is obtained by performing a convolution operation on the voxelized first laser point cloud using the first sparse convolution module, and the second feature is obtained by performing a convolution operation on the voxelized first two-dimensional information using the second sparse convolution module.

11. The method according to claim 9, characterized in that, The step of concatenating the first feature and the second feature and then inputting them into the convolution module for convolution operation includes: The first feature and the second feature are concatenated to obtain a concatenated feature; The cascaded features are devoxed to obtain devoxed cascaded features; The devoxated cascaded features are input into the convolution module for convolution operation.

12. The method according to any one of claims 7-8, characterized in that, The method further includes: Repeat the above steps until the first training set reaches the first preset number of training epochs; The initial training set is used as the new first training set, and the above steps are repeated until the training rounds of the new first training set reach the second preset round.

13. An execution device, characterized in that, The device includes: The acquisition module is used to obtain two-dimensional information corresponding to the laser point cloud based on the acquired image and laser point cloud. The two-dimensional information includes: the target RGB information corresponding to each laser point when the laser point cloud is projected onto the image, and the target semantic segmentation score corresponding to each laser point when the laser point cloud is projected onto the semantic segmentation map. The semantic segmentation score of each pixel in the image constitutes the semantic segmentation map, and the semantic segmentation score is used to represent the probability that each pixel belongs to its respective classification category. The first operation module is used to perform a convolution operation on the laser point cloud through a three-dimensional first sparse convolution module to obtain a first feature, and to perform a convolution operation on the two-dimensional information through a three-dimensional second sparse convolution module to obtain a second feature; The second operation module is used to concatenate the first feature and the second feature and then input them into a two-dimensional convolution module for convolution operation to obtain the third feature; A cascading module is used to cascade the first feature, the second feature, and the third feature to obtain a fourth feature; The detection module is used to input the fourth feature into the classification and regression module to obtain the 3D target box and the classification category to which the target object within the 3D target box belongs.

14. The device according to claim 13, characterized in that, The acquisition module is specifically used for: The acquired image is semantically segmented using a semantic segmentation model to obtain the semantic segmentation score for each pixel in the image. The laser point cloud is projected onto the image to obtain the target RGB information corresponding to each laser point in the laser point cloud. The laser point cloud is then projected onto the semantic segmentation map to obtain the target semantic segmentation score corresponding to each laser point. The target RGB information and the target semantic segmentation score constitute the two-dimensional information.

15. The device according to any one of claims 13-14, characterized in that, The first operation module is specifically used for: The laser point cloud is voxelized to obtain a voxelized laser point cloud. The voxelized laser point cloud is convolved using a first sparse convolution module in three dimensions.

16. The device according to any one of claims 13-14, characterized in that, The first operation module is further configured to: The two-dimensional information is voxelized to obtain voxelized two-dimensional information; The voxelized two-dimensional information is convolved using a second sparse convolution module in three dimensions.

17. The device according to any one of claims 13-14, characterized in that, The first operation module is further configured to: The laser point cloud and the two-dimensional information are concatenated to obtain multimodal information; The multimodal information is voxelized to obtain voxelized multimodal information, which includes voxelized laser point cloud and voxelized two-dimensional information. The voxelized laser point cloud is convolved by a first sparse convolution module in three dimensions to obtain a first feature, and the voxelized two-dimensional information is convolved by a second sparse convolution module in three dimensions to obtain a second feature.

18. The device according to claim 16, characterized in that, The second operation module is specifically used for: The first feature and the second feature are concatenated to obtain a concatenated feature; The cascaded features are devoxed to obtain devoxed cascaded features; The devoxated cascaded features are input into a two-dimensional convolution module for convolution operation.

19. A training device, characterized in that, The device includes: The construction module is used to construct a first training set based on an initial training set. The initial training samples in the initial training set include an initial laser point cloud and initial two-dimensional information corresponding to the initial laser point cloud. The first training sample in the first training set includes a first laser point cloud and first two-dimensional information corresponding to the first laser point cloud. The first training sample is a training sample obtained by extracting target objects from each initial training sample in the initial training set and randomly copying the target objects into each initial training sample. The initial training sample can be any training sample in the initial training set. The initial two-dimensional information includes the initial RGB information corresponding to each initial laser point in the initial laser point cloud obtained by projecting the initial laser point cloud onto an initial image, and the initial RGB information obtained by projecting the initial laser point cloud onto an initial semantic segmentation map. The initial semantic segmentation score corresponding to each initial laser point is obtained. The initial semantic segmentation score is the probability that each pixel in the initial image belongs to its respective classification category, obtained by semantic segmentation of the initial image through a semantic segmentation model. The initial semantic segmentation scores of each pixel in the initial image constitute the initial semantic segmentation map. The first two-dimensional information includes the first RGB information corresponding to each first laser point in the first laser point cloud obtained by projecting the first laser point cloud onto the first image, and the first semantic segmentation score corresponding to each first laser point obtained by projecting the first laser point cloud onto the first semantic segmentation map. The first semantic segmentation score is the probability that each pixel in the first image belongs to its respective classification category, obtained by semantic segmentation of the first image through the semantic segmentation model. The first semantic segmentation scores of each pixel in the first image constitute the first semantic segmentation map. The first operation module is used to perform a convolution operation on the first laser point cloud through a three-dimensional first sparse convolution module to obtain a first feature, and to perform a convolution operation on the first two-dimensional information through a three-dimensional second sparse convolution module to obtain a second feature. The second operation module is used to concatenate the first feature and the second feature and then input them into a two-dimensional convolution module for convolution operation to obtain the third feature; A cascading module is used to cascade the first feature, the second feature, and the third feature to obtain a fourth feature; The prediction module is used to input the fourth feature into the classification and regression module to obtain a 3D predicted target box and the predicted classification category to which the target object in the 3D predicted target box belongs; The training module is used to iteratively train the model using a target loss function based on the 3D ground truth bounding box, the ground truth classification category of the target object within the 3D ground truth bounding box, the 3D predicted bounding box, and the predicted classification category of the target object within the 3D predicted bounding box. The model includes a first sparse convolution module, a second sparse convolution module, the convolution module, and the classification and regression module.

20. The device according to claim 19, characterized in that, The first operation module is specifically used for: The first laser point cloud is voxelized to obtain the voxelized first laser point cloud. The first sparse convolution module performs a convolution operation on the voxelized first laser point cloud.

21. The device according to any one of claims 19-20, characterized in that, The first operation module is further configured to: The first two-dimensional information is voxelized to obtain the voxelized first two-dimensional information. The second sparse convolution module performs a convolution operation on the voxelized first two-dimensional information.

22. The device according to claim 19, characterized in that, The first operation module is further configured to: The first laser point cloud and the first two-dimensional information are concatenated to obtain multimodal information; The multimodal information is voxelized to obtain voxelized multimodal information, which includes a voxelized first laser point cloud and voxelized first two-dimensional information. The first feature is obtained by performing a convolution operation on the voxelized first laser point cloud using the first sparse convolution module, and the second feature is obtained by performing a convolution operation on the voxelized first two-dimensional information using the second sparse convolution module.

23. The device according to claim 21, characterized in that, The second operation module is specifically used for: The first feature and the second feature are concatenated to obtain a concatenated feature; The cascaded features are devoxed to obtain devoxed cascaded features; The devoxated cascaded features are input into the convolution module for convolution operation.

24. The device according to any one of claims 19-20, characterized in that, The training module is also used for: Repeat the steps performed by the construction module, the first operation module, the second operation module, the cascade module, the prediction module, and the training module until the training epochs of the first training set reach the first preset epochs. Using the initial training set as the new first training set, repeat the steps performed by the construction module, the first operation module, the second operation module, the cascade module, the prediction module, and the training module until the training rounds of the new first training set reach the second preset round.

25. An execution device comprising a processor and a memory, the processor being coupled to the memory, characterized in that, The memory is used to store programs; The processor is configured to execute a program in the memory, causing the execution device to perform the method as described in any one of claims 1-6.

26. A training device comprising a processor and a memory, the processor being coupled to the memory, characterized in that, The memory is used to store programs; The processor is configured to execute a program in the memory, causing the training device to perform the method as described in any one of claims 7-12.

27. A computer-readable storage medium comprising a program, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1-6, or causes the computer to perform the method as described in any one of claims 7-12.

28. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1-6, or causes the computer to perform the method as described in any one of claims 7-12.

29. A chip system, characterized in that, The chip system includes a processor and a communication interface coupled to the processor. The processor is configured to run computer programs or instructions that cause the method of any one of claims 1-6 to be executed, or cause the method of any one of claims 7-12 to be executed.

Images