A 3D target detection method based on multiple sensors

Abstract

This invention relates to the field of 3D target detection technology, and in particular to a multi-sensor-based 3D target detection method. The specific steps are as follows: Step S1, data construction and preprocessing; Step S2, camera image feature extraction; Step S3, LiDAR-Image Fusion Module: Constructing a LiDAR-Image Fusion module to map the hierarchical semantic features generated from RGB images to LiDAR point cloud features, point-wise image features, and fusing them with LiDAR features to obtain the final enhanced features; Step S4, LiDAR point cloud feature extraction. This invention designs a 3D target detection method based on a LiDAR-Image Fusion module, using the LiDAR-Image Fusion module to refine and map LiDAR feature points point by point onto the image, ultimately obtaining a high-precision 3D detection network that ranks among the top on open-source datasets.

Description

Technical Field

[0001] This invention relates to the field of 3D target detection technology, specifically a 3D target detection method based on multiple sensors. Background Technology

[0002] Current popular 3D object detection methods fuse multi-dimensional images and point cloud information acquired by different types of sensors, such as 2D monocular images, stereo cameras, and radar, achieving good accuracy in 3D object detection. Camera images contain a large number of semantic features such as color and texture, but lack depth information of objects. LiDAR point cloud data is in the form of discrete points, characterized by sparseness, disorder, and uneven distribution. It contains depth information of the target, but suffers from chaotic bounding box distribution when detecting geometrically similar objects. Therefore, current 3D object detection methods fuse sensors such as cameras and LiDAR to obtain more accurate detection results.

[0003] Depending on how sensors are used, current sensor fusion methods are mainly divided into two categories: one is the cascade method that uses different sensors at different stages, and the other is the fusion method that jointly infers multiple sensor inputs. Both types of methods can effectively fuse information from multiple sensors, but they still have some shortcomings. The first type of cascade method does not utilize the complementarity between sensors, and its performance is limited by a single stage. The second type of method first requires converting 3D point cloud data into BEV data through perspective projection and voxelization, a process that results in the loss of a lot of information. Furthermore, the second type of method only provides a rough correspondence between voxel features and semantic features, introducing many detection interference terms.

[0004] In summary, this invention addresses the existing problems by designing a multi-sensor-based 3D target detection method. Summary of the Invention

[0005] The purpose of this invention is to provide a 3D target detection method based on multiple sensors to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A 3D target detection method based on multiple sensors, the specific steps of which are as follows:

[0008] Step S1, Data Construction and Preprocessing: Open source datasets for 3D object detection include KITTIDataset and SUN-RGBD Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above two datasets. After preprocessing the data, extract the features of LiDAR point cloud data and RGB image features.

[0009] Step S2, Camera Image Feature Extraction: RGB image features are extracted using a set of convolutional operations to extract semantic image information. The architecture consists of four lightweight convolutional blocks; each block comprises two 3×3 convolutional layers, followed by a BN layer and a ReLU activation function. A second convolutional layer with a stride of 2 is set in each block to expand the receptive field and save GPU memory. The semantic features extracted by the four lightweight convolutional blocks are fused with LiDAR point cloud features of different scales, enriching the LiDAR point cloud features. Simultaneously, four parallel transposed convolutions with different strides are used to restore the image resolution, ultimately obtaining a feature map of the same size as the original RGB image. The four transposed convolutions of different scales are then concatenated to enhance the LiDAR point cloud features and generate more accurate proposals.

[0010] Step S3, LiDAR-Image Fusion Module: Construct the LiDAR-Image Fusion module to map the hierarchical semantic features generated from the RGB image to the LiDAR point cloud features, point-wise image features, and fuse them with the LiDAR features to obtain the final enhanced features;

[0011] Step S4, LiDAR point cloud feature extraction: LiDAR features are taken as input from the point cloud and 3D proposals are generated. Four pairs of set abstraction modules SA are used, where SA stands for Set Abstraction and feature propagation module FP stands for Feature Propogation. Feature maps of the same size as the original image generated by transposed convolution in step S2 are put into the LiDAR-Image Fusion module and fused with the fourth FP module for feature fusion, which is used for final point segmentation and 3D proposal generation.

[0012] As a preferred embodiment of the present invention, the RGB image dimension in S1 is W*H*3, where H and W are the height and width of the feature map, respectively, 3 is the number of channels, and N*3 is the number of point clouds.

[0013] As a preferred embodiment of the present invention, the four lightweight convolutional layers in S2 have an input of 1280*384, with channels of [3, 64, 128, 256, 512], and a stride of 2 for the 3×3 convolutional blocks; the number of LiDAR point cloud feature points is 16384, and the number of channels for the point features is [96, 256, 512, 1024].

[0014] As a preferred embodiment of the present invention, the LiDAR-Image Fusion module constructed in step S3 includes a grid generator (Grid Generator) and an image collector (ImageSampler). The calculation steps of the LiDAR-Image Fusion module are as follows:

[0015] First, based on the mapping matrix M, LiDAR points are projected onto the image. Taking the LiDAR point cloud and the mapping matrix M as input, the output is the correspondence between the LiDAR point cloud and the camera image at different resolutions. Given a point cloud p(x,y,z), for an image position p′=(x′,y′), the mapping formula is:

[0016] p′=M×p (1)

[0017] The dimension of M is 3*4.

[0018] Secondly, an image acquisition device is used to obtain the semantic features of each point. Taking the sampling position p′ and the image feature map F as input, a point-wise image feature representation V is generated for each sampling position. Since the sampling position may fall between adjacent pixels, bilinear interpolation is used to obtain the image features in continuous coordinates.

[0019] V (p) =K(F (Ν(p′)) (2)

[0020] Where V (p) Let P be the image feature of point p, K be the bilinear interpolation function, and F be the interpolation function. (N(p′)) These are image features of pixels adjacent to the sampling position p′;

[0021] Finally, LiDAR point cloud features F P and image features F I First, the weights are mapped to the same channel using a fully connected layer. Then, another fully connected layer compresses them into a single-channel weight mapping w. Finally, the sigmoid function is used for normalization. The specific formula is as follows:

[0022] w=σ(W*tanh(μF P +vF I (3)

[0023] Where w, μ, and ν are learnable weights, and σ is the sigmoid activation function;

[0024] After obtaining the weights w, the LiDAR point cloud features F are then... P and semantic features F I To perform a Concat connection, use the following formula:

[0025] F LI =F P ||wF I (4).

[0026] As a preferred embodiment of the present invention, in S4, the LiDAR feature is taken as point cloud as input, the number of input feature points is 16384, and the number of channels of the point feature is [96,256,512,1024]. Feature extraction is performed through four pairs of set abstraction modules SA and feature propagation modules FP. Modules SA and FP are feature extraction methods from the PointNet++ algorithm, wherein module SA contains a sampling layer.

[0027] For the sampling layer, assuming the point cloud data consists of N points, N1 points are sampled from it. The input size is set to N*(d+C), where d is the xyz 3D coordinate and C is the attribute feature describing the point, which is usually 0. After passing through the sampling layer, the output is N1*(d+C). For the grouping layer, N1 sub-regions are found centered on the N1 sampled points. Each sub-region contains k points, and each point has a dimension of (d+C). After passing through the sampling layer, the output is N1*k*(d+C).

[0028] The continuous SA module downsamples the original points to obtain fewer feature points, and the hierarchical feature propagation strategy based on distance difference, i.e., the FP module, is used to achieve the segmentation task of the original feature points.

[0029] After extracting LiDAR features through four pairs of SA and FP, the feature map generated by the transposed convolution in step S2, which is the same size as the original image, is put into the LiDAR-Image Fusion module and fused with the fourth FP module for final point segmentation and 3D proposal generation.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] 1. In this invention, a 3D target detection method based on the LiDAR-Image Fusion module is designed. The LiDAR-Image Fusion module is used to refine and map LiDAR feature points onto the image point by point, and finally a high-precision 3D detection network is obtained, which ranks among the top in open source datasets. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the camera image feature extraction process structure of the present invention;

[0033] Figure 2This is a schematic diagram of the LiDAR-Image Fusion module flow structure of the present invention;

[0034] Figure 3 This is a schematic diagram of the SA module structure of the present invention;

[0035] Figure 4 This is a schematic diagram of the overall network framework structure for LiDAR point cloud feature extraction in this invention. Detailed Implementation

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0037] To facilitate understanding of the present invention, a more comprehensive description of the invention will be given below with reference to the accompanying drawings, and several embodiments of the invention will be provided. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0038] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0040] For examples, please refer to Figure 1-4 The present invention provides a technical solution:

[0041] A 3D target detection method based on multiple sensors, the specific steps of which are as follows:

[0042] Step S1, Data Construction and Preprocessing: Open source datasets for 3D object detection include KITTIDataset and SUN-RGBD Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above two datasets. After preprocessing the data, extract the features of LiDAR point cloud data and RGB image features.

[0043] In S1, the RGB image dimension is W*H*3, where H and W are the height and width of the feature map, 3 is the number of channels, and N*3 is the number of point clouds.

[0044] Step S2, Camera Image Feature Extraction: RGB image features are extracted using a set of convolutional operations to extract semantic image information. The architecture consists of four lightweight convolutional blocks; each block comprises two 3×3 convolutional layers, followed by a BN layer and a ReLU activation function. A second convolutional layer with a stride of 2 is set in each block to expand the receptive field and save GPU memory. The semantic features extracted by the four lightweight convolutional blocks are fused with LiDAR point cloud features of different scales, enriching the LiDAR point cloud features. Simultaneously, four parallel transposed convolutions with different strides are used to restore the image resolution, ultimately obtaining a feature map of the same size as the original RGB image. These four transposed convolutions of different scales are then concatenated to enhance the LiDAR point cloud features and generate more accurate proposals, such as... Figure 1 ;

[0045] Specifically, in S2, the four lightweight convolutional layers have an input of 1280*384, with channels of [3, 64, 128, 256, 512], and the stride of the 3×3 convolutional block is 2; the number of LiDAR point cloud feature points is 16384, and the number of channels for the point features is [96, 256, 512, 1024].

[0046] Step S3, LiDAR-Image Fusion Module: Construct the LiDAR-Image Fusion module to map the hierarchical semantic features generated from the RGB image to the LiDAR point cloud features, point-wise image features, and fuse them with the LiDAR features to obtain the final enhanced features;

[0047] Specifically, the LiDAR-Image Fusion module constructed in S3 includes a grid generator and an image sampler. The calculation steps of the LiDAR-Image Fusion module are as follows:

[0048] First, based on the mapping matrix M, LiDAR points are projected onto the image. Taking the LiDAR point cloud and the mapping matrix M as input, the output is the correspondence between the LiDAR point cloud and the camera image at different resolutions. Given a point cloud p(x,y,z), for an image position p′=(x′,y′), the mapping formula is:

[0049] p′=M×p (1)

[0050] The dimension of M is 3*4.

[0051] Secondly, an image acquisition device is used to obtain the semantic features of each point. Taking the sampling position p′ and the image feature map F as input, a point-wise image feature representation V is generated for each sampling position. Since the sampling position may fall between adjacent pixels, bilinear interpolation is used to obtain the image features in continuous coordinates.

[0052] V (p) =K(F (Ν(p′)) (2)

[0053] Where V (p) Let P be the image feature of point p, K be the bilinear interpolation function, and F be the interpolation function. (N(p′)) These are image features of pixels adjacent to the sampling position p′;

[0054] Finally, LiDAR point cloud features F P and image features F I First, the weights are mapped to the same channel using a fully connected layer. Then, another fully connected layer compresses them into a single-channel weight mapping w. Finally, the sigmoid function is used for normalization. The specific formula is as follows:

[0055] w=σ(W*tanh(μF P +vF I (3) where w, μ, and v are learnable weights, and σ is the sigmoid activation function;

[0056] After obtaining the weights w, the LiDAR point cloud features F are then... P and semantic features F I To perform a Concat connection, use the following formula:

[0057] F LI =F P ||wF I (4).

[0058] The flowchart of the LiDAR-Image Fusion module is as follows: Figure 2 As shown:

[0059] Step S4, LiDAR point cloud feature extraction: LiDAR features are taken as input from the point cloud and 3D proposals are generated. Four sets of abstraction modules SA are used, where SA stands for Set Abstraction and feature propagation module FP stands for Feature Propogation. Feature maps of the same size as the original image generated by transposed convolution in step S2 are put into the LiDAR-Image Fusion module and fused with the fourth FP module for final point segmentation and 3D proposal generation.

[0060] Specifically, in S4, LiDAR features are taken as input as point clouds, with 16384 input feature points and [96, 256, 512, 1024] channels. Feature extraction is performed through four pairs of set abstraction modules (SA) and feature propagation modules (FP). Modules SA and FP are feature extraction methods derived from the PointNet++ algorithm. Module SA contains a summarizing layer, such as... Figure 3 ;

[0061] For the sampling layer, assuming the point cloud data consists of N points, N1 points are sampled from it. The input size is set to N*(d+C), where d is the xyz 3D coordinate and C is the attribute feature describing the point, which is usually 0. After passing through the sampling layer, the output is N1*(d+C). For the grouping layer, N1 sub-regions are found centered on the N1 sampled points. Each sub-region contains k points, and each point has a dimension of (d+C). After passing through the sampling layer, the output is N1*k*(d+C).

[0062] The continuous SA module downsamples the original points to obtain fewer feature points, and the hierarchical feature propagation strategy based on distance difference, i.e., the FP module, is used to achieve the segmentation task of the original feature points.

[0063] After LiDAR feature extraction using four pairs of SA and FP modules, the feature map of the same size as the original image generated by the transposed convolution in step S2 is fed into the LiDAR-Image Fusion module and fused with the fourth FP module for final point segmentation and 3D proposal generation. The overall network framework is as follows: Figure 4 .

[0064] Specific Implementation Cases

[0065] The specific implementation of the 3D target detection method based on the LiDAR-Image Fusion module is divided into the following parts:

[0066] Step 1: Data Preparation

[0067] The main open-source datasets for 3D object detection include KITTIDataset and SUN-RGBD Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above two datasets. After preprocessing the data, extract the features of LiDAR point cloud data and RGB image features.

[0068] Step 2: Training Phase

[0069] The input image size is 1280*384*3, and the number of image feature channels is [64, 128, 256, 512]. Image features are extracted through four convolutional layers. The input LiDAR point cloud data has 16384 points and the number of point cloud feature channels is [96, 256, 512, 1024]. First, point cloud features are extracted through SA and FP modules. The first convolutional layer is enhanced by the LiDAR-Image Fusion module and fused with the first SA model. Similarly, the subsequent three convolutional layers are fused with the following three SA modules. Second, the SA and FP modules are stacked using a skip-link approach. Specifically, the skip-link feature stacking is: original point cloud data with FP4, SA1 with FP3, SA2 with FP2, SA3 and SA4 with FP1. After feature stacking, the fused feature FP4 is obtained. After four convolutional layers, transposed convolution is used to generate a feature map of the same size as the original image and used as the LiDAR-Image Fusion feature map. The input to the Fusion module is fused with the FP4 feature. For the classification part, the Sigmoid FocalLoss loss function is used, while for the segmentation part, cross-entropy loss and smooth L1 loss functions are used. Training is complete when the loss functions converge, finally yielding the final 3D detection categories and 3D bounding boxes.

[0070] Step 3: Testing Phase

[0071] For the test image, it is fed into the network for inference. The predicted values ​​of the classification and segmentation parts are compared with the ground truth to calculate the mAP, which gives the final accuracy.

[0072] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A 3D target detection method based on multiple sensors, the specific steps of which are as follows: Step S1, Data Construction and Preprocessing: Open source datasets for 3D object detection include KITTIDataset and SUN-RGBD Dataset. When training on your own dataset, you need to prepare the data format to be consistent with the above two datasets. After preprocessing the data, extract LiDAR point cloud data features and RGB image features. Step S2, Camera Image Feature Extraction: RGB image features are extracted using a set of convolutional operations to extract semantic image information. The architecture consists of four lightweight convolutional blocks; each block comprises two 3×3 convolutional layers, followed by a BN layer and a ReLU activation function. A second convolutional layer with a stride of 2 is set in each block to expand the receptive field and save GPU memory. The semantic features extracted by the four lightweight convolutional blocks are fused with LiDAR point cloud features of different scales, enriching the LiDAR point cloud features. Simultaneously, four parallel transposed convolutions with different strides are used to restore the image resolution, ultimately obtaining a feature map of the same size as the original RGB image. The four transposed convolutions of different scales are then concatenated to enhance the LiDAR point cloud features and generate more accurate proposals. Step S3, LiDAR-Image Fusion Module: Construct the LiDAR-Image Fusion module to map the hierarchical semantic features generated from the RGB image to the LiDAR point cloud features, point-wise image features, and fuse them with the LiDAR features to obtain the final enhanced features; Step S4, LiDAR point cloud feature extraction: LiDAR features are taken as input from the point cloud and 3D proposals are generated. Four pairs of set abstraction modules SA are used, where SA stands for Set Abstraction and feature propagation module FP stands for Feature Propogation. Feature maps of the same size as the original image generated by transposed convolution in step S2 are put into the LiDAR-Image Fusion module and fused with the fourth FP module for feature fusion, which is used for final point segmentation and 3D proposal generation.

2. The 3D target detection method based on multiple sensors according to claim 1, characterized in that: In S1, the RGB image dimension is W*H*3, where H and W are the height and width of the feature map, 3 is the number of channels, and N*3 is the number of point clouds.

3. The 3D target detection method based on multiple sensors according to claim 1, characterized in that: The input of the four lightweight convolutional layers in S2 is 1280*384, with channels [3, 64, 128, 256, 512], and the stride of the 3×3 convolutional block is 2; the number of LiDAR point cloud feature points is 16384, and the number of channels of the point features is [96, 256, 512, 1024].

4. The 3D target detection method based on multiple sensors according to claim 1, characterized in that: The LiDAR-Image Fusion module constructed in S3 includes a grid generator and an image sampler. The calculation steps of the LiDAR-Image Fusion module are as follows: First, based on the mapping matrix M, LiDAR points are projected onto the image. Taking the LiDAR point cloud and the mapping matrix M as input, the output is the correspondence between the LiDAR point cloud and the camera image at different resolutions. Given a point cloud p(x,y,z), for an image position p′=(x′,y′), the mapping formula is: p′=M×p (1) Where M has a dimension of 3*4; Secondly, an image acquisition device is used to obtain the semantic features of each point. Taking the sampling position p′ and the image feature map F as input, a point-wise image feature representation V is generated for each sampling position. Since the sampling position may fall between adjacent pixels, bilinear interpolation is used to obtain the image features in continuous coordinates. V (p) =K(F (Ν(p′)) ) (2) Where V (p) Let P be the image feature of point p, K be the bilinear interpolation function, and F be the interpolation function. (N(p′)) These are image features of pixels adjacent to the sampling position p′; Finally, LiDAR point cloud features F P and image features F I First, the weights are mapped to the same channel using a fully connected layer. Then, another fully connected layer compresses them into a single-channel weight mapping w. Finally, the sigmoid function is used for normalization. The specific formula is as follows: w=σ(W*tanh(μF P +vF I )) (3) Where w, μ, and v are learnable weights, and σ is the sigmoid activation function; after obtaining the weights w, the LiDAR point cloud features F are then... P and semantic features F I To perform a Concat connection, use the following formula: F LI =F P ||wF I (4)。 5. The 3D target detection method based on multiple sensors according to claim 1, characterized in that: In S4, the LiDAR features are taken as point cloud input, with 16384 input feature points and [96,256,512,1024] channels for the point features. Feature extraction is performed through four pairs of set abstraction modules SA and feature propagation modules FP. Modules SA and FP are feature extraction methods from the PointNet++ algorithm, where module SA contains a sampling layer. For the sampling layer, assuming the point cloud data consists of N points, N1 points are sampled from it. The input size is set to N*(d+C), where d is the xyz 3D coordinate and C is the attribute feature describing the point. After passing through the sampling layer, the output is N1*(d+C). For the grouping layer, N1 sub-regions are found centered on the N1 sampled points. Each sub-region contains k points, and each point has a dimension of (d+C). After passing through the sampling layer, the output is N1*k*(d+C). The continuous SA module downsamples the original points to obtain fewer feature points, and the hierarchical feature propagation strategy based on distance difference, i.e., the FP module, is used to achieve the segmentation task of the original feature points. After extracting LiDAR features through four pairs of SA and FP, the feature map generated by the transposed convolution in step S2, which is the same size as the original image, is put into the LiDAR-Image Fusion module and fused with the fourth FP module for final point segmentation and 3D proposal generation.

Images