3D target detection method based on local-to-global multi-modal fusion

Abstract

The invention discloses a 3D target detection method based on local-to-global multi-modal fusion, and the method comprises the steps: firstly obtaining a point cloud and a six-view image associated with the point cloud, carrying out the intensive processing of the point cloud through a polar coordinate sampling technology, generating voxel features of different sampling rates through a 3D backbone network, and carrying out the detection of the voxel features. Generating an initial 3D suggestion box by using the regional suggestion network; meanwhile, dense semantic image features are extracted from the six-view image by using a 2D backbone network. In the multi-modal fusion stage, the local fusion module and the global fusion module work cooperatively, the multi-level voxel features, the semantic image features, the initial 3D suggestion box and the position information of the original point cloud are adaptively integrated together, and information interaction between the local fusion features and the global fusion features is realized through the dynamic aggregation module. According to the method, local-to-global cross-modal fusion is utilized, fine granularity and overall features are integrated, and the 3D target detection performance is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of 3D target detection, and in particular to a 3D target detection method based on local to global multimodal fusion. Background Art

[0002] 3D object detection is an essential perception task that plays a key role in safety-critical autonomous driving, which aims to locate and classify objects in three-dimensional space. LiDAR and camera are two widely used sensors. Existing 3D object detection methods have some limitations in multimodal fusion. First, existing multimodal fusion methods mainly focus on the global level, that is, fusing point cloud features and image features in the entire scene. Although this global fusion method can take advantage of the advantages of both LiDAR and camera sensors, it ignores the fine-grained information in the local area, resulting in less than ideal fusion effect, especially when dealing with foreground objects. Second, LiDAR point clouds are usually sparse and cannot provide enough contextual information to distinguish distant areas, which limits the performance of point cloud-based object detection methods, especially in complex environments. In addition, how to accurately align point cloud features with image features during the fusion process is also a key issue. Traditional practices usually use voxel centers to represent locations, but this ignores the distribution of actual points, resulting in low alignment accuracy.

[0003] Therefore, it is necessary to provide a 3D object detection method based on local to global multimodal fusion to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a 3D target detection method based on local to global multimodal fusion, which solves the problem of the lack of fine-grained regional level information in 3D target detection in existing LiDAR-camera fusion methods.

[0005] To achieve the above object, the present invention provides a 3D target detection method based on local to global multimodal fusion, comprising the following steps:

[0006] S1: Acquire point cloud data and its associated six-view images.

[0007] S2: Use polar coordinate sampling technology to densify the point cloud.

[0008] S3: The densified point cloud is passed through the 3D backbone network to generate voxel features with different downsampling rates, and the initial 3D proposal box is generated from these voxel features through the region proposal network.

[0009] S4: Extracting dense semantic image features from six-view images using a 2D backbone network.

[0010] S5: In the multimodal fusion stage, it specifically includes a local fusion module and a global fusion module. These two modules work together to adaptively integrate the multi-level voxel features, semantic image features, initial 3D proposal boxes, and the position information of the original point cloud generated above.

[0011] S6: The information interaction between local and global fusion features is realized through the dynamic aggregation module, and finally the predicted 3D box is obtained through the feedforward neural network.

[0012] Preferably, in step S2, the following steps are specifically included:

[0013] S2.1: Obtain raw point cloud data containing sparse objects and background, as well as the orientation and rotation information of these objects.

[0014] S2.2: Establish a polar coordinate system for each object based on the center of the scene, and set the sensor front direction as the 0 degree reference for subsequent direction and rotation measurements.

[0015] S2.3: Build a database to store dense objects, which will be used to densify other sparse objects. Divide the objects into N×N groups according to their orientation and rotation relative to the LiDAR sensor. The specific grouping formula is as follows:

[0016]

[0017] Among them, α l ′,β l ′∈{1,2,......,N} is the direction and rotation index of object l, and N is the number of divided groups.

[0018] S2.4: Calculate the orientation and rotation to query dense objects of similar density from the database. Add additional points to each object point set in the original point cloud data to obtain enhanced point cloud data.

[0019] Preferably, in step S3, the following steps are specifically included:

[0020] S3.1: Convert the densified point cloud to voxel representation.

[0021] S3.2: The voxelized point cloud is input into the 3D backbone network, which can extract spatial features from different scales. As the network depth increases, a series of voxel features with different downsampling rates will be obtained.

[0022] S3.3: Since voxel features at different downsampling rates carry different information, these multi-level features need to be properly fused.

[0023] S3.4: Based on the above fused multi-level voxel features, a region proposal network is used to predict the location of potential targets. Specifically, RPN evaluates the possibility of each position in the feature map as the center of the object and generates a set of candidate boxes based on this.

[0024] Preferably, in step S5, the global fusion module specifically includes the following steps:

[0025] S5.1.1: For each non-empty voxel, calculate its centroid c based on the spatial coordinates of all the points contained in it. i Position, the specific formula is as follows:

[0026]

[0027] Among them, p i =(x i ,y i ,z i ) is the spatial coordinate, |P(V i )| is the voxel V i The points within.

[0028] S5.1.2: Use the camera intrinsic and extrinsic parameter matrix to project the centroid of each voxel onto the image plane to obtain the reference point p i , the specific formula is as follows:

[0029] p i =M·c i (3)

[0030] Where M is the product of the camera's internal matrix and external matrix.

[0031] S5.1.3: Based on the reference point, the learned offset is applied to the image feature to select surrounding image features. These selected image features are weighted averaged to generate the aggregated image feature.

[0032] S5.1.4: A deformable cross-attention module is used to take each voxel feature as the query and the aggregated image features as the key and value. This mechanism is used to adaptively fuse point cloud features with corresponding image features to generate cross-modal features.

[0033] S5.1.5: Concatenate the enhanced voxel features with the original voxel features to form a new fused feature. Use a pooling operation to extract features within a specific proposal box from the fused feature map to generate the final global fused feature.

[0034] Preferably, in step S5, the local fusion module specifically includes the following steps:

[0035] S5.2.1: Divide each 3D proposal box evenly and calculate the center point of each grid as the representative point of the grid.

[0036] S5.2.2: Use the position information encoder to calculate the position of each grid center point z j Encode the location information to obtain the grid features The specific formula is as follows:

[0037]

[0038] Where γ = z j -c B Suggest a centroid c for each mesh relative to the bounding box B The relative position of For each voxel grid G j is the number of points in , and τ is a constant offset.

[0039] S5.2.3: Project each calculated grid center point onto the multi-view image plane. Use the camera intrinsic and extrinsic parameter matrix to transform the 3D coordinates into 2D image coordinates.

[0040] S5.2.4: Based on the projected grid center point, sample surrounding image features on the image plane.

[0041] S5.2.5: Use the cross-attention mechanism to fuse the encoded grid features with the sampled image features to produce local fused features.

[0042] Preferably, in step S6, the following steps are specifically included:

[0043] S6.1: Preliminarily combine the grid features, local fusion features and global fusion features to form an initial aggregation feature F S , the specific formula is as follows:

[0044]

[0045] in, is the grid feature, is the local fusion feature, It is the global fusion feature.

[0046] S6.2: Use the self-attention mechanism to establish relationships between grid points in the aggregated features. The self-attention mechanism calculates the correlation between each grid point and other grid points to achieve information interaction between features.

[0047] S6.3: The output of the self-attention mechanism is added to the input features to form a residual connection. Next, the features after the residual connection are further processed through the feedforward network to generate the final fused features for subsequent object detection and bounding box refinement.

[0048] Therefore, the present invention adopts the above-mentioned 3D target detection method based on local to global multimodal fusion, which has the following beneficial effects:

[0049] (1) The present invention overcomes the problem of lack of fine-grained information in traditional global fusion methods by synergizing the local fusion module and the global fusion module, and can more comprehensively capture the local details and overall structure of the scene.

[0050] (2) The present invention uses point centroids to more accurately represent the position of each voxel feature, thereby achieving better cross-modal alignment and solving the problem of low alignment accuracy caused by the use of voxel centers in traditional methods.

[0051] (3) The present invention dynamically enhances the object density in sparse LiDAR data by using polar coordinate sampling technology, making objects that were originally difficult to detect easier to identify.

[0052] (4) The present invention introduces a dynamic aggregation module to achieve information interaction between local and global fusion features, further enhances the feature representation capability, generates richer multimodal features, and significantly improves the overall performance of 3D object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a framework diagram of a 3D target detection method based on local to global multimodal fusion of the present invention;

[0054] Figure 2 It is a processing block diagram of the global fusion module of the present invention;

[0055] Figure 3 It is a processing block diagram of the local fusion module of the present invention;

[0056] Figure 4 It is a processing block diagram of the dynamic aggregation module of the present invention; DETAILED DESCRIPTION

[0057] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0058] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0059] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] like Figure 1 As shown, the present invention provides a 3D target detection method based on local to global multimodal fusion, which mainly includes the following steps:

[0061] S1: Acquire point cloud data and its associated six-view images.

[0062] Preferably, the original point cloud data is collected by the LiDAR sensor, and then the image information of the same scene is captured from different perspectives using a multi-camera system to form a set of six-view images that match the point cloud data. For each point cloud, the system projects it onto the corresponding camera image according to its spatial position, thereby establishing a correspondence between the point cloud and the image. This point cloud-image pairing not only retains the geometric structure information in the three-dimensional space, but also combines the rich visual features in the two-dimensional image, providing a basis for subsequent local to global cross-modal fusion.

[0063] S2: Use polar coordinate sampling technology to densify the point cloud.

[0064] Preferably, the original point cloud data including sparse objects and background, as well as the orientation and rotation information of these objects are obtained. A polar coordinate system is established for each object based on the center of the scene, and the sensor front direction is set as the 0 degree reference for subsequent orientation and rotation measurements.

[0065] Furthermore, a database is constructed to store dense objects, which will be used to densify other sparse objects. Objects are divided into N×N groups according to their orientation and rotation relative to the LiDAR sensor. The specific grouping formula is as follows:

[0066]

[0067] Among them, α l ′,β l ′∈{1,2,......,N} is the direction and rotation index of object l, and N is the number of divided groups.

[0068] Finally, the orientation and rotation are calculated to query dense objects of similar density from the database. Additional points are added to each object point set in the original point cloud data to obtain the enhanced point cloud data.

[0069] S3: The densified point cloud is passed through the 3D backbone network to generate voxel features with different downsampling rates, and the initial 3D proposal box is generated from these voxel features through the region proposal network.

[0070] Preferably, the densified point cloud is converted to a voxel representation. The voxelized point cloud is input into a 3D backbone network, which can extract spatial features from different scales. As the network depth increases, a series of voxel features with different downsampling rates will be obtained.

[0071] Furthermore, since the information carried by voxel features at different downsampling rates is different, it is necessary to properly fuse these multi-level features. Based on the above fused multi-level voxel features, a region proposal network is used to predict the location of potential targets. Specifically, RPN evaluates the possibility of each position in the feature map as the center of the object and generates a set of candidate boxes based on this.

[0072] S4: Extracting dense semantic image features from six-view images using a 2D backbone network.

[0073] Preferably, the six-view images collected from multiple camera perspectives are fed as input into a pre-trained 2D detector or feature extraction network. Through this process, the 2D backbone network can capture the high-level semantic information in each image and generate a series of dense feature maps with rich contextual information.

[0074] S5: In the multimodal fusion stage, it specifically includes a local fusion module and a global fusion module. These two modules work together to adaptively integrate the multi-level voxel features, semantic image features, initial 3D proposal boxes, and the position information of the original point cloud generated above.

[0075] Preferably, Figure 2 As shown, in the global fusion module, for each non-empty voxel, its centroid c is calculated according to the spatial coordinates of all the points contained in it. i Position, the specific formula is as follows:

[0076]

[0077] Among them, p i =(x i ,y i ,z i ) is the spatial coordinate, |P(V i )| is the voxel V i The points within.

[0078] Furthermore, the center of mass of each voxel is projected onto the image plane using the camera internal and external parameter matrix to obtain the reference point p i , the specific formula is as follows:

[0079]

[0080] Where M is the product of the camera's internal matrix and external matrix.

[0081] It is worth noting that, according to the reference point, the learned offset is applied on the image feature to select the surrounding image features. These selected image features are weighted averaged to generate the aggregated image features. A deformable cross attention module is used, with each voxel feature as the query and the aggregated image features as the key and value. Through this mechanism, the point cloud features are adaptively fused with the corresponding image features to generate cross-modal features.

[0082] Finally, the enhanced voxel features are concatenated with the original voxel features to form a new fused feature. The pooling operation is used to extract the features within a specific proposal box from the fused feature map to generate the final global fused feature.

[0083] Preferably, Figure 3 As shown in the figure, in the local fusion module, each 3D proposal box is evenly divided, and the center point of each grid is calculated as the representative point of the grid. The position information encoder is used to encode the center point z of each grid. j Encode the location information to obtain the grid features The specific formula is as follows:

[0084]

[0085] Where γ = z j -c B Suggest a centroid c for each mesh relative to the bounding box B The relative position of For each voxel grid G j is the number of points in , and τ is a constant offset.

[0086] Furthermore, each calculated grid center point is projected onto the multi-view image plane. The 3D coordinates are converted to 2D image coordinates using the camera internal and external parameter matrix. Based on the projected grid center point, the surrounding image features are sampled on the image plane.

[0087] Finally, the encoded grid features are fused with the sampled image features using a cross-attention mechanism to generate local fused features.

[0088] S6: The information interaction between local and global fusion features is realized through the dynamic aggregation module, and finally the predicted 3D box is obtained through the feedforward neural network.

[0089] Preferably, Figure 4 As shown in Figure 2, the grid features, local fusion features and global fusion features are preliminarily combined to form an initial aggregation feature F S , the specific formula is as follows:

[0090]

[0091] in, is the grid feature, is the local fusion feature, It is the global fusion feature.

[0092] Furthermore, the self-attention mechanism is used to establish relationships between grid points in the aggregated features. The self-attention mechanism calculates the correlation between each grid point and other grid points, thereby realizing information interaction between features.

[0093] Finally, the output of the self-attention mechanism is added to the input features to form a residual connection. Next, the features after the residual connection are further processed by the feedforward network to generate the final fused features for subsequent object detection and bounding box refinement.

[0094] Therefore, the present invention adopts the above-mentioned 3D target detection method based on local to global multimodal fusion, enhances the density of sparse objects by using polar coordinate sampling technology in the point cloud data processing process, and combines the multi-scale 3D backbone network with the 2D image feature extraction network to achieve more accurate positioning and classification of objects in three-dimensional space.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A 3D object detection method based on local to global multimodal fusion, characterized in that: The following steps are involved: S1: Acquire point cloud data and its associated six-view images. S2: Use polar coordinate sampling technology to densify the point cloud. S3: The densified point cloud is passed through the 3D backbone network to generate voxel features with different downsampling rates, and the initial 3D proposal box is generated from these voxel features through the region proposal network. S4: Extracting dense semantic image features from six-view images using a 2D backbone network. S5: In the multimodal fusion stage, it specifically includes a local fusion module and a global fusion module. These two modules work together to adaptively integrate the multi-level voxel features, semantic image features, initial 3D proposal boxes, and the position information of the original point cloud generated above. S6: The information interaction between local and global fusion features is realized through the dynamic aggregation module, and finally the predicted 3D box is obtained through the feedforward neural network.

2. The 3D object detection method based on local to global multimodal fusion according to claim 1, characterized in that: The specific method of step S2 is as follows: S2.1: Obtain raw point cloud data containing sparse objects and background, as well as the orientation and rotation information of these objects. S2.2: Establish a polar coordinate system for each object based on the center of the scene, and set the sensor front direction as the 0 degree reference for subsequent direction and rotation measurements. S2.3: Build a database to store dense objects, which will be used to densify other sparse objects. Divide the objects into N×N groups according to their orientation and rotation relative to the LiDAR sensor. The specific grouping formula is as follows: Among them, α l ′,β l ′∈{1,2,......,N} is the direction and rotation index of object l, and N is the number of divided groups. S2.4: Calculate the orientation and rotation to query dense objects of similar density from the database. Add additional points to each object point set in the original point cloud data to obtain enhanced point cloud data.

3. The 3D object detection method based on local to global multimodal fusion according to claim 1, characterized in that: The specific method of step S3 is as follows: S3.1: Convert the densified point cloud to voxel representation. S3.2: The voxelized point cloud is input into the 3D backbone network, which can extract spatial features from different scales. As the network depth increases, a series of voxel features with different downsampling rates will be obtained. S3.3: Since voxel features at different downsampling rates carry different information, these multi-level features need to be properly fused. S3.4: Based on the above fused multi-level voxel features, a region proposal network is used to predict the location of potential targets. Specifically, RPN evaluates the possibility of each position in the feature map as the center of the object and generates a set of candidate boxes based on this.

4. The 3D object detection method based on local to global multimodal fusion according to claim 1, characterized in that: The specific method of the global fusion module in step S5 is as follows: S5.1.1: For each non-empty voxel, calculate its centroid c based on the spatial coordinates of all the points contained in it. i Position, the specific formula is as follows: Among them, p i =(x i ,y i ,z i ) is the spatial coordinate, |P(V i )| is the voxel V i The points within. S5.1.2: Use the camera intrinsic and extrinsic parameter matrix to project the centroid of each voxel onto the image plane to obtain the reference point p i , the specific formula is as follows: p i =M·c i (3) in, M is the product of the camera's intrinsic matrix and extrinsic matrix. S5.1.3: Based on the reference point, the learned offset is applied to the image feature to select surrounding image features. These selected image features are weighted averaged to generate the aggregated image feature. S5.1.4: A deformable cross-attention module is used to take each voxel feature as the query and the aggregated image features as the key and value. This mechanism is used to adaptively fuse point cloud features with corresponding image features to generate cross-modal features. S5.1.5: Concatenate the enhanced voxel features with the original voxel features to form a new fused feature. Use a pooling operation to extract features within a specific proposal box from the fused feature map to generate the final global fused feature.

5. The 3D object detection method based on local to global multimodal fusion according to claim 1, characterized in that: The specific method of the local fusion module in step S5 is as follows: S5.2.1: Divide each 3D proposal box evenly and calculate the center point of each grid as the representative point of the grid. S5.2.2: Use the position information encoder to calculate the position of each grid center point z j Encode the location information to obtain the grid features The specific formula is as follows: Where γ = z j -c B Suggest a centroid c for each mesh relative to the bounding box B The relative position of For each voxel grid G j is the number of points in , and τ is a constant offset. S5.2.3: Project each calculated grid center point onto the multi-view image plane. Use the camera intrinsic and extrinsic parameter matrix to transform the 3D coordinates into 2D image coordinates. S5.2.4: Based on the projected grid center point, sample surrounding image features on the image plane. S5.2.5: Use the cross-attention mechanism to fuse the encoded grid features with the sampled image features to produce local fused features.

6. The 3D object detection method based on local to global multimodal fusion according to claim 1, characterized in that: The specific method of step S6 is as follows: S6.1: Preliminarily combine the grid features, local fusion features and global fusion features to form an initial aggregation feature F S , the specific formula is as follows: in, is the grid feature, is the local fusion feature, It is the global fusion feature. S6.2: Use the self-attention mechanism to establish relationships between grid points in the aggregated features. The self-attention mechanism calculates the correlation between each grid point and other grid points to achieve information interaction between features. S6.3: The output of the self-attention mechanism is added to the input features to form a residual connection. Next, the features after the residual connection are further processed through the feedforward network to generate the final fused features for subsequent object detection and bounding box refinement.

Images