Robust multi-mode 3D target detection method based on frame matching

Abstract

The invention discloses a robust multi-modal 3D target detection method based on frame matching, and the method comprises the steps: firstly, respectively extracting features from a dense LiDAR point cloud and a multi-view image which are processed through Polar Sampling, and generating 3D and 2D candidate frames; by learning the corresponding relation between the 3D candidate box and the 2D candidate box, the dependence on accurate calibration is reduced, and efficient multi-modal feature fusion is realized. The process comprises matching of two levels: view level matching determines most relevant image view features, and proposal level matching establishes accurate correspondence of 3D and 2D candidate boxes. A matching-based fusion module combines the ROI features of the candidate frames to generate a final predicted 3D frame. According to the method, cross-modal feature alignment is realized through frame-level matching, fine granularity and overall features are integrated, the ability of the system to deal with sensor asynchronization, position offset, image degradation and other practical problems is remarkably enhanced, and thus the stability and safety of 3D target detection are greatly improved.

Description

Technical Field

[0001] The present invention relates to the fields of 3D target detection and autonomous driving, and in particular to a robust multimodal 3D target detection method based on frame matching. Background Art

[0002] In the field of multimodal 3D object detection, existing technologies face challenges in many aspects. First, data fusion is a major difficulty. Since this field usually needs to integrate data from different sensors (such as LiDAR laser radar and RGB camera), and these data sources have significant differences in information format and characteristics, how to efficiently and accurately fuse these heterogeneous data has become a primary problem. Secondly, the accuracy problem of cross-modal matching is also prominent. The information captured by different sensors is essentially different, which makes it difficult for existing matching algorithms to achieve high-precision matching in complex scenes, thereby affecting the overall performance. In addition, existing technologies perform poorly when dealing with environmental changes such as severe weather and extreme lighting, which directly affects the practicality and reliability of the system. Furthermore, in order to pursue higher detection accuracy, the design of some advanced algorithms has become too complex, which not only greatly increases the demand for computing resources, but also reduces the real-time response capability of the system. Finally, the existing methods have insufficient generalization ability when dealing with unseen objects or detection tasks in new environments, which limits the application scope and development potential of the technology. These challenges jointly restrict the progress of multimodal 3D object detection technology.

[0003] Therefore, it is necessary to provide a robust multimodal 3D object detection method based on box matching to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a robust multimodal 3D target detection method based on frame matching, which solves the problems of difficult data fusion, low cross-modal matching accuracy, poor environmental adaptability, low computational efficiency and insufficient generalization ability in multimodal 3D object detection.

[0005] To achieve the above object, the present invention provides a robust multimodal 3D target detection method based on frame matching, comprising the following steps:

[0006] S1: Acquire 3D point cloud data from the LiDAR sensor and simultaneously acquire image data from cameras with multiple viewpoints.

[0007] S2: Use Polar Sampling technology to pre-process 3D point cloud data to enhance the quality and density of point cloud data.

[0008] S3: Use the 3D backbone network to extract features from the preprocessed point cloud data and generate 3D candidate boxes; at the same time, use the 2D backbone network to extract dense semantic features from multi-view images and generate 2D candidate boxes.

[0009] S4: Perform view-level matching to determine the most relevant image view features for each 3D candidate box.

[0010] S5: Perform proposal-level matching to establish an accurate correspondence between 3D and 2D candidate boxes.

[0011] S6: The matching-based fusion module combines the ROI features of the 3D and 2D candidate boxes to achieve efficient fusion of multimodal features, and outputs the final 3D object detection results through a feedforward neural network.

[0012] Preferably, in step S2, the following steps are specifically included:

[0013] S2.1: Obtain raw point cloud data containing sparse objects and background, as well as the orientation and rotation information of these objects.

[0014] S2.2: Establish a polar coordinate system for each object based on the center of the scene, and set the sensor front direction as the 0 degree reference for subsequent direction and rotation measurements.

[0015] S2.3: Build a database to store dense objects, which will be used to densify other sparse objects. Divide the objects into N×N groups according to their orientation and rotation relative to the LiDAR sensor. The specific grouping formula is as follows:

[0016]

[0017] Among them, α l ′,β l ′∈{1,2,......,N} is the direction and rotation index of object l, and N is the number of divided groups.

[0018] S2.4: Calculate the orientation and rotation to query dense objects of similar density from the database. Add additional points to each object point set in the original point cloud data to obtain enhanced point cloud data.

[0019] Preferably, in step S3, the following steps are specifically included:

[0020] S3.1: Input the preprocessed point cloud data into the 3D backbone network, and generate voxel features at different levels by performing multi-layer convolution operations on the point cloud data. At the same time, input the multi-view images into the 2D backbone network, and extract dense semantic features through multi-layer convolution operations.

[0021] S3.2: Generate initial 3D candidate boxes from the voxel features extracted by the 3D backbone network using the region proposal network (RPN). Similarly, generate initial 2D candidate boxes from the semantic feature map extracted by the 2D backbone network using RPN.

[0022] S3.3: Through the non-maximum suppression (NMS) operation, the high-quality 3D candidate boxes that are most likely to contain the target are screened out from a large number of initial 3D candidate boxes. At the same time, through the NMS operation, the high-quality 2D candidate boxes that are most likely to contain the target are screened out from a large number of initial 2D candidate boxes.

[0023] Preferably, in step S4, the following steps are specifically included:

[0024] S4.1: Extract features of each 3D candidate box from the 3D backbone network, which are called 3D object queries.

[0025] S4.2: Use multi-view image features as values ​​and keys, 3D candidate box features as queries, and cross-attention mechanism to achieve feature interaction between 3D candidate box features and multi-view image features to generate query image features F ca , the specific formula is as follows:

[0026] F ca = CrossAtt(F 3d ,F im ) (2)

[0027] Among them, F 3d is the 3D candidate box feature, F im is the image feature.

[0028] S4.3: Further perform high-dimensional 3D position encoding on the center position of each 3D object candidate box to generate a 3D position embedding feature.

[0029] S4.4: Query image feature F ca , 3D position embedding features And the original 3D candidate box feature F 3d Combined in a connected manner and aggregated into a compact feature representation through a multi-layer perceptron (MLP), the specific formula is as follows:

[0030]

[0031] Among them, P cls Represents the output of view classification.

[0032] S4.5: The aggregated features are fed into a classification network to predict the view-level matching classification. The output of the network represents the matching classification of each 3D candidate box with each view.

[0033] Preferably, in step S5, the following steps are specifically included:

[0034] S5.1: Extract the 3D candidate box features F from the 3D backbone network3d , 3D position embedding features and 3D classification features Connect to form comprehensive features of 3D candidate boxes The specific formula is as follows:

[0035]

[0036] S5.2: Extract the 2D candidate box features F from the 2D backbone network 2d , 2D position embedding features and 2D categorical features Connect to form the comprehensive features of the 2D candidate box. The specific formula is as follows:

[0037]

[0038] S5.3: Combine the comprehensive features of the 3D candidate box and the comprehensive features of the 2D candidate box to form a combined feature. Input the combined feature into a multi-layer perceptron (MLP) to calculate the matching matrix between the 3D candidate box and the 2D candidate box. Each element of the matching matrix represents the similarity between the 3D candidate box and the 2D candidate box.

[0039] Preferably, in step S6, the following steps are specifically included:

[0040] S6.1: Using the Transformer decoder layer, the 3D candidate box features are used as queries 2D ROI feature For keys and values, perform self-attention and cross-attention operations. The specific formula is as follows:

[0041]

[0042] in, Represents the pixel feature mask within the 2D ROI area, ensuring that only pixel features in the matching image are retained.

[0043] S6.2: Realizing 3D candidate box features through simple feature concatenation and multi-layer perceptron With 2D ROI features The specific formula is as follows:

[0044]

[0045] Where S represents the maximum matching score, which is used to re-weight the 2D ROI features, and · represents element-wise multiplication.

[0046] S6.3: Through the Transformer decoder layer, the 3D ROI features With 2D ROI features The specific formula is as follows:

[0047]

[0048] in, Represents the pixel feature mask within the 2D ROI area, ensuring that only pixel features in the matching image are retained.

[0049] S6.4: Combine the features O1, O2, and O3 obtained by the above three fusion methods to form the final multimodal feature representation. Input the final multimodal feature into the feedforward neural network, and output the final 3D object detection result through the fully connected layer and regression layer.

[0050] Therefore, the present invention adopts the above-mentioned robust multimodal 3D object detection method based on frame matching, which has the following beneficial effects:

[0051] (1) The present invention proposes a robust multimodal fusion framework based on a frame matching mechanism, which eliminates the heavy reliance of existing multimodal 3D detection methods on the projection matrix.

[0052] (2) The present invention dynamically enhances the object density in sparse LiDAR data by using Polar Sampling technology, making objects that were originally difficult to detect easier to identify.

[0053] (3) We propose a two-stage matching process (i.e., view-level matching and proposal-level matching) to effectively learn to associate 3D and 2D proposals in a coarse-to-fine manner, without the need for calibration during the inference phase, and compared to existing methods, it is robust to asynchronous sensors, misaligned sensor placements, and even degraded camera images. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a framework diagram of a robust multimodal 3D object detection method based on frame matching of the present invention;

[0055] Figure 2 is a view level matching processing block diagram of the present invention;

[0056] Figure 3 is a block diagram of the proposal level matching process of the present invention; DETAILED DESCRIPTION

[0057] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0058] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0059] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] like Figure 1 As shown, the present invention provides a robust multimodal 3D target detection method based on frame matching, which mainly includes the following steps:

[0061] S1: Acquire 3D point cloud data from the LiDAR sensor and simultaneously acquire image data from cameras with multiple viewpoints.

[0062] Preferably, a high-resolution LiDAR sensor is used to collect 3D point clouds, and preprocessing is performed through denoising, downsampling, and coordinate transformation; at the same time, cameras with multiple viewing angles (such as front-view, side-view, and rear-view cameras) are used to collect image data, and preprocessing such as distortion correction, cropping, and scaling is performed. The temporal consistency of the data is ensured through hardware and software synchronization, and the spatial consistency of the data is ensured through initial alignment and dynamic alignment. These high-quality multimodal data provide a reliable foundation for subsequent feature fusion and target detection.

[0063] S2: Use Polar Sampling technology to pre-process 3D point cloud data to enhance the quality and density of point cloud data.

[0064] Preferably, the original point cloud data including sparse objects and background, as well as the orientation and rotation information of these objects are obtained. A polar coordinate system is established for each object based on the center of the scene, and the sensor front direction is set as the 0 degree reference for subsequent orientation and rotation measurements.

[0065] Furthermore, a database is constructed to store dense objects, which will be used to densify other sparse objects. Objects are divided into N×N groups according to their orientation and rotation relative to the LiDAR sensor. The specific grouping formula is as follows:

[0066]

[0067] Among them, α l ′,β l ′∈{1,2,......,N} is the direction and rotation index of object l, and N is the number of divided groups.

[0068] Finally, the orientation and rotation are calculated to query dense objects of similar density from the database. Additional points are added to each object point set in the original point cloud data to obtain the enhanced point cloud data.

[0069] S3: Use the 3D backbone network to extract features from the preprocessed point cloud data and generate 3D candidate boxes; at the same time, use the 2D backbone network to extract dense semantic features from multi-view images and generate 2D candidate boxes.

[0070] Preferably, the preprocessed point cloud data is input into the 3D backbone network, and voxel features at different levels are generated by performing multi-layer convolution operations on the point cloud data. At the same time, the multi-view images are input into the 2D backbone network, and dense semantic features are extracted by multi-layer convolution operations.

[0071] Furthermore, the region proposal network (RPN) is used to generate the initial 3D candidate box from the voxel features extracted by the 3D backbone network. Similarly, the RPN is used to generate the initial 2D candidate box from the semantic feature map extracted by the 2D backbone network.

[0072] Finally, the non-maximum suppression (NMS) operation is used to select the high-quality 3D candidate boxes that are most likely to contain the target from a large number of initial 3D candidate boxes. At the same time, the NMS operation is used to select the high-quality 2D candidate boxes that are most likely to contain the target from a large number of initial 2D candidate boxes.

[0073] S4: Perform view-level matching to determine the most relevant image view features for each 3D candidate box.

[0074] Preferably, the features of each 3D candidate box are extracted from the 3D backbone network, and the multi-view image features are used as values ​​and keys, and the 3D candidate box features are used as queries. The cross attention mechanism is used to realize the feature interaction between the 3D candidate box features and the multi-view image features to generate the query image features F ca , the specific formula is as follows:

[0075] F ca = CrossAtt(F 3d ,F im ) (2)

[0076] Among them, F 3dis the 3D candidate box feature, F im is the image feature.

[0077] The center position of each 3D object candidate box is further encoded in high dimensional 3D position to generate 3D position embedding features. ca , 3D position embedding features And the original 3D candidate box feature F 3d Combined in a connected manner and aggregated into a compact feature representation through a multi-layer perceptron (MLP), the specific formula is as follows:

[0078]

[0079] Among them, P cls Represents the output of view classification.

[0080] Finally, the aggregated features are fed into a classification network to predict the view-level matching classification. The output of this network represents the matching classification of each 3D candidate box with each view.

[0081] S5: Perform proposal-level matching to establish an accurate correspondence between 3D and 2D candidate boxes.

[0082] Preferably, the 3D candidate box feature F extracted from the 3D backbone network 3d , 3D position embedding features and 3D classification features Connect to form comprehensive features of 3D candidate boxes The specific formula is as follows:

[0083]

[0084] Furthermore, the 2D candidate box features F extracted from the 2D backbone network are 2d , 2D position embedding features and 2D categorical features Connect to form the comprehensive features of the 2D candidate box. The specific formula is as follows:

[0085]

[0086] Finally, the comprehensive features of the 3D candidate box and the comprehensive features of the 2D candidate box are combined to form a combined feature. The combined feature is input into a multi-layer perceptron (MLP) to calculate the matching matrix between the 3D candidate box and the 2D candidate box. Each element of the matching matrix represents the similarity between the 3D candidate box and the 2D candidate box.

[0087] S6: The matching-based fusion module combines the ROI features of the 3D and 2D candidate boxes to achieve efficient fusion of multimodal features, and outputs the final 3D object detection results through a feedforward neural network.

[0088] Preferably, a Transformer decoder layer is used to take the 3D candidate box features as the query 2D ROI feature For keys and values, perform self-attention and cross-attention operations. The specific formula is as follows:

[0089]

[0090] in, Represents the pixel feature mask within the 2D ROI area, ensuring that only pixel features in the matching image are retained.

[0091] Furthermore, 3D candidate box features are realized through simple feature splicing and multi-layer perceptron With 2DROI feature The specific formula is as follows:

[0092]

[0093] Where S represents the maximum matching score, which is used to re-weight the 2D ROI features, and · represents element-wise multiplication.

[0094] It is worth noting that through the Transformer decoder layer, the 3D ROI features With 2D ROI features The specific formula is as follows:

[0095]

[0096] in, Represents the pixel feature mask within the 2D ROI area, ensuring that only pixel features in the matching image are retained.

[0097] Finally, the features O1, O2, and O3 obtained by the above three fusion methods are combined to form the final multimodal feature representation. The final multimodal feature is input into the feedforward neural network, and the final 3D object detection result is output through the fully connected layer and regression layer.

[0098] Therefore, the present invention adopts the above-mentioned robust multimodal 3D target detection method based on frame matching, and uses Polar Sampling technology to enhance the density of sparse objects in the point cloud data processing process, and combines frame matching technology to reduce the dependence on precise calibration, thereby achieving efficient multimodal feature fusion and significantly improving the stability and accuracy of 3D target detection.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A robust multimodal 3D object detection method based on box matching, characterized in that: The following steps are involved: S1: Acquire 3D point cloud data from the LiDAR sensor and simultaneously acquire image data from cameras with multiple viewpoints. S2: Use Polar Sampling technology to pre-process 3D point cloud data to enhance the quality and density of point cloud data. S3: Use the 3D backbone network to extract features from the preprocessed point cloud data and generate 3D candidate boxes; at the same time, use the 2D backbone network to extract dense semantic features from multi-view images and generate 2D candidate boxes. S4: Perform view-level matching to determine the most relevant image view features for each 3D candidate box. S5: Perform proposal-level matching to establish an accurate correspondence between 3D and 2D candidate boxes. S6: The matching-based fusion module combines the ROI features of the 3D and 2D candidate boxes to achieve efficient fusion of multimodal features, and outputs the final 3D object detection results through a feedforward neural network.

2. The robust multimodal 3D object detection method based on box matching according to claim 1, characterized in that: The specific method of step S2 is as follows: S2.1: Obtain raw point cloud data containing sparse objects and background, as well as the orientation and rotation information of these objects. S2.2: Establish a polar coordinate system for each object based on the center of the scene, and set the sensor front direction as the 0 degree reference for subsequent direction and rotation measurements. S2.3: Build a database to store dense objects, which will be used to densify other sparse objects. Divide the objects into N×N groups according to their orientation and rotation relative to the LiDAR sensor. The specific grouping formula is as follows: Among them, α l ′,β l ′∈{1,2,......,N} is the direction and rotation index of object l, and N is the number of divided groups. S2.4: Calculate the orientation and rotation to query dense objects of similar density from the database. Add additional points to each object point set in the original point cloud data to obtain enhanced point cloud data.

3. The robust multimodal 3D object detection method based on box matching according to claim 1, characterized in that: The specific method of step S3 is as follows: S3.1: Input the preprocessed point cloud data into the 3D backbone network, and generate voxel features at different levels by performing multi-layer convolution operations on the point cloud data. At the same time, input the multi-view images into the 2D backbone network, and extract dense semantic features through multi-layer convolution operations. S3.2: Generate initial 3D candidate boxes from the voxel features extracted by the 3D backbone network using the region proposal network (RPN). Similarly, generate initial 2D candidate boxes from the semantic feature map extracted by the 2D backbone network using RPN. S3.3: Through the non-maximum suppression (NMS) operation, the high-quality 3D candidate boxes that are most likely to contain the target are screened out from a large number of initial 3D candidate boxes. At the same time, through the NMS operation, the high-quality 2D candidate boxes that are most likely to contain the target are screened out from a large number of initial 2D candidate boxes.

4. The robust multimodal 3D object detection method based on box matching according to claim 1, characterized in that: The specific method of step S4 is as follows: S4.1: Extract features of each 3D candidate box from the 3D backbone network, which are called 3D object queries. S4.2: Use multi-view image features as values ​​and keys, 3D candidate box features as queries, and cross-attention mechanism to achieve feature interaction between 3D candidate box features and multi-view image features to generate query image features F ca , the specific formula is as follows: F ca =CrossAtt(F 3d ,F im ) (2) Among them, F 3d is the 3D candidate box feature, F im is the image feature. S4.3: Further perform high-dimensional 3D position encoding on the center position of each 3D object candidate box to generate a 3D position embedding feature. S4.4: Query image feature F ca , 3D position embedding features And the original 3D candidate box feature F 3d Combined in a connected manner and aggregated into a compact feature representation through a multi-layer perceptron (MLP), the specific formula is as follows: Among them, P cls Represents the output of view classification. S4.5: The aggregated features are fed into a classification network to predict the view-level matching classification. The output of the network represents the matching classification of each 3D candidate box with each view.

5. The robust multimodal 3D object detection method based on box matching according to claim 1, characterized in that: The specific method of step S5 is as follows: S5.1: Extract the 3D candidate box features F from the 3D backbone network 3d , 3D position embedding features and 3D classification features Connect to form comprehensive features of 3D candidate boxes The specific formula is as follows: S5.2: Extract the 2D candidate box features F from the 2D backbone network 2d , 2D position embedding features and 2D categorical features Connect to form the comprehensive features of the 2D candidate box. The specific formula is as follows: S5.3: Combine the comprehensive features of the 3D candidate box and the comprehensive features of the 2D candidate box to form a combined feature. Input the combined feature into a multi-layer perceptron (MLP) to calculate the matching matrix between the 3D candidate box and the 2D candidate box. Each element of the matching matrix represents the similarity between the 3D candidate box and the 2D candidate box.

6. The robust multimodal 3D object detection method based on box matching according to claim 1, characterized in that: The specific method of step S6 is as follows: S6.1: Using the Transformer decoder layer, the 3D candidate box features are used as queries 2D ROI feature For keys and values, perform self-attention and cross-attention operations. The specific formula is as follows: in, Represents the pixel feature mask within the 2D ROI area, ensuring that only pixel features in the matching image are retained. S6.2: Realizing 3D candidate box features through simple feature concatenation and multi-layer perceptron With 2D ROI features The specific formula is as follows: in, S represents the maximum matching score, which is used to re-weight the 2D ROI features, and · represents element-wise multiplication. S6.3: Through the Transformer decoder layer, the 3D ROI features With 2D ROI features The specific formula is as follows: in, Represents the pixel feature mask within the 2D ROI area, ensuring that only pixel features in the matching image are retained. S6.4: Combine the features O1, O2, and O3 obtained by the above three fusion methods to form the final multimodal feature representation. Input the final multimodal feature into the feedforward neural network, and output the final 3D object detection result through the fully connected layer and regression layer.

Images