A multi-view multi-modal information fusion target detection method

Abstract

The present invention discloses a multi-view multimodal information fusion target detection method, which uses the front view and bird's-eye view of the camera, laser point cloud and millimeter wave point cloud as the input of the perception system, designs a multi-scale fusion module to fuse the front view and bird's-eye view features of multiple sensors, fully extracts semantic and spatial information, and uses a self-attention mechanism to adaptively allocate weight coefficients of the front view and bird's-eye view features. Compared with target detection under a single view, the target detection task under multiple view angles of the present invention fully extracts semantic information and spatial position information. Therefore, the present invention effectively combines the features of the multi-sensor front view and the bird's-eye view to further improve the accuracy of target detection in complex road environments.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a target detection method based on multi-view multi-modal information fusion. Background Art

[0002] At present, due to its own characteristics, a single sensor is difficult to independently cope with target detection tasks under complex road conditions and bad weather. Therefore, multi-sensor information fusion technology has attracted widespread attention. Multi-sensor fusion technology fully combines the advantages of each sensor and can better extract semantic information and spatial location information, so as to accurately identify traffic participants on the road.

[0003] Patent CN202410774348.X proposes a multimodal 3D target detection method and device for autonomous driving. This method extracts image features from image data and point cloud features from point cloud data; bidirectionally enhances and fuses image features and point cloud features to obtain point cloud enhanced image features and image enhanced point cloud features, and fuses them; the point cloud features fused with image features are sent to the 3D region proposal network and refinement module for 3D target detection. This method uses a bidirectional enhancement method to enhance the feature information of the image and point cloud, but does not use the point cloud information of the millimeter wave radar, making the perception system not robust under complex weather conditions.

[0004] Patent CN202410551956.4 proposes a camouflaged target recognition method and system based on multimodal information fusion. This method aggregates information of multiple frames of RGB images, infrared images and point cloud images collected in the same time period, realizes time synchronization of multimodal information, and performs feature fusion of multimodal information; most of them perform target detection tasks under a single perspective, and do not combine the respective advantages of cameras, lidar and millimeter-wave radar sensors, making it difficult to achieve accurate regression of the category and position of road traffic participants under complex weather conditions. Therefore, multimodal information fusion target detection combining front view and bird's-eye view features is the key to intelligent driving perception systems.

[0005] Existing target detection technology is performed under a single perspective. Target detection only under the front view can obtain rich semantic information, but it is difficult to accurately obtain spatial position information; target detection only under the bird's-eye view can obtain accurate spatial position information, but it is difficult to capture rich semantic information. Therefore, it is difficult to achieve high-precision recognition of targets under a single perspective. Therefore, a target detection method that can combine the features of the front view and the bird's-eye view is needed to improve the accuracy of the target detection network. Summary of the invention

[0006] In order to solve at least one of the problems raised in the above background technology, the present invention provides a multi-view multi-modal information fusion target detection method.

[0007] To achieve the above object, the present invention provides the following technical solution: a multi-view multi-modal information fusion target detection method, comprising the following steps:

[0008] Step 1: Receive raw data from the camera, lidar and millimeter-wave radar;

[0009] Step 2: Process the raw data collected by the camera, laser radar and millimeter wave radar, and use the inverse perspective mapping method to convert the camera front view into a bird's-eye view; use the projection method to convert the laser radar point cloud and the millimeter wave point cloud into a front view and a bird's-eye view;

[0010] Step 3: Use the feature extraction module to extract features from the front view and bird's-eye view of the camera, the front view and bird's-eye view of the lidar, and the bird's-eye view of the millimeter-wave radar to obtain the camera front view features. With bird's eye view feature , LiDAR front view features With bird's eye view feature , millimeter wave radar bird's eye view features ;

[0011] Step 4: Fusion processing is performed on the front view and bird's-eye view features of different sensors to obtain semantic information and spatial position information;

[0012] Step 5: The forward-looking fusion features obtained after fusion in step 4 , Bird's-eye view fusion features and features containing semantic and spatial information , and fuse their respective multi-scale features to obtain , , , and then transmit these three fused multi-scale features to the detection head for data inference; in the data inference stage, the detection head will sequentially , , Predict the confidence level and category information of traffic participants’ positions.

[0013] Preferably, step 4 specifically includes the following steps:

[0014] Step 401: Use the front view fusion module to fuse the camera front view features Front view features with LiDAR , obtain the forward-looking fusion features containing dense semantic information ;

[0015] Step 402: Use the bird's eye view fusion module to fuse the camera's bird's eye view features , LiDAR bird's eye view features Bird's-eye view feature with 4D millimeter wave radar , obtain the bird's-eye view fusion features containing spatial location information ;

[0016] Step 403: Use the attention fusion module to fuse the front-view fusion features Merge features with bird's eye view , adaptively assign weight coefficients of semantic information and spatial information to obtain features containing both semantic and spatial information ;

[0017] Step 404: upsample the features of each unimodal branch, and use the front view fusion module, the bird's eye view fusion module and the attention fusion module to fusion the upsampled unimodal features ( , , , , ) are fused to generate multi-scale fusion features with scales of (128, 256, 512, 1024).

[0018] In step 401, obtaining the front view features of the camera and the laser radar includes the following steps: first, the front view features of the camera and the laser radar are spliced ​​to obtain a fusion feature ; Secondly, normalize and convolve the spliced ​​features to obtain the features ; The processed features Add the camera's front view feature to get the fused front view feature .

[0019] In step 402, obtaining the fusion features of the bird's-eye view of the camera, the laser radar and the millimeter-wave radar includes the following steps: first, the bird's-eye view features of the camera, the laser radar and the millimeter-wave radar are spliced ​​to obtain the fusion features ; Secondly, normalize and convolve the spliced ​​features to obtain the features ; Finally, the processed features Add the features of the laser radar to the bird's-eye view to obtain the fused bird's-eye view features. .

[0020] In step 403, obtain and The fused features include the following steps: First, the fused and Each is transmitted into the convolution layer and the activation function layer to obtain the features and Secondly, and Features are spliced ​​to obtain features , and the features Through the Softmax layer, the normalization operation is performed to obtain the features ; Again, the feature and , Multiply to get the feature and Finally, the feature and Splice and get .

[0021] Preferably, in step 2, converting the camera's front view to a bird's-eye view includes the following steps: using the camera to batch collect front views; determining the internal and external parameters of the camera, including focal length, principal point position, and camera position and posture, and deriving an inverse perspective change matrix M through a series of parameters; selecting the ground corresponding to the front view as a reference plane; and converting the front view collected by the camera into a bird's-eye view through the inverse perspective change matrix M.

[0022] Preferably, in step 2, the conversion of the lidar point cloud data into a front view and a bird's-eye view includes the following steps: mapping the lidar point cloud to the image using a lidar→camera→image coordinate system transformation matrix, and assigning color to the image point cloud according to the depth information to generate a front view of the laser point cloud; setting the z coordinate of the point (x, y, z) projected in the front view to 0 and projecting it onto the bird's-eye view to generate a laser point cloud bird's-eye view.

[0023] Preferably, in step 2, the conversion of millimeter-wave radar point cloud data into a bird's-eye view includes the following steps: filtering out the point cloud mapped to the image through the millimeter-wave radar→camera→image coordinate system transformation matrix, mapping these points to the bird's-eye view according to the coordinate values ​​(x, y), and generating a millimeter-wave point cloud bird's-eye view.

[0024] Preferably, in step 3, feature extraction of the front view and bird's-eye view features of the camera, lidar and millimeter-wave radar includes the following steps: transmitting the front view and bird's-eye view features of each sensor into the ConvNext module, performing convolution and normalization operations on the features, and finally performing upsampling operations on the processed features three times to obtain feature maps of four different scales respectively.

[0025] Preferably, during the model training phase, the multi-scale fusion features are passed into the detection head to generate a prediction value, the deviation between the prediction value and the true value is calculated through the loss function, and back propagation is performed to update the network weights, thereby gradually improving the detection performance of the model.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The present invention converts the front view of the camera into a bird's-eye view through an inverse perspective mapping method, and obtains the front view, bird's-eye view and millimeter-wave point cloud bird's-eye view respectively through a projection method; secondly, the front view and bird's-eye view features of the camera, the front view and bird's-eye view features of the laser point cloud, and the bird's-eye view features of the millimeter-wave point cloud are extracted; thirdly, the front view and bird's-eye view features of each sensor are fused using each fusion module; finally, multi-scale features containing semantic information and spatial position information are obtained, and the multi-scale features are used as the input of the detection head, thereby improving the detection accuracy of the position information of traffic participants and the detection accuracy of the category information.

[0028] 2. The present invention uses a self-attention mechanism to adaptively fuse the features of the front view and the bird's-eye view, thereby achieving adaptive adjustment of the weights of semantic information and position information. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0030] Figure 1 is a flow chart of the steps of the present invention;

[0031] Figure 2 This is a target detection model framework diagram of the present invention;

[0032] Figure 3 It is a framework diagram of the front view fusion module of the present invention;

[0033] Figure 4 This is a framework diagram of a bird's-eye view fusion module of the present invention;

[0034] Figure 5 This is a framework diagram of the weight graph fusion module of the present invention;

[0035] Figure 6 It is a framework diagram of the feature interaction module of the present invention;

[0036] Figure 7 This is a framework diagram of the self-attention fusion module of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0038] First, refer to Figures 1 to 7 A multi-view multi-modal information fusion target detection method according to an embodiment of the present invention is specifically described.

[0039] A multi-view multi-modal information fusion target detection method includes the following steps: Step 1: in the stage of building a hardware platform for a perception system, calibrate the internal and external parameters of a camera and a laser radar, complete the spatial synchronization between the camera, the laser radar and the millimeter-wave radar, and complete the mapping of the laser radar and the millimeter-wave radar point cloud on the camera image; at the beginning, the camera, the laser radar and the millimeter-wave radar have completed the alignment of the spatial coordinate system, and completed the time synchronization by means of external triggering;

[0040] Step 2: Complete the conversion between the front view and the bird's-eye view of the camera image, lidar point cloud, and millimeter-wave radar point cloud data:

[0041] Regarding converting the camera front view into a bird's-eye view, this implementation example adopts an inverse perspective mapping method, obtains an inverse perspective transformation matrix through the internal and external parameters of the camera and the coordinate system of the camera, and converts the camera front view into a bird's-eye view through the inverse perspective mapping matrix;

[0042] Regarding converting laser point cloud data into a front view image, this implementation example uses the laser radar → camera → image coordinate system transformation matrix to map the laser radar point cloud to the image, and assigns color to the image point cloud according to the depth information to generate a front view of the laser point cloud; the z coordinate of the point (x, y, z) projected in the front view is set to 0 and projected onto the bird's-eye view to generate a laser point cloud bird's-eye view.

[0043] Regarding converting millimeter-wave point cloud data into a bird's-eye view, this implementation example uses the millimeter-wave radar→camera→image coordinate system transformation matrix to filter out the point cloud mapped to the image, and maps these points to the bird's-eye view according to the coordinate values ​​​​(x, y) to generate a millimeter-wave point cloud bird's-eye view.

[0044] Step 3: Use the feature extraction module to extract features from the front view and bird's-eye view of the camera, the front view and bird's-eye view of the lidar, and the bird's-eye view of the millimeter-wave radar to obtain the camera front view features With bird's eye view feature , LiDAR front view features With bird's eye view feature , millimeter wave radar bird's eye view features ;

[0045] Step 4: Fusion processing is performed on the front view and bird's-eye view features of different sensors to obtain semantic information and spatial position information;

[0046] Step 401: extracting features from the front view and bird's-eye view data of the camera and laser point cloud, and the bird's-eye view features of the millimeter-wave point cloud by using a ConvNeXt feature extractor;

[0047] Step 402: Figure 5 As shown in FIG. 1 , in order to enhance the features of the camera and laser point cloud bird's-eye view, the weight map generated by the front view features is multiplied by the respective bird's-eye view features to obtain the enhanced bird's-eye view features. First, the front view features of the camera or laser point cloud extracted in step 401 are passed through a Sigmoid activation function layer to generate a weight map ; Secondly, combine the camera or laser point cloud bird's-eye view features with the weight map Multiply to get the weight feature ;

[0048] ;

[0049] in The front view features extracted for the Camera or Lidar single peak branch; Bird's-eye view features extracted from the single-peak branch of Camera or Lidar; is the sigmoid activation function; Represents feature multiplication.

[0050] Step 403: Figure 3 As shown in Figure 1, the front view fusion module is used to fuse the front view features of the camera and the laser point cloud. First, the front view features of the camera and the laser point cloud are spliced ​​to obtain the feature ; Secondly, the characteristics After passing through the LayerNorm layer and the convolution layer in turn, the features are obtained ;Finally, combine the camera front view features with the features Add together to get the features ;

[0051] ;

[0052] in , The front view features output by the feature extraction module in each unimodal; It is the fusion feature output by the front view fusion module; To reduce the dimension of features through convolution; It is the feature normalization operation; Indicates feature concatenation; + indicates feature addition.

[0053] like Figure 4 As shown in Figure 1, the bird's-eye view fusion module is used to fuse the processed camera, laser point cloud and millimeter wave point cloud features. First, the bird's-eye view features of the camera, laser point cloud and millimeter wave point cloud are spliced ​​to obtain the feature ; Secondly, the characteristics After passing through the LayerNorm layer and the convolution layer once, the features are obtained ;Finally, combine the camera bird's-eye view features with the features Add together to get the features ;

[0054] ;

[0055] in , , The bird's-eye view features output by the feature extraction module in each single peak; The fusion features output by the bird's-eye view fusion module; To reduce the dimension of features through convolution; It is the feature normalization operation; Indicates feature concatenation; + indicates feature addition.

[0056] Step 404: Figure 6 As shown in the figure, the feature interaction module is used to fuse the features of the unimodal branch with the fusion features of the front view or the bird's-eye view. First, the features of the unimodal branch are concatenated with the fusion features of the front view or the bird's-eye view to obtain the feature ; Secondly, the features are passed through the convolution layer, GELU activation function layer, LayerNorm layer, etc. to obtain the features ; Finally, the feature Add to the unimodal feature to get the feature ;

[0057] ;

[0058] in Features output by the front view or bird's eye view fusion module; Features extracted for unimodal branches; To reduce the dimension of features through convolution; It is a convolution operation with a convolution kernel dimension of 3; is the GELU activation function; It is the feature normalization operation; Indicates feature concatenation; + indicates feature addition.

[0059] Step 405: Figure 7 As shown in Figure 1, the self-attention fusion module is used to fuse the front view fusion features and the bird's-eye view fusion features. First, the front view fusion features and the bird's-eye view fusion features are respectively subjected to the convolution layer, the Tanh activation function layer, etc. to obtain the feature , Secondly, and Features are spliced ​​to obtain features , and the features Through the Softmax layer, the normalization operation is performed to obtain the features ; Finally, the feature and , Multiply to get the feature and , and the features and Splice and get ;

[0060] ;

[0061] in , It is the feature output by the front view and bird's eye view fusion module; is the convolution block; is the Tanh activation function; is the Softmax activation function; Represents feature splicing; Represents feature multiplication.

[0062] Step 406: Figure 2 As shown, fusion , and The fused features are then transferred to the detection head for classification and regression tasks. , and The features of three different scales are fused respectively; secondly, the three multi-scale feature sets are passed into the detection head, and the category and position are regressed through the set Loss function.

[0063] Among them, Figure 2 As shown, the features , and By fusing the upsampled branch features, fused features with feature channel sizes of 256, 512, and 1024 are generated.

[0064] Among them, the detection head uses multi-level loss to regress the target category and detection box information, and the total loss is:

[0065] ;

[0066] in represents the total loss; , , Represents the loss of front view features, bird's-eye view features, and fusion features; , , is the loss weight, which is used to optimize the balance of each loss.

[0067] The present invention adopts Focal loss as the category loss, L1 loss and Generalized louloss (GIoU) ​​as the bounding box position loss, and features , and The loss is:

[0068] ;

[0069] in( , , )、( , , )and( , , ) are the Focal loss, L1 loss and GIoU loss weights of the front view, bird's-eye view and fusion features respectively; ( , , )、( , , )and( , , ) are Focalloss, L1 loss and GIoU loss of the front view, bird's-eye view and fusion features respectively.

[0070] Among them, in order to improve the accuracy of category and bounding box regression, the The weight coefficient of Focal loss in loss; improvement in The weights of L1 loss and Generalized lou (GIoU) ​​loss in the proposed method are and , Assign a larger value.

[0071] Step 5: Fusion of the forward-looking features obtained after fusion in step 4 , Bird's-eye view fusion features and features containing semantic and spatial information , and fuse their respective multi-scale features to obtain , , , and then transmit these three fused multi-scale features to the detection head for data inference; in the data inference stage, the detection head will sequentially , , Predict the confidence level and category information of traffic participants’ positions.

Claims

1. A multi-view multi-modal information fusion target detection method, characterized in that: The following steps are involved: Step 1: Receive raw data from the camera, lidar and millimeter-wave radar; Step 2: Process the raw data collected by the camera, laser radar and millimeter wave radar, and use the inverse perspective mapping method to convert the camera front view into a bird's-eye view; use the projection method to convert the laser radar point cloud and the millimeter wave point cloud into a front view and a bird's-eye view; Step 3: Use the feature extraction module to extract features from the front view and bird's-eye view of the camera, the front view and bird's-eye view of the lidar, and the bird's-eye view of the millimeter-wave radar to obtain the camera front view features. With bird's eye view feature , LiDAR front view features With bird's eye view feature , millimeter wave radar bird's eye view features ; Step 4: Fusion processing is performed on the front view and bird's-eye view features of different sensors to obtain semantic information and spatial position information; Step 5: Fusion features of the front view obtained after fusion in step 4 , Bird's-eye view fusion features and features containing semantic and spatial information , and fuse their respective multi-scale features to obtain , , , and then transmit these three fused multi-scale features to the detection head for data inference; in the data inference stage, the detection head will sequentially , , Predict the confidence level and category information of traffic participants’ positions; Step 4 specifically includes the following steps: Step 401: Use the front view fusion module to fuse the camera front view features Front view features with LiDAR , obtain the forward-looking fusion features containing dense semantic information ; Step 402: Use the bird's eye view fusion module to fuse the camera's bird's eye view features , LiDAR bird’s eye view features Bird's-eye view feature with millimeter wave radar , obtain the bird's-eye view fusion features containing spatial location information ; Step 403: Use the attention fusion module to fuse the front-view fusion features Merge features with bird's eye view , adaptively assign weight coefficients of semantic information and spatial information to obtain features containing both semantic and spatial information ; Step 404: upsample the features of each unimodal branch, and use the front view fusion module, the bird's eye view fusion module and the attention fusion module to fusion the upsampled unimodal features. , , , , Fusion is performed to generate multi-scale fusion features with scales of (128, 256, 512, 1024).

2. The method for target detection based on multi-view multi-modal information fusion according to claim 1, characterized in that: In step 401, obtaining the front view features of the camera and the laser radar includes the following steps: first, the front view features of the camera and the laser radar are spliced ​​to obtain a fusion feature ; Secondly, normalize and convolve the spliced ​​features to obtain the features ; The processed features Add the camera's front view feature to get the front view fusion feature .

3. The method for target detection based on multi-view multi-modal information fusion according to claim 2, characterized in that: In step 402, obtaining the fusion features of the bird's-eye view of the camera, the laser radar and the millimeter-wave radar includes the following steps: first, the bird's-eye view features of the camera, the laser radar and the millimeter-wave radar are spliced ​​to obtain the fusion features ; Secondly, normalize and convolve the spliced ​​features to obtain the features ; Finally, the processed features Add the features of the laser radar to the bird's-eye view to obtain the fused bird's-eye view features. .

4. The method for target detection based on multi-view multi-modal information fusion according to claim 3, characterized in that: In step 403, obtain and The fused features include the following steps: First, the fused and Each is transmitted into the convolution layer and the activation function layer to obtain the features and Secondly, and Features are spliced ​​to obtain features , and the features Through the Softmax layer, the normalization operation is performed to obtain the features ; Again, the feature and , Multiply to get the feature and Finally, the feature and Splice and get .

5. The method for target detection based on multi-view multi-modal information fusion according to claim 1, characterized in that: In step 2, the conversion of the camera's front view to a bird's-eye view includes the following steps: using the camera to batch collect front views; determining the internal and external parameters of the camera, including focal length, principal point position, and camera position and posture, and deriving the inverse perspective change matrix M through a series of parameters; selecting the ground corresponding to the front view as the reference plane; and converting the front view collected by the camera into a bird's-eye view through the inverse perspective change matrix M.

6. The method for target detection based on multi-view multi-modal information fusion according to claim 1, characterized in that: In step 2, the conversion of the lidar point cloud data into a front view and a bird's-eye view includes the following steps: using the lidar → camera → image coordinate system transformation matrix to map the lidar point cloud to the image, and assigning color to the image point cloud according to the depth information to generate a front view of the laser point cloud; setting the z coordinate of the point (x, y, z) projected in the front view to 0 and projecting it onto the bird's-eye view to generate a laser point cloud bird's-eye view.

7. The method for target detection based on multi-view multi-modal information fusion according to claim 1, characterized in that: In step 2, the conversion of millimeter-wave radar point cloud data into a bird's-eye view includes the following steps: filtering out the point cloud mapped to the image through the millimeter-wave radar→camera→image coordinate system transformation matrix, mapping these points to the bird's-eye view according to the coordinate values ​​(x, y), and generating a millimeter-wave point cloud bird's-eye view.

8. The method for target detection based on multi-view multi-modal information fusion according to claim 1, characterized in that: In step 3, feature extraction of the front view and bird's-eye view features of the camera, lidar and millimeter-wave radar includes the following steps: transmitting the front view and bird's-eye view features of each sensor into the ConvNext module, performing convolution and normalization operations on the features, and finally performing upsampling operations on the processed features three times to obtain feature maps of four different scales.

9. The method for target detection based on multi-view multi-modal information fusion according to claim 1, characterized in that: During the model training phase, the multi-scale fusion features are passed into the detection head to generate prediction values. The deviation between the prediction values ​​and the true values ​​is calculated through the loss function, and backpropagation is performed to update the network weights, thereby gradually improving the detection performance of the model.

Images