Laser point cloud and camera multi-modal feature fusion target sensing method based on Mama

Abstract

The invention discloses a Mama-based laser point cloud and camera multi-modal feature fusion target sensing method, and relates to the technical field of machine vision, and the method comprises the following steps: carrying out the feature fusion of features extracted from a camera and features extracted from a laser radar point cloud, mixing the time domain frame information with the current frame information, and carrying out the fusion of the time domain frame information and the current frame information; the Bev features of the laser radar data and the Bev features of the camera data are fused through a multiplication operation and splicing method, the features are reordered through combination of convolution and a VMama method to achieve serialization, and then the features are extracted through Mama; and a channel attention module SE-Block is combined to realize extraction of target features. According to the invention, the object information is increased by mixing the time domain information and fusing the multi-view camera image and the laser radar data, the feature description capability of the object is enhanced through the TemporalMerage module and the Mama Encoder module, the detection capability of the model to the target is improved, and the detection rate of the target can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a target perception method based on Mamba-based laser point cloud and camera multimodal feature fusion. Background Art

[0002] Autonomous driving technology aims to enable vehicles to drive safely and efficiently without human intervention. To achieve this goal, vehicles need to accurately perceive the surrounding environment, including identifying roads, traffic signs, other vehicles, pedestrians, etc. Therefore, reliable environmental perception technology is the key foundation of autonomous driving.

[0003] Camera is one of the commonly used sensors in autonomous driving systems. It can obtain visual image information around the vehicle, has high resolution and rich color information, and can identify the shape, color, texture and other features of objects. For example, the camera can identify the color and shape of traffic signs to determine traffic rules; it can also identify the appearance and movements of pedestrians to determine their behavioral intentions.

[0004] LiDAR can measure the distance, direction, speed and other information of an object by emitting laser beams and receiving reflected signals, generating high-precision 3D point cloud data. It is not affected by lighting conditions and can work normally at night or in bad weather. It plays an important role in detecting obstacles and identifying road boundaries.

[0005] A single sensor cannot meet all the requirements of autonomous driving for environmental perception. Although cameras can provide rich visual information, their performance will be greatly affected in the case of insufficient light, bad weather or occlusion; although lidar has high-precision three-dimensional measurement capabilities, its point cloud data is relatively sparse and lacks color and texture information. Therefore, fusing the information of cameras and lidar can give full play to the advantages of both and improve the accuracy and reliability of environmental perception. Summary of the invention

[0006] The purpose of the present invention is to provide a target perception method based on Mamba's laser point cloud and camera multimodal feature fusion to overcome the above-mentioned defects in the prior art.

[0007] The target perception method based on Mamba laser point cloud and camera multimodal feature fusion includes the following steps:

[0008] S1: Send the features extracted from the camera and the features extracted from the lidar point cloud to the Fuse Module for feature fusion, and then input the fused features into the 3D object detection head module to detect the vehicle's surrounding environment;

[0009] S2, mixing the time domain frame information with the current frame information to facilitate the subsequent feature extraction;

[0010] S3, the Bev features of the lidar data and the Bev features of the camera data are multiplied and concatenated to achieve fusion of the two types of data;

[0011] S4, combining convolution and VMamba method to reorder the features to achieve serialization, then Mamba extracts the features, and then combines the channel attention module SE-Block to extract the target features.

[0012] Preferably, in the multi-view camera feature extraction branch in step S1, the Image Backbone module and the 2D-3D prediction module are passed through, wherein the Image Backbone module can be the backbone network in VMamba or the backbone network of ResNet, and the 2D-3D prediction module predicts the 3D information of the object from the 2D information in the image.

[0013] Preferably, in the lidar point cloud feature extraction branch in step S1, voxelization is first performed, followed by a 3D Backbone module, wherein the 3D Backbone module can be a backbone network in VMamba or a backbone network in ResNet, and the 2D-3D prediction module predicts the 3D information of the object from the 2D information in the image.

[0014] Preferably, the specific mixing method in step S2 is as shown in formula (1):

[0015] F C =F3(F1(h t ,h t-1 ),F2(h t ,h t-1 ))(1)

[0016] In formula (1), h represents the Bev feature of the laser point cloud or camera; F1 represents the direct addition of two features; F2 and F3 represent the concatenation of two features.

[0017] 5. The target perception method based on Mamba laser point cloud and camera multimodal feature fusion according to claim 1 is characterized in that the fusion method in step S3 is as shown in formula (2):

[0018] G C =G3(G1(C bev ,L bev ),G2(C bev ,L bev ))(2)

[0019] In formula (2), C represents the Bev feature of the camera data after time domain fusion, L represents the Bev feature of the laser point cloud data after time domain fusion; G1 represents the direct addition of the two features; G2 and G3 represent the concatenation of the two features.

[0020] Preferably, the state space model of the Mamba algorithm in step S4 is as shown in formula (3) and formula (4):

[0021] h ′ (t)=Ah(t)+Bu(t)(3)

[0022] y(t)=Ch(t)+Du(t)(4)

[0023] In formulas (3) and (4), h is the hidden state, y is the response state, and A∈R NxN , B∈R Nx1 , C∈R 1xN , D∈R 1 Represents the learnable weight coefficient.

[0024] The beneficial effects achieved by the present invention are:

[0025] The Mamba Encoder module of this application is a feature extraction module designed in combination with the VMamba method. This module first extracts feature information through the convolution method, and uses the state space model SS2D in VMamba to extract features. Since the advantage of Mamba is to process serialized features, SS2D serializes the features, and finally passes through the channel attention module SE-Block to enhance the model's ability to extract and utilize important features, thereby improving the performance of the model;

[0026] By mixing time domain information and fusing multi-view camera images with lidar data, object information is added. The TemporalMerage module and Mamba Encoder module are used to enhance the feature description capability of the object, improve the model's target detection capability, and effectively improve the target detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is the network structure diagram of the fusion of laser point cloud and multi-frame camera;

[0028] Figure 2 This is the time domain feature fusion network structure diagram of Bev feature;

[0029] Figure 3 This is the structure diagram of the fusion of laser point cloud and multi-frame camera Bev features;

[0030] Figure 4 This is the structure diagram of Mamba Encoder;

[0031] Figure 5 Figure 2 is the structure diagram of the Squeeze-and-Excitation (SE) module used. DETAILED DESCRIPTION

[0032] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the concept and technical solution of the present invention.

[0033] The present invention is a target perception method based on Mamba's laser point cloud and camera multimodal feature fusion. By aligning and fusing the multi-view images of 6 cameras in the front, back, left and right of the vehicle with the laser radar data, the vehicle's surrounding environment can be perceived, which is convenient for improving the detection effect of obstacles.

[0034] The Mamba algorithm has shown its impressive potential in multiple downstream tasks. In order to improve the global detection capability and time domain feature fusion capability of the algorithm, the technical solution adopted by the present invention is as follows:

[0035] The laser radar data of the present invention is combined with the multi-view camera data in combination with the Mamba algorithm in the time domain and space dimension to realize the perception of the environment around the vehicle.

[0036] Figure 1 It is a network structure diagram of the present invention. The network includes multi-view camera images and laser radar point cloud data. Features are extracted through the backbone network and then converted into Bev features. Bev features are fused with time domain information through the Mamba module for feature extraction. Then the camera and laser radar fusion module is used to fuse the two types of features and then sent to the 3D ObjectHead module to realize the perception of the vehicle's surrounding environment.

[0037] The specific analysis is as follows:

[0038] 1. Overall network structure

[0039] The present invention selects the laser radar point cloud and multi-view camera data for fusion because the laser radar point cloud is not affected by lighting conditions and has little effect in bad weather, but it has no color information and cannot obtain rich texture information of objects, which is the strength of camera images. The weakness of camera images is that they are easily affected by lighting and bad weather. Fusion of the two data can complement the advantages of the two.

[0040] The overall network of the present invention has two branches, one is the multi-view camera feature extraction branch, and the other is the lidar point cloud feature extraction branch. After the two branches extract features respectively, they will enter the fusion module (Fuse Module) for fusion, and then the fused features will be input into the 3D object detection head module to detect the vehicle's surrounding environment.

[0041] In the multi-view camera feature extraction branch, it passes through the Image Backbone module and the 2D-3D prediction module, wherein the Image Backbone module can be the backbone network in VMamba or the backbone network of ResNet; the 2D-3D prediction module mainly predicts the 3D information of the object from the 2D information in the image; secondly, it passes through the temporal feature (TemporalMerge) mixing module and then is sent to the Mamba Encoder module.

[0042] In the laser radar point cloud feature extraction branch, it first passes through voxelization, then the 3D Backbone module, and then enters the temporal feature (Temporal Merge) hybrid module. The 3D Backbone module can be the backbone network in VMamba or the backbone network of ResNet, etc.; the 2D-3D prediction module mainly predicts the 3D information of the object from the 2D information in the image; secondly, it passes through the temporal feature (Temporal Merge) hybrid module and then enters the Mamba Encoder module.

[0043] The features extracted from the camera and the features extracted from the lidar point cloud are sent to the Fuse Module for feature fusion, and finally sent to the detection head to complete the detection.

[0044] 2. Temporal Merge Mixing Module

[0045] Figure 1 The Temporal Merge module in the image mixes the temporal frame information with the current frame information to facilitate the subsequent feature extraction.

[0046] Figure 2 is its specific mixing method, which can be expressed using formula (1):

[0047] F C =F3(F1(h t ,h t-1 ),F2(h t ,h t-1 ))(1)

[0048] In formula (1), h represents the Bev feature of the laser point cloud or camera; F1 represents the direct addition of two features, which helps the model learn the complementary relationship between the two features; F2 and F3 are the concatenation of two features. The concatenation operation is used because it can completely retain the information of the two original features, making it easier for the model to use the two features separately. However, when concatenating the features obtained from F1 and F2, there may be a problem of inconsistent dimensions, so the convolution operation is introduced inside the function F3 to make the concatenation dimensions consistent.

[0049] 3. Multimodal Fusion (Fuse Module)

[0050] The detection method obtains two feature maps through the camera branch and the laser radar branch. In order to improve the detection effect and realize the supplement of different information, the present invention uses the Fuse Module module to fuse the two types of information. The structure is as follows Figure 3 The fusion method is shown in formula (2):

[0051] G C =G3(G1(C bev ,L bev ),G2(C bev ,L bev ))(2)

[0052] In formula (2), C represents the Bev feature of camera data after time domain fusion, L represents the Bev feature of laser point cloud data after time domain fusion; G1 represents the direct addition of two features, which helps the model learn the complementary relationship between the two features; G2 and G3 represent the concatenation of two features. Figure 2 , Figure 3 It can be seen that when fusing features of different modalities, concatenation and cross product are used, which can fully preserve the information of the two original features and highlight the synergistic effect between the two features.

[0053] 4. Mamba Encoder Module

[0054] The Mamba algorithm mainly combines the state space model SSM and the multi-layer perceptron MLP. The state space model is shown in equations (3) and (4):

[0055] h ′ (t)=Ah(t)+Bu(t)(3)

[0056] y(t)=Ch(t)+Du(t)(4)

[0057] Where h is the hidden state, y is the response state, A∈R NxN , B∈R Nx1, C∈R 1xN , D∈R 1 Represents the learnable weight coefficient.

[0058] Since Mamba accepts serialized information, we split the features into different blocks in different directions, that is, rearrange the features, and then restore the order of the blocks after calculating the features.

[0059] Figure 4 The CBA module represents the combination of convolution, normalization, and activation functions; the DCN module refers to variable convolution, which can mine more informative features from the input features and enhance the model's expressiveness and generalization capabilities; the LN module is a layer normalization module; Conv is a convolution module; Act is an activation function; SS2D is a state space 2D module in Mamba, and the implementation details of SE-Block are as follows: Figure 5 As shown in the figure, it is possible to adaptively recalibrate the importance of channel features, enhance the model's ability to extract and utilize important features, and thus improve the performance of the model. It enables the model to better adapt to different data distributions and task requirements.

[0060] In summary, the present invention improves the ability to characterize target features and improves the accuracy of target detection by the model by extracting features from multi-view camera images and lidar data and fusing them with time domain information.

[0061] The above-described embodiments of the present invention do not constitute a limitation on the protection scope of the present invention. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A target perception method based on Mamba laser point cloud and camera multimodal feature fusion, characterized by: The following steps are involved: S1: Send the features extracted from the camera and the features extracted from the lidar point cloud to the Fuse Module for feature fusion, and then input the fused features into the 3D object detection head module to detect the vehicle's surrounding environment; S2, mixing the time domain frame information with the current frame information to facilitate the subsequent feature extraction; S3, the Bev features of the lidar data and the Bev features of the camera data are multiplied and concatenated to achieve fusion of the two types of data; S4, combining convolution and VMamba method to reorder the features to achieve serialization, then Mamba extracts the features, and then combines the channel attention module SE-Block to extract the target features.

2. The target perception method based on Mamba laser point cloud and camera multimodal feature fusion according to claim 1 is characterized by: In the multi-view camera feature extraction branch in step S1, the image backbone module and the 2D-3D prediction module are passed through, wherein the image backbone module can be the backbone network in VMamba or the backbone network of ResNet, and the 2D-3D prediction module predicts the 3D information of the object from the 2D information in the image.

3. The target perception method based on Mamba laser point cloud and camera multimodal feature fusion according to claim 1 is characterized by: In the lidar point cloud feature extraction branch in step S1, voxelization is first performed, followed by the 3D Backbone module, where the 3D Backbone module can be the backbone network in VMamba or the backbone network of ResNet, and the 2D-3D prediction module predicts the 3D information of the object from the 2D information in the image.

4. The target perception method based on Mamba laser point cloud and camera multimodal feature fusion according to claim 1 is characterized by: The specific mixing method in step S2 is shown in formula (1): F C =F3(F1(h t ,h t-1 ),F2(h t ,h t-1 )) (1) In formula (1), h represents the Bev feature of the laser point cloud or camera; F1 represents the direct addition of two features; F2 and F3 are two features spliced ​​together.

5. The target perception method based on Mamba laser point cloud and camera multimodal feature fusion according to claim 1 is characterized by: The fusion method in step S3 is shown in formula (2): G C =G3(G1(C bev ,L bev ),G2(C bev ,L bev )) (2) In formula (2), C represents the Bev feature of the camera data after time domain fusion, L represents the Bev feature of the laser point cloud data after time domain fusion; G1 represents the direct addition of the two features; G2 and G3 represent the concatenation of the two features.

6. The target perception method based on Mamba laser point cloud and camera multimodal feature fusion according to claim 1 is characterized by: The state space model of the Mamba algorithm in step S4 is shown in equations (3) and (4): h ′ (t)= Ah(t) + Bu(t) (3) y(t) = Ch(t) + Du(t) (4) In formulas (3) and (4), h is the hidden state, y is the response state, and A∈R NxN , B∈R Nx1 , C∈R 1xN , D∈R 1 Represents the learnable weight coefficient.

Images