Multi-sensor fusion perception method based on attention mechanism and ensemble learning

Abstract

The invention relates to a multi-sensor fusion perception method based on an attention mechanism and ensemble learning, and the method comprises the steps: respectively extracting camera image features and laser radar point cloud features of camera image data and laser radar point cloud data; obtaining a depth map corresponding to at least one laser radar point cloud feature, generating an image BEV feature map by combining the camera image features, and generating a point cloud BEV feature map according to the laser radar point cloud features; and performing enhanced feature processing and spatial feature processing on the image BEV feature map and the point cloud BEV feature map through an enhanced channel attention mechanism and a spatial attention mechanism to obtain an enhanced feature map and a spatial feature map, and fusing the enhanced feature map and the spatial feature map to generate a fused feature map to obtain a target detection result of the to-be-detected target. Therefore, the problems that in the prior art, complementary advantages between sensors cannot be fully utilized, depth information in an image is not accurate, and the accuracy of automatic driving perception and the adaptive capacity to the environment are greatly limited are solved.

Description

Technical Field

[0001] The present application relates to the field of transportation and electronic equipment technology, and in particular to a multi-sensor fusion perception method based on attention mechanism and integrated learning. Background Art

[0002] With the increasing popularity of autonomous vehicles and the spread of connected and intelligent transportation technologies, the need for effective and reliable autonomous driving perception solutions has become more and more urgent.

[0003] Traditional methods of autonomous driving perception have been widely used in intelligent transportation systems (ITS) and have shown good performance in some scenarios. 3D object detection based on bird's eye view (BEV) can transform multimodal features from different sensors into consistent BEV representations, effectively integrate geometric and semantic details, and reduce local misalignment between different features, which has significant potential compared to traditional methods.

[0004] Although traditional autonomous driving perception methods for ITS perform well in some scenarios, traditional methods for generating BEV views usually use relatively simple fusion strategies, such as directly concatenating point clouds and image feature vectors or using simple fully connected networks, which cannot fully exploit the complementary advantages of these sensors.

[0005] Secondly, existing methods often find it difficult to solve the problem that camera-based perception methods cannot accurately obtain image depth information, which limits the accuracy of autonomous driving perception and its adaptability in complex traffic conditions.

[0006] In summary, existing technologies cannot fully utilize the complementary advantages between sensors, and the depth information in the image is inaccurate, which greatly limits the accuracy of autonomous driving perception and environmental adaptability, and needs to be urgently resolved. Summary of the invention

[0007] The present application provides a multi-sensor fusion perception method based on attention mechanism and integrated learning to solve the problems that the existing technology cannot fully utilize the complementary advantages between sensors, and the depth information in the image is inaccurate, which greatly limits the accuracy of autonomous driving perception and environmental adaptability.

[0008] The first aspect of the present application provides a multi-sensor fusion perception method based on attention mechanism and integrated learning, including the following steps: obtaining camera image data and laser radar point cloud data of a target to be measured, and extracting at least one camera image feature and at least one laser radar point cloud feature of the camera image data and the laser radar point cloud data respectively; obtaining a depth map corresponding to the at least one laser radar point cloud feature, and generating an image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map, and generating a point cloud BEV feature map corresponding to the laser radar point cloud data according to the laser radar point cloud feature; performing enhanced feature processing on the image BEV feature map through a preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and performing spatial feature processing on the point cloud BEV feature map using a preset spatial attention mechanism to obtain a spatial processing feature map, and fusing the enhanced processing feature map and the spatial processing feature map according to a preset integrated learning strategy to generate a fused feature map, and sensing the target detection result of the target to be measured through the fused feature map.

[0009] Optionally, in one embodiment of the present application, the acquiring of a depth map corresponding to the at least one lidar point cloud feature and the generating of an image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map include: arranging the lidar point cloud features in the channel dimension and performing a linear transformation on the arranged lidar point cloud features to obtain a query vector corresponding to the lidar point cloud features; performing a maximum gradient calculation in multiple target directions on the point cloud Z axis of the lidar point cloud features to obtain a calculation result for each of the multiple target directions, and concatenating the calculation result and the lidar point cloud features to obtain the depth map; based on the query vector and a preset position encoding strategy, Generate an initial image feature query vector, and perform reverse flattening and linear transformation operations on the initial image feature query vector to obtain a target query vector; based on the target query vector and the depth map, extract the depth distribution information of the camera image feature, and weight the camera image feature by the depth distribution information to obtain a weighted camera image feature; flatten the weighted camera image feature to obtain geometric information corresponding to the weighted camera image feature, and map the feature points of the weighted camera image feature to a preset bird's-eye view coordinate system by the geometric information; perform a preset pooling operation on the feature points to aggregate the feature points into corresponding bird's-eye view grid cells to generate an image BEV feature map corresponding to the camera image data.

[0010] Optionally, in one embodiment of the present application, the enhanced feature processing of the image BEV feature map by a preset enhanced channel attention mechanism to obtain enhanced processing features includes: calculating the global average pool corresponding to the image BEV feature map, and obtaining the average feature map corresponding to the image BEV feature map according to the global average pool; obtaining the target feature of each feature map channel in the image BEV feature map to determine the maximized feature map corresponding to the image BEV feature map according to the target feature of each feature map channel; generating an initial convolution average feature map and an initial convolution maximized feature map based on the average feature map, the maximized feature map, the ReLU activation function and a preset initial convolution weight matrix; obtaining a secondary convolution average feature map and a secondary convolution maximized feature map according to the initial convolution average feature map, the initial convolution maximized feature map, the ReLU activation function and a preset secondary convolution weight matrix; performing a sum operation on the secondary convolution average feature map and the secondary convolution maximized feature map to obtain a combined feature map, and calculating the channel significance score corresponding to the combined feature map;

[0011] An element-wise multiplication operation is performed on the image BEV feature map and the channel significance score to obtain the enhanced processing feature.

[0012] Optionally, in one embodiment of the present application, the point cloud BEV feature map is spatially processed using a preset spatial attention mechanism to obtain a spatially processed feature map, including: performing a global average pooling operation on each spatial position in all channels of the point cloud BEV feature map to obtain an average pooled feature map corresponding to the point cloud BEV feature map; extracting the channel maximum feature of each spatial position corresponding to the point cloud BEV feature map, and determining the channel maximum feature map corresponding to the point cloud BEV feature map through the channel maximum feature, and performing a channel dimension connection operation on the average pooled feature map and the channel maximum feature map to obtain a connection feature map; performing convolution processing on the connection feature map, and generating a spatial attention map according to a preset sigmoid function and the connection feature map after convolution processing, and performing an element-wise multiplication operation on the spatial attention map and the point cloud BEV feature map to generate the spatial processing feature map.

[0013] The second aspect of the present application provides a multi-sensor fusion perception device based on an attention mechanism and integrated learning, including: an extraction module, used to obtain camera image data and laser radar point cloud data of a target to be measured, and respectively extract at least one camera image feature and at least one laser radar point cloud feature of the camera image data and the laser radar point cloud data; a generation module, used to obtain a depth map corresponding to the at least one laser radar point cloud feature, and based on the camera image feature and the depth map, generate an image BEV feature map corresponding to the camera image data, and generate a point cloud BEV feature map corresponding to the laser radar point cloud data according to the laser radar point cloud feature; a fusion module, used to perform enhanced feature processing on the image BEV feature map through a preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and perform spatial feature processing on the point cloud BEV feature map using a preset spatial attention mechanism to obtain a spatial processing feature map, and fuse the enhanced processing feature map and the spatial processing feature map according to a preset integrated learning strategy to generate a fused feature map, and perceive the target detection result of the target to be measured through the fused feature map.

[0014] Optionally, in one embodiment of the present application, the generation module includes: an arrangement unit, used to arrange the laser radar point cloud features in the channel dimension, and perform a linear transformation on the arranged laser radar point cloud features to obtain a query vector corresponding to the laser radar point cloud features; a first calculation unit, used to perform a maximum gradient calculation in multiple target directions on the point cloud Z axis of the laser radar point cloud features, obtain a calculation result for each target direction in the multiple target directions, and splice the calculation result and the laser radar point cloud features to obtain the depth map; a transformation unit, used to generate an initial image feature query vector based on the query vector and a preset position encoding strategy, and perform a linear transformation on the initial image feature query vector. The camera image data is subjected to inverse flattening and linear transformation operations to obtain a target query vector; a weighting unit is used to extract the depth distribution information of the camera image feature based on the target query vector and the depth map, and weight the camera image feature by the depth distribution information to obtain a weighted camera image feature; a flattening unit is used to flatten the weighted camera image feature to obtain geometric information corresponding to the weighted camera image feature, and map the feature points of the weighted camera image feature to a preset bird's-eye view coordinate system by the geometric information; a pooling unit is used to perform a preset pooling operation on the feature points to aggregate the feature points into corresponding bird's-eye view grid cells to generate an image BEV feature map corresponding to the camera image data.

[0015] Optionally, in one embodiment of the present application, the fusion module includes: a second calculation unit, used to calculate the global average pool corresponding to the image BEV feature map, and obtain the average feature map corresponding to the image BEV feature map according to the global average pool; an acquisition unit, used to acquire the target feature of each feature map channel in the image BEV feature map, so as to determine the maximized feature map corresponding to the image BEV feature map according to the target feature of each feature map channel; a first convolution unit, used to generate an initial convolution average feature map based on the average feature map, the maximized feature map, the ReLU activation function and the preset initial convolution weight matrix. A feature map and an initial convolution maximization feature map; a second convolution unit, used to obtain a secondary convolution average feature map and a secondary convolution maximization feature map according to the initial convolution average feature map, the initial convolution maximization feature map, the ReLU activation function and a preset secondary convolution weight matrix; a summation unit, used to sum the secondary convolution average feature map and the secondary convolution maximization feature map to obtain a combined feature map, and calculate the channel significance score corresponding to the combined feature map; a first element multiplication unit, used to perform an element multiplication operation on the image BEV feature map and the channel significance score to obtain the enhanced processing feature.

[0016] Optionally, in one embodiment of the present application, the fusion module also includes: a processing unit, used to perform a global average pooling operation on each spatial position in all channels of the point cloud BEV feature map to obtain an average pooling feature map corresponding to the point cloud BEV feature map; a connection unit, used to extract the channel maximum feature of each spatial position corresponding to the point cloud BEV feature map, and determine the channel maximum feature map corresponding to the point cloud BEV feature map through the channel maximum feature, and perform a channel dimension connection operation on the average pooling feature map and the channel maximum feature map to obtain a connection feature map; a second element multiplication unit, used to perform convolution processing on the connection feature map, and generate a spatial attention map according to a preset sigmoid function and the connection feature map after convolution processing, and perform element multiplication operation on the spatial attention map and the point cloud BEV feature map to generate the spatial processing feature map.

[0017] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-sensor fusion perception method based on attention mechanism and integrated learning as described in the above embodiments.

[0018] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned multi-sensor fusion perception method based on attention mechanism and integrated learning.

[0019] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned multi-sensor fusion perception method based on attention mechanism and integrated learning.

[0020] Therefore, the embodiments of the present application have the following beneficial effects:

[0021] The embodiments of the present application can obtain the camera image data and the laser radar point cloud data of the target to be measured, and extract at least one camera image feature and at least one laser radar point cloud feature of the camera image data and the laser radar point cloud data respectively; obtain the depth map corresponding to the at least one laser radar point cloud feature, and generate the image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map, and generate the point cloud BEV feature map corresponding to the laser radar point cloud data according to the laser radar point cloud feature; perform enhanced feature processing on the image BEV feature map through the preset enhanced channel attention mechanism to obtain the enhanced processing feature map, and perform spatial feature processing on the point cloud BEV feature map using the preset spatial attention mechanism to obtain the spatial processing feature map, and fuse the enhanced processing feature map and the spatial processing feature map according to the preset integrated learning strategy to generate a fused feature map, and perceive the target detection result of the target to be measured through the fused feature map. The present application uses point cloud supervision to extract depth information from the image, and uses attention mechanism and integrated learning to solve the problem of inaccurate depth information in the image and improve the integration accuracy of point cloud and image information. This solves the problem that the existing technology cannot fully utilize the complementary advantages of sensors, and the depth information in the image is inaccurate, which greatly limits the accuracy of autonomous driving perception and environmental adaptability.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a multi-sensor fusion perception method based on attention mechanism and integrated learning provided according to an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the execution logic of an attention-based integrated learning fusion strategy provided according to an embodiment of the present application;

[0026] Figure 3A schematic diagram of the execution logic of a channel and space attention mechanism fusion strategy provided according to an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a logical architecture of an enhanced channel attention mechanism provided according to an embodiment of the present application;

[0028] Figure 5 A schematic diagram of a logical architecture of a spatial attention mechanism provided according to an embodiment of the present application;

[0029] Figure 6 This is an example diagram of a multi-sensor fusion perception device based on attention mechanism and integrated learning according to an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0031] Among them, 10-multi-sensor fusion perception device based on attention mechanism and integrated learning; 100-extraction module, 200-generation module, 300-fusion module; 701-memory, 702-processor, 703-communication interface. DETAILED DESCRIPTION

[0032] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0033] The following describes a multi-sensor fusion perception method based on attention mechanism and integrated learning of an embodiment of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, the present application provides a multi-sensor fusion perception method based on attention mechanism and integrated learning, in which the camera image data and the laser radar point cloud data of the target to be measured are obtained, and at least one camera image feature and at least one laser radar point cloud feature of the camera image data and the laser radar point cloud data are extracted respectively; a depth map corresponding to at least one laser radar point cloud feature is obtained, and an image BEV feature map corresponding to the camera image data is generated based on the camera image feature and the depth map, and a point cloud BEV feature map corresponding to the laser radar point cloud data is generated according to the laser radar point cloud feature; the image BEV feature map is enhanced by a preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and the point cloud BEV feature map is spatially processed by a preset spatial attention mechanism to obtain a spatial processing feature map, and the enhanced processing feature map and the spatial processing feature map are fused according to a preset integrated learning strategy to generate a fused feature map, and the target detection result of the target to be measured is perceived by the fused feature map. This application uses point cloud supervision to extract depth information from images, and uses attention mechanisms and integrated learning to solve the problem of inaccurate depth information in images and improve the accuracy of integration of point cloud and image information. This solves the problem that the existing technology cannot fully utilize the complementary advantages between sensors, and the depth information in the image is inaccurate, which greatly limits the accuracy of autonomous driving perception and environmental adaptability.

[0034] Specifically, Figure 1 A flowchart of a multi-sensor fusion perception method based on attention mechanism and integrated learning provided in an embodiment of the present application.

[0035] like Figure 1 As shown, the multi-sensor fusion perception method based on attention mechanism and integrated learning includes the following steps:

[0036] In step S101, camera image data and lidar point cloud data of a target to be measured are obtained, and at least one camera image feature and at least one lidar point cloud feature of the camera image data and the lidar point cloud data are extracted respectively.

[0037] In step S102, a depth map corresponding to at least one lidar point cloud feature is obtained, and an image BEV feature map corresponding to the camera image data is generated based on the camera image features and the depth map, and a point cloud BEV feature map corresponding to the lidar point cloud data is generated according to the lidar point cloud features.

[0038] The embodiment of the present application first constructs an attention-based ensemble learning fusion (AEFusion) strategy, such as Figure 2 As shown in the figure, this strategy can improve the existing point cloud and image information merging methods by focusing on the fusion of multi-sensor data for 3D target detection, especially multi-view cameras and lidars. Each sensor input is processed by a specific modality encoder to extract multimodal features from the multimodal input through AEFusion; then, a new supervision module is used to optimize the generation of image depth information, which effectively transforms these features into a shared BEV space through view conversion, and fuses the unified BEV features through an integrated learning module based on an attention mechanism, thereby transforming these multimodal features into a coherent BEV representation that effectively integrates geometric and semantic details.

[0039] Optionally, in one embodiment of the present application, a depth map corresponding to at least one lidar point cloud feature is obtained, and based on the camera image feature and the depth map, an image BEV feature map corresponding to the camera image data is generated, including: arranging the lidar point cloud features in the channel dimension, and performing a linear transformation on the arranged lidar point cloud features to obtain a query vector corresponding to the lidar point cloud feature; performing a maximum gradient calculation on the point cloud Z axis of the lidar point cloud feature in multiple target directions to obtain a calculation result for each target direction in multiple target directions, and splicing the calculation result and the lidar point cloud feature to obtain a depth map; based on the query vector and a preset position encoding strategy, Generate an initial image feature query vector, and perform reverse flattening and linear transformation operations on the initial image feature query vector to obtain a target query vector; based on the target query vector and the depth map, extract the depth distribution information of the camera image feature, and weight the camera image feature by the depth distribution information to obtain a weighted camera image feature; flatten the weighted camera image feature to obtain the geometric information corresponding to the weighted camera image feature, and map the feature points of the weighted camera image feature to a preset bird's-eye view coordinate system through the geometric information; perform a preset pooling operation on the feature points to aggregate the feature points into corresponding bird's-eye view grid cells to generate an image BEV feature map corresponding to the camera image data.

[0040] In the actual implementation process, the embodiment of the present application can extract features from the image to obtain (B, C, H, W) two-dimensional image features (i.e., laser radar point cloud features), and arrange them in the channel dimension, and flatten them to obtain a tensor of shape (B, H*W, C), as shown in the following formula:

[0041] X flat =reshape(X,(B,H*W,C)) (1)

[0042] Among them, B is the batch size, C is the number of channels, H and W are the height and width of the feature map respectively.

[0043] Afterwards, the embodiment of the present application can convert the image features into a query vector Q through linear transformation, as shown in formula (2):

[0044] Q=X flat W+b(2)

[0045] Where W is a shape of (C,d model ), b is a weight matrix with shape d model The bias vector of .

[0046] Furthermore, the embodiment of the present application may use a position encoding formula, as shown in formula (3), to generate a position encoding for each (x, y) in the i-dimensional space:

[0047]

[0048] Among them, d model is the dimension of the embedding vector, i is the dimension index, and pos represents the position.

[0049] Then, embodiments of the present application may flatten the position encoding tensor into a shape (H×W, d model ), and then added to the query vector Q to obtain the initial image feature query vector Q:

[0050] PE flat =reshape(PE,(H×W,d model ))

[0051] Q encoded =Q+PE flat (4)

[0052] It should be noted that the embodiment of the present application is not a simple point cloud z-axis. At this time, the kernel can be first set to 7, and the convolution calculation form can be used to perform the maximum gradient calculation in 255 directions (i.e., multiple target directions) on the z-axis points, and the new parameters calculated in each direction are used as a dimension, which is spliced ​​with the original eigenvalues ​​(i.e., lidar point cloud features) to obtain a 255-dimensional depth feature, which is flattened and linearly transformed, and its dimension is converted into a shape compatible with the query vector Q as K and V; thereafter, the embodiment of the present application can query the z-axis points in the original point cloud image through the Attention mechanism, thereby updating the two-dimensional image detail information and achieving the purpose of supervising and enriching image features.

[0053] Furthermore, the embodiment of the present application can reversely flatten the initial image feature query vector Q and convert it back to (B, d model,H,W) shape, and after a layer of linear transformation, the shape is adjusted to (B,C,H,W) to obtain the target query vector, and then the BEV feature is generated through the LSS module. The specific process is as follows:

[0054] 1. Define linspace according to the internal and external parameters of the camera to obtain (x s ,y s ,z s ), and generate the BEV cone space through stack;

[0055] 2. Convert the number of image feature channels into the number of depth channels and feature channels through a 1×1 convolution layer, and output the depth distribution and feature representation of each pixel. By concatenating the depth information and image features and using the depth distribution to weight the feature map, the depth feature representation of each pixel is obtained. The depth shape is (B, C, D, H, W), which is adjusted to six dimensions (B, N, D, H, W, C);

[0056] 3. Perform a feature mapping operation in the pooling layer to map the feature into a preset BEV space. As shown in formula (5), in an embodiment of the present application, the pooling layer may flatten the input feature x from a shape (B, N, D, H, W, C) to a shape (N′, C), where N′=B×N×D×H×W, to obtain geometric information (N′, 3);

[0057] 4. Use geometric information to map feature points into a bird’s-eye view coordinate system and normalize the geometric positions into discrete grid coordinates;

[0058] 5. Assign the valid feature points to the grid cells of the bird's-eye view according to their geometric positions, and use the pooling operation to aggregate the feature points into the corresponding grid cells of the bird's-eye view to form the corresponding BEV feature map.

[0059] Y=Conv 1×1 (X)

[0060] P = softmax(Y[:,:D,:,:]

[0061] F=Y[:,D:D+C,:,:]

[0062]

[0063] Among them, the shape of Y is (B, D+C, H, W), and the first D channels of the output are processed by the softmax function to obtain the depth distribution of each pixel; the shape of P is (B, D, H, W)(B, D, H, W), which represents the probability distribution of each pixel at different depths; the shape of F is (B, C, H, W), which represents the feature channel; the shape of Z is (B, C, D, H, W), which represents the feature representation of each pixel at different depths.

[0064] Therefore, the embodiments of the present application use the depth map generated by the lidar as a supervision signal to guide the extraction of depth information in the process of converting image data into a BEV view; in addition, the embodiments of the present application can guide the generation of image depth information by using more accurate lidar depth information. The depth supervision strategy of the embodiments of the present application can use an attention mechanism to perform convolution calculations on the maximum gradient on the z-axis of the point cloud, thereby generating depth features, effectively solving the problem of inaccurate depth information in the image.

[0065] In step S103, the image BEV feature map is enhanced by the preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and the point cloud BEV feature map is spatially processed by the preset spatial attention mechanism to obtain a spatial processing feature map, and the enhanced processing feature map and the spatial processing feature map are fused according to the preset integrated learning strategy to generate a fused feature map, and the target detection result of the target to be measured is perceived through the fused feature map.

[0066] Furthermore, since it is difficult for the traditional method to fully maintain the integrity of information through the fully connected layer or convolution, the embodiments of the present application can be implemented as follows: Figure 3 The fusion strategy based on attention mechanism and ensemble learning shown in the figure utilizes channel attention mechanism and spatial attention mechanism to process the image feature map (i.e., image BEV feature map) and point cloud feature map (i.e., point cloud BEV feature map) respectively, thereby enhancing key area features while suppressing noise and background interference; thereafter, in the fusion stage, the embodiment of the present application can shift from simple connection to the use of ensemble learning concepts, and introduce an enhanced channel attention block that prioritizes spatial positioning, integrates the features of the image and point cloud, and outputs the final feature map (i.e., fused feature map).

[0067] Therefore, the embodiments of the present application improve the accuracy and efficiency of multi-sensor data fusion by utilizing point cloud supervision to extract depth information from images and combining integrated learning with attention mechanisms, thereby improving the existing point cloud and image information merging methods, ensuring accurate feature enhancement and effective information synthesis, and improving the integration of point cloud and image information.

[0068] Optionally, in an embodiment of the present application, the image BEV feature map is enhanced by a preset enhanced channel attention mechanism to obtain enhanced processed features, including: calculating the global average pooling corresponding to the image BEV feature map, and obtaining the average feature map corresponding to the image BEV feature map according to the global average pooling; obtaining the target features of each feature map channel in the image BEV feature map, so as to determine the maximized feature map corresponding to the image BEV feature map according to the target features of each feature map channel; generating an initial convolutional average feature map and an initial convolutional maximized feature map based on the average feature map, the maximized feature map, the ReLU activation function, and a preset initial convolutional weight matrix; obtaining a secondary convolutional average feature map and a secondary convolutional maximized feature map according to the initial convolutional average feature map, the initial convolutional maximized feature map, the ReLU activation function, and a preset secondary convolutional weight matrix; performing a summation operation on the secondary convolutional average feature map and the secondary convolutional maximized feature map to obtain a combined feature map, and calculating the channel saliency score corresponding to the combined feature map; performing an element-wise multiplication operation on the image BEV feature map and the channel saliency score to obtain the enhanced processed features.

[0069] Those skilled in the art should understand that in the fusion strategy, the prediction results from a basic model are usually represented by a single channel of the feature map. Generally, these basic models have different prediction accuracies, so the focus of channel attention is on the most valuable channels in the input.

[0070] Specifically, as Figure 4 shown, the embodiment of the present application calculates the average value of the features across the spatial dimensions through global average pooling, so as to obtain a feature vector (i.e., the average feature map) that retains the global front and back information; for the input feature map (i.e., the image BEV feature map) X ∈ R C×H×W , where C is the number of channels, H is the height, and W is the width, the calculation method of global average pooling is as follows:

[0071]

[0072] where, X avg ∈R C×1×1 is the average value of each channel, c (0 ≤ c < C) is the index of the channel, i (0 ≤ i < H) is the index of the height dimension, and j (0 ≤ j < W) is the index of the width dimension.

[0073] In addition, the global max pooling is used to identify the maximum value in each channel of the feature map, capturing the most significant features (i.e., the maximized feature map) of each feature map channel:

[0074] X max (c) = max H max W X(c, i, j) (7)

[0075] Among them, X max (c) represents the maximum value of each channel.

[0076] Afterwards, the embodiment of the present application may apply a 1×1 convolutional layer to the averaged and maximized feature maps to perform channel feature conversion, as shown in the following formula:

[0077] X conv1，avg =ReLU(W1X avg ) (8)

[0078] X conv1,max =ReLU(W1X max ) (9)

[0079] X conv2，avg =ReLU(W2X conv1，avg ) (10)

[0080] X conv2,max =ReLU(W2X conv1,max ) (11)

[0081] in, represents the weight matrix of the first 1×1 convolution (i.e., the initial convolution weight matrix), r is the channel attenuation ratio, It is the weight matrix of the second 1×1 convolution (i.e., the quadratic convolution weight matrix).

[0082] It should be noted that the conversion from fully connected layers to 1×1 convolutional layers has multiple advantages. These layers only focus on the interactions between channels, significantly reducing the parameter count and computational complexity, thereby maintaining the spatial structure of the feature map. By utilizing the parallel processing advantages of convolution, the embodiments of the present application effectively improve the computational efficiency of large feature maps, etc.

[0083] In addition, 1×1 convolution supports rich feature fusion and cross-channel transformation. Using two 1×1 convolutions with a ReLU activation function in between introduces necessary nonlinearity, which helps complex feature learning. The initial convolution layer reduces the number of channels to reduce computational requirements, while the subsequent convolution layer restores the original channel count, thereby achieving detailed and efficient feature processing. This setting not only enhances nonlinear feature transformation, but also improves the overall feature integration and representation in the model.

[0084] After two convolutional layers, the converted feature maps are summed to integrate the features processed by different pooling strategies (i.e., combined feature maps):

[0085] X out =X conv2,avg +X conv2,max (12)

[0086] Sigmoid activation σ is applied to the combined feature map to generate a channel saliency score:

[0087] M channel =σ(X out ) (13)

[0088] Finally, the weighted feature map (i.e., enhanced processing feature) is obtained by multiplying the channel weighted mask and the feature map element by element. The calculation process is shown in the following formula:

[0089] X weighted =X⊙M channel (14)

[0090] Here, ⊙ represents element-wise multiplication.

[0091] Therefore, the embodiments of the present application can enhance the features of key areas in the image BEV feature map through the enhanced channel attention machine, while suppressing noise and background interference, thereby improving the model's response to important features.

[0092] Optionally, in one embodiment of the present application, a preset spatial attention mechanism is used to perform spatial feature processing on the point cloud BEV feature map to obtain a spatially processed feature map, including: performing a global average pooling operation on each spatial position in all channels of the point cloud BEV feature map to obtain an average pooled feature map corresponding to the point cloud BEV feature map; extracting the channel maximum feature of each spatial position corresponding to the point cloud BEV feature map, and determining the channel maximum feature map corresponding to the point cloud BEV feature map through the channel maximum feature, and performing a channel dimension connection operation on the average pooled feature map and the channel maximum feature map to obtain a connection feature map; performing convolution processing on the connection feature map, and generating a spatial attention map according to a preset sigmoid function and the connection feature map after convolution processing, and performing element-wise multiplication operation on the spatial attention map and the point cloud BEV feature map to generate a spatial processing feature map.

[0093] It can be understood that point cloud data is composed of a series of points distributed in three-dimensional space and lacks the fixed spatial arrangement characteristics of image pixels. The spatial attention mechanism used in the embodiment of the present application is good at processing such unstructured data. By focusing on key spatial positions in the point cloud, valuable geographic and shape information can be extracted.

[0094] Since there is an inherent inter-region correlation in the context of the image, such as Figure 5 As shown, the embodiment of the present application can be given a point cloud BEV feature map X∈R C×H×W , and the global average pooling operation is applied to each spatial position (i, j) on all channels c of the point cloud BEV feature map to obtain the average pooled feature map, as shown in the following formula:

[0095]

[0096] The application of maximum pooling is similar to the above-mentioned average pooling, so the embodiment of the present application can extract the maximum value of all channels at each spatial position (i.e., the channel maximum feature map):

[0097] X max (i, j) = max C X(c,i,j) (16)

[0098] Afterwards, the embodiment of the present application may connect the outputs of the average pool and the maximum pool along the channel dimension to obtain a connection feature map:

[0099] X cat =concat(X avg , X max ) (17)

[0100] Then, the connected feature map is processed by a convolution operation:

[0101] X conv =W*X cat (18)

[0102] Finally, a sigmoid-type function is applied to the convolution output to generate a spatial attention map, which is weighted over the original input by element-wise multiplication to obtain a spatially processed feature map:

[0103] M spatial =σ(X conv ) (19)

[0104] X weighted =X☉M spatial (20)

[0105] Therefore, the embodiments of the present application target the unstructured characteristics of point cloud data, and the spatial attention mechanism can identify and focus on key spatial locations, thereby extracting valuable geographic and shape information and improving the processing capabilities of unstructured data.

[0106] In summary, the embodiments of the present application optimize the generation of image depth information by extracting features from multimodal inputs, and effectively convert these features into a shared BEV space using view transformation, and fuse unified BEV features through an attention-based ensemble learning strategy. Therefore, the embodiments of the present application optimize the generation of depth information while generating the BEV space, and reduce local misalignment between different features; in addition, experiments on the nuScenes dataset and self-collected datasets show that the embodiments of the present application are suitable for full-range docking perception and show good generalization capabilities at different distances.

[0107] According to the multi-sensor fusion perception method based on attention mechanism and integrated learning proposed in the embodiment of the present application, by acquiring the camera image data and the laser radar point cloud data of the target to be measured, and extracting at least one camera image feature and at least one laser radar point cloud feature of the camera image data and the laser radar point cloud data respectively; acquiring the depth map corresponding to at least one laser radar point cloud feature, and generating the image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map, and generating the point cloud BEV feature map corresponding to the laser radar point cloud data according to the laser radar point cloud feature; performing enhanced feature processing on the image BEV feature map through the preset enhanced channel attention mechanism to obtain the enhanced processing feature map, and performing spatial feature processing on the point cloud BEV feature map using the preset spatial attention mechanism to obtain the spatial processing feature map, and fusing the enhanced processing feature map and the spatial processing feature map according to the preset integrated learning strategy to generate a fused feature map, and perceiving the target detection result of the target to be measured through the fused feature map. The present application uses point cloud supervision to extract depth information from the image, and uses attention mechanism and integrated learning to solve the problem of inaccurate depth information in the image and improve the integration accuracy of point cloud and image information.

[0108] Secondly, the multi-sensor fusion perception device based on attention mechanism and integrated learning proposed in accordance with the embodiment of the present application is described with reference to the accompanying drawings.

[0109] Figure 6 It is a block diagram of a multi-sensor fusion perception device based on attention mechanism and integrated learning in an embodiment of the present application.

[0110] like Figure 6 As shown, the multi-sensor fusion perception device 10 based on attention mechanism and integrated learning includes: an extraction module 100, a generation module 200 and a fusion module 300.

[0111] Among them, the extraction module 100 is used to obtain camera image data and lidar point cloud data of the target to be measured, and respectively extract at least one camera image feature and at least one lidar point cloud feature from the camera image data and the lidar point cloud data.

[0112] The generation module 200 is used to obtain a depth map corresponding to at least one lidar point cloud feature, and generate an image BEV feature map corresponding to the camera image data based on the camera image features and the depth map, and generate a point cloud BEV feature map corresponding to the lidar point cloud data based on the lidar point cloud features.

[0113] The fusion module 300 is used to perform enhanced feature processing on the image BEV feature map through a preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and to perform spatial feature processing on the point cloud BEV feature map using a preset spatial attention mechanism to obtain a spatial processing feature map, and to fuse the enhanced processing feature map and the spatial processing feature map according to a preset integrated learning strategy to generate a fused feature map, and to perceive the target detection result of the target to be measured through the fused feature map.

[0114] Optionally, in one embodiment of the present application, the generation module 200 includes: an arrangement unit, a first calculation unit, a transformation unit, a weighting unit, a flattening unit and a pooling unit.

[0115] Among them, the arrangement unit is used to arrange the laser radar point cloud features in the channel dimension, and perform linear transformation on the arranged laser radar point cloud features to obtain a query vector corresponding to the laser radar point cloud features.

[0116] The first calculation unit is used to perform maximum gradient calculation in multiple target directions on the point cloud Z axis of the laser radar point cloud feature, obtain the calculation result of each target direction in the multiple target directions, and splice the calculation result and the laser radar point cloud feature to obtain a depth map.

[0117] The transformation unit is used to generate an initial image feature query vector based on the query vector and a preset position encoding strategy, and perform reverse flattening and linear transformation operations on the initial image feature query vector to obtain a target query vector.

[0118] The weighting unit is used to extract the depth distribution information of the camera image feature based on the target query vector and the depth map, and weight the camera image feature by the depth distribution information to obtain the weighted camera image feature.

[0119] The flattening unit is used to flatten the weighted camera image features to obtain geometric information corresponding to the weighted camera image features, and map the feature points of the weighted camera image features to a preset bird's-eye view coordinate system through the geometric information.

[0120] The pooling unit is used to perform a preset pooling operation on the feature points to aggregate the feature points into corresponding bird's-eye view grid cells to generate an image BEV feature map corresponding to the camera image data.

[0121] Optionally, in one embodiment of the present application, the fusion module 300 includes: a second calculation unit, an acquisition unit, a first convolution unit, a second convolution unit, a summation unit and a first element multiplication unit.

[0122] Among them, the second calculation unit is used to calculate the global average pool corresponding to the image BEV feature map, and obtain the average feature map corresponding to the image BEV feature map according to the global average pool.

[0123] The acquisition unit is used to acquire the target feature of each feature map channel in the image BEV feature map, so as to determine the maximized feature map corresponding to the image BEV feature map according to the target feature of each feature map channel.

[0124] The first convolution unit is used to generate an initial convolution average feature map and an initial convolution maximum feature map based on the average feature map, the maximized feature map, the ReLU activation function and a preset initial convolution weight matrix.

[0125] The second convolution unit is used to obtain a secondary convolution average feature map and a secondary convolution maximum feature map according to the initial convolution average feature map, the initial convolution maximum feature map, the ReLU activation function and a preset secondary convolution weight matrix.

[0126] The summing unit is used to sum the quadratic convolution average feature map and the quadratic convolution maximum feature map to obtain a combined feature map, and calculate the channel significance score corresponding to the combined feature map.

[0127] The first element multiplication unit is used to perform an element multiplication operation on the image BEV feature map and the channel significance score to obtain an enhanced processing feature.

[0128] Optionally, in one embodiment of the present application, the fusion module 300 further includes: a processing unit, a connection unit and a second element multiplication unit.

[0129] Among them, the processing unit is used to perform a global average pooling operation on each spatial position in all channels of the point cloud BEV feature map to obtain an average pooling feature map corresponding to the point cloud BEV feature map.

[0130] The connection unit is used to extract the channel maximum feature of each spatial position corresponding to the point cloud BEV feature map, determine the channel maximum feature map corresponding to the point cloud BEV feature map through the channel maximum feature, and perform a channel dimension connection operation on the average pooling feature map and the channel maximum feature map to obtain a connection feature map.

[0131] The second element multiplication unit is used to perform convolution processing on the connection feature map, and generate a spatial attention map according to a preset sigmoid function and the connection feature map after convolution processing, and perform element multiplication operation on the spatial attention map and the point cloud BEV feature map to generate a spatial processing feature map.

[0132] It should be noted that the aforementioned explanation of the embodiment of the multi-sensor fusion perception method based on attention mechanism and integrated learning is also applicable to the multi-sensor fusion perception device based on attention mechanism and integrated learning of this embodiment, and will not be repeated here.

[0133] A multi-sensor fusion perception device based on attention mechanism and integrated learning proposed in an embodiment of the present application includes an extraction module for obtaining camera image data and lidar point cloud data of a target to be measured, and extracting at least one camera image feature and at least one lidar point cloud feature of the camera image data and the lidar point cloud data respectively; a generation module for obtaining a depth map corresponding to at least one lidar point cloud feature, and generating an image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map, and generating a point cloud BEV feature map corresponding to the lidar point cloud data according to the lidar point cloud feature; a fusion module for performing enhanced feature processing on the image BEV feature map through a preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and performing spatial feature processing on the point cloud BEV feature map using a preset spatial attention mechanism to obtain a spatial processing feature map, and fusing the enhanced processing feature map and the spatial processing feature map according to a preset integrated learning strategy to generate a fused feature map, and perceiving the target detection result of the target to be measured through the fused feature map. This application uses point cloud supervision to extract depth information from images, and uses attention mechanism and integrated learning to solve the problem of inaccurate depth information in images and improve the integration accuracy of point cloud and image information.

[0134] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0135] A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0136] When the processor 702 executes the program, the multi-sensor fusion perception method based on the attention mechanism and integrated learning provided in the above embodiment is implemented.

[0137] Furthermore, the electronic device further comprises:

[0138] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0139] The memory 701 is used to store computer programs that can be executed on the processor 702 .

[0140] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0141] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0142] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0143] The processor 702 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0144] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-sensor fusion perception method based on attention mechanism and integrated learning.

[0145] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned multi-sensor fusion perception method based on attention mechanism and integrated learning.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0148] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0149] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0150] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0151] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0153] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-sensor fusion perception method based on attention mechanism and ensemble learning, characterized in that: The following steps are involved: Acquire camera image data and laser radar point cloud data of the target to be measured, and extract at least one camera image feature and at least one laser radar point cloud feature from the camera image data and the laser radar point cloud data, respectively; Acquire a depth map corresponding to the at least one lidar point cloud feature, and generate an image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map, and generate a point cloud BEV feature map corresponding to the lidar point cloud data according to the lidar point cloud feature; The image BEV feature map is enhanced by a preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and the point cloud BEV feature map is spatially processed by a preset spatial attention mechanism to obtain a spatial processing feature map, and the enhanced processing feature map and the spatial processing feature map are fused according to a preset integrated learning strategy to generate a fused feature map, and the target detection result of the target to be measured is perceived through the fused feature map.

2. The method according to claim 1, characterized in that: The obtaining of a depth map corresponding to the at least one laser radar point cloud feature, and generating an image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map, comprises: Arranging the laser radar point cloud features in the channel dimension, and performing a linear transformation on the arranged laser radar point cloud features to obtain a query vector corresponding to the laser radar point cloud features; Performing maximum gradient calculation in multiple target directions on the point cloud Z axis of the laser radar point cloud feature to obtain a calculation result of each target direction in the multiple target directions, and splicing the calculation result and the laser radar point cloud feature to obtain the depth map; Based on the query vector and a preset position encoding strategy, an initial image feature query vector is generated, and reverse flattening and linear transformation operations are performed on the initial image feature query vector to obtain a target query vector; Extracting depth distribution information of the camera image feature based on the target query vector and the depth map, and weighting the camera image feature by the depth distribution information to obtain a weighted camera image feature; Flattening the weighted camera image feature to obtain geometric information corresponding to the weighted camera image feature, and mapping feature points of the weighted camera image feature to a preset bird's-eye view coordinate system through the geometric information; A preset pooling operation is performed on the feature points to aggregate the feature points into corresponding bird's-eye view grid cells to generate an image BEV feature map corresponding to the camera image data.

3. The method according to claim 2, characterized in that The enhanced feature processing of the image BEV feature map is performed by a preset enhanced channel attention mechanism to obtain enhanced processing features, including: Calculating a global average pool corresponding to the BEV feature map of the image, and obtaining an average feature map corresponding to the BEV feature map of the image according to the global average pool; Obtaining a target feature of each feature map channel in the image BEV feature map, so as to determine a maximized feature map corresponding to the image BEV feature map according to the target feature of each feature map channel; Based on the average feature map, the maximized feature map, the ReLU activation function and a preset initial convolution weight matrix, an initial convolution average feature map and an initial convolution maximized feature map are generated; Obtaining a secondary convolution average feature map and a secondary convolution maximum feature map according to the initial convolution average feature map, the initial convolution maximum feature map, the ReLU activation function and a preset secondary convolution weight matrix; Performing a sum operation on the secondary convolution average feature map and the secondary convolution maximum feature map to obtain a combined feature map, and calculating a channel significance score corresponding to the combined feature map; An element-wise multiplication operation is performed on the image BEV feature map and the channel significance score to obtain the enhanced processing feature.

4. The method according to claim 3, characterized in that The method of performing spatial feature processing on the point cloud BEV feature map by using a preset spatial attention mechanism to obtain a spatial processing feature map includes: Performing a global average pooling operation on each spatial position in all channels of the point cloud BEV feature map to obtain an average pooling feature map corresponding to the point cloud BEV feature map; Extracting the maximum channel feature of each spatial position corresponding to the point cloud BEV feature map, determining the maximum channel feature map corresponding to the point cloud BEV feature map through the maximum channel feature, and performing a channel dimension connection operation on the average pooling feature map and the maximum channel feature map to obtain a connection feature map; The connection feature map is convolved, and a spatial attention map is generated according to a preset sigmoid function and the connection feature map after the convolution process, and an element-wise multiplication operation is performed on the spatial attention map and the point cloud BEV feature map to generate the spatial processing feature map.

5. A multi-sensor fusion perception device based on attention mechanism and integrated learning, characterized in that: include: An extraction module, used to obtain camera image data and lidar point cloud data of the target to be measured, and extract at least one camera image feature and at least one lidar point cloud feature from the camera image data and the lidar point cloud data, respectively; A generating module, configured to obtain a depth map corresponding to the at least one laser radar point cloud feature, and generate an image BEV feature map corresponding to the camera image data based on the camera image feature and the depth map, and generate a point cloud BEV feature map corresponding to the laser radar point cloud data according to the laser radar point cloud feature; A fusion module is used to perform enhanced feature processing on the image BEV feature map through a preset enhanced channel attention mechanism to obtain an enhanced processing feature map, and to perform spatial feature processing on the point cloud BEV feature map using a preset spatial attention mechanism to obtain a spatial processing feature map, and to fuse the enhanced processing feature map and the spatial processing feature map according to a preset integrated learning strategy to generate a fused feature map, and to perceive the target detection result of the target to be measured through the fused feature map.

6. The device according to claim 5, characterized in that The generation module comprises: An arrangement unit, used for arranging the laser radar point cloud features in a channel dimension and performing a linear transformation on the arranged laser radar point cloud features to obtain a query vector corresponding to the laser radar point cloud features; A first calculation unit is used to perform maximum gradient calculation in multiple target directions on the point cloud Z axis of the laser radar point cloud feature, obtain a calculation result of each target direction in the multiple target directions, and splice the calculation result and the laser radar point cloud feature to obtain the depth map; A transformation unit, configured to generate an initial image feature query vector based on the query vector and a preset position encoding strategy, and perform reverse flattening and linear transformation operations on the initial image feature query vector to obtain a target query vector; a weighting unit, configured to extract depth distribution information of the camera image feature based on the target query vector and the depth map, and weight the camera image feature by the depth distribution information to obtain a weighted camera image feature; A flattening unit, used for flattening the weighted camera image feature to obtain geometric information corresponding to the weighted camera image feature, and mapping the feature points of the weighted camera image feature to a preset bird's-eye view coordinate system through the geometric information; A pooling unit is used to perform a preset pooling operation on the feature points to aggregate the feature points into corresponding bird's-eye view grid cells to generate an image BEV feature map corresponding to the camera image data.

7. The device according to claim 6, characterized in that The fusion module includes: A second calculation unit is used to calculate a global average pool corresponding to the image BEV feature map, and obtain an average feature map corresponding to the image BEV feature map according to the global average pool; An acquisition unit, used for acquiring a target feature of each feature map channel in the image BEV feature map, so as to determine a maximized feature map corresponding to the image BEV feature map according to the target feature of each feature map channel; A first convolution unit, used to generate an initial convolution average feature map and an initial convolution maximum feature map based on the average feature map, the maximized feature map, a ReLU activation function and a preset initial convolution weight matrix; A second convolution unit, used to obtain a secondary convolution average feature map and a secondary convolution maximum feature map according to the initial convolution average feature map, the initial convolution maximum feature map, the ReLU activation function and a preset secondary convolution weight matrix; A summing unit, used for summing the quadratic convolution average feature map and the quadratic convolution maximum feature map to obtain a combined feature map, and calculating a channel significance score corresponding to the combined feature map; The first element multiplication unit is used to perform an element multiplication operation on the image BEV feature map and the channel significance score to obtain the enhanced processing feature.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-sensor fusion perception method based on attention mechanism and integrated learning as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a multi-sensor fusion perception method based on attention mechanism and integrated learning as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the multi-sensor fusion perception method based on attention mechanism and integrated learning as described in any one of claims 1 to 4.

Images