Multi-sensor feature fusion method based on multi-scale local cross attention

The multi-scale local cross-attention algorithm is used to weight fusion of image and point cloud features, which solves the matching problems and insufficient weight attention in feature-level fusion, and improves the perceptual performance and computing efficiency.

CN119066613AInactive Publication Date: 2024-12-03CHONGQING UNIV

Patent Information

Application Number
CN202411193492.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the existing feature-level fusion scheme, images and point clouds cannot be fully matched, and the weights of different modes are not fully paid attention to, resulting in information loss and degraded perceptual performance.

Method used

Using multi-scale local cross attention algorithm, the image and point cloud feature tensor are used as query and key-value matrix source data respectively. Multi-scale attention weighted features are obtained through convolution processing and dot product operations, and modal fusion is performed to ensure that Feature alignment and weighting.

Benefits of technology

The complete matching of images and point clouds is achieved, which reduces information loss, improves the perceived performance of modal fusion features, and reduces the computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066613A_ABST
    Figure CN119066613A_ABST
Patent Text Reader

Abstract

The invention provides a multi-scale local cross attention-based multi-sensor feature fusion method, which comprises the following steps of: taking an image feature tensor as query matrix source data, and respectively taking a point cloud feature tensor as key matrix source data and value matrix source data, using a multi-scale local cross attention algorithm to obtain image multi-scale attention weighted features; using the point cloud feature tensor as query matrix source data, using the image feature tensor as key matrix source data and value matrix source data, and obtaining point cloud multi-scale attention weighted features by using a multi-scale local cross attention algorithm; and fusing the image multi-scale attention weighted features and the point cloud multi-scale attention weighted features to obtain modal fusion features. The invention further discloses a multi-sensor feature fusion device based on the multi-scale local cross attention, a computer program product and electronic equipment. According to the method, information loss is reduced, and the perception performance of the finally obtained modal fusion features is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of sensor data fusion technology, and in particular to a multi-sensor feature fusion method based on multi-scale local cross attention. Background Art

[0002] In the environmental perception technology of multi-sensor fusion, researchers are paying more and more attention to the fusion based on LiDAR and camera, the purpose of which is to combine the semantic information of the image with the location information of the point cloud, thereby improving the performance of environmental perception. This technology needs to consider the feature alignment and fusion methods of different modalities. With the development of deep learning technology, more and more research uses deep learning methods to realize the fusion of multi-sensor modalities at the feature level.

[0003] The feature-level fusion scheme fuses sensor data at the feature level to achieve performance complementarity between different sensors. This type of method uses the corresponding backbone network to extract features from images and point clouds respectively to obtain feature maps of the two modalities. Then the feature spaces of the two modalities are unified by projection, and a fusion network module is used to achieve multi-sensor fusion. For example, in Example 1 of the prior art, it is proposed to use CNN and PointNet structures to extract features from images and point clouds respectively, and then use a fusion network to process the feature maps of the two modalities, and predict the position and category of the target in the scene. Similarly, Example 2 in the prior art adds an image-based 2D detection objective function to the network branch of image feature extraction to achieve pre-training of the image network branch, and then aligns and fuses the features of the image and point cloud by projection. Obviously, both Example 1 and Example 2 have the potential problem of feature mismatch, that is, whether the image is projected to 3D or the point cloud is projected to the 2D plane, there is a problem that some pixels cannot match the 3D point cloud, or some point clouds cannot match the 2D pixels. To address this problem, Example 3 in the prior art proposes converting the features of the image and point cloud to the BEV view for unification, and performing feature matching and fusion in the BEV view. The fusion method of this example is simple and direct, but it relies on the accuracy of converting different modal features to the BEV modality, introduces conversion errors, and there is a possibility of reducing the perceptual performance of the fusion result.

[0004] In addition, existing feature-level fusion schemes rarely pay attention to the weights of different modalities. Due to the different characteristics of different sensors, one of the modalities often needs to be projected and transformed to unify with the other modality during fusion. Although both the camera and the radar have high resolution and can accurately detect the environment, some information will be lost in the projection transformation, which will reduce their performance. Taking the above examples 1, 2, and 3 as examples, if the point cloud is projected to a 2D plane to correspond to the image, most of the point clouds will not be able to match the image pixels due to the wider perception range of the radar. At this time, the perception efficiency of the radar point cloud will decrease, and it will serve more as an additional supplement to the image features. For example, if the image is projected to a 3D space to correspond to the point cloud, or the image is projected to a BEV view and fused with the point cloud, at this time, due to the limited range of the image, all point clouds cannot be matched, and a large amount of information will be lost when the image is converted to the BEV space, resulting in a serious performance degradation. At this time, more attention should be paid to the radar point cloud features, and the image features are used as an additional supplement. It can be seen that paying attention to the weights of different modalities in the feature-level fusion scheme is conducive to improving the perception performance of the fusion result. Summary of the invention

[0005] The present invention aims to at least solve the technical problems in existing feature-level fusion schemes, such as the inability to fully match pixels and point clouds, and the lack of attention to the weights of different modalities, which leads to information loss and the possibility of reducing the perceptual performance of the fusion results. A multi-sensor feature fusion method, device, product and electronic device based on multi-scale local cross-attention are provided.

[0006] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a multi-sensor feature fusion method based on multi-scale local cross-attention, including: obtaining an image and a point cloud to be fused; converting the image into an image feature tensor, and converting the point cloud into a point cloud feature tensor; using the image feature tensor as the query matrix source data of a preset multi-scale local cross-attention algorithm, and using the point cloud feature tensor as the key matrix source data and the value matrix source data of the multi-scale local cross-attention algorithm, respectively, and using the multi-scale local cross-attention algorithm to obtain the image multi-scale attention weighted features; using the point cloud feature tensor as the query matrix source data of the multi-scale local cross-attention algorithm, and using the image feature tensor as the key matrix source data and the value matrix source data of the multi-scale local cross-attention algorithm, respectively, and using the multi-scale local cross-attention algorithm to obtain the point cloud multi-scale attention weighted features; fusing the image multi-scale attention weighted features and the point cloud multi-scale attention weighted features to obtain the modal fusion features.

[0007] Preferably, the multi-scale local cross-attention algorithm includes: using the first convolution to convolve the query matrix source data to obtain the query matrix; using the second convolution to convolve the key matrix source data to obtain the key matrix; using the third convolution to convolve the value matrix source data to obtain the value matrix; traversing the preset convolution kernel size set to select the convolution kernel size of the neighborhood convolution kernel, and performing the following steps after each traversal to obtain the attention weighted features corresponding to the selected convolution kernel size: using the neighborhood convolution kernel to traverse the key matrix to obtain the key neighborhood that needs to be paid attention to for each pixel in the query matrix in the key matrix, and all key neighborhoods constitute a key neighborhood set V_patch; use the neighborhood convolution kernel to traverse the value matrix to obtain the value neighborhood that needs to be paid attention to for each pixel in the query matrix in the value matrix, and all value neighborhoods constitute the value neighborhood set K_patch; obtain the similarity score matrix between the query matrix and the key matrix by performing a dot product operation with the key neighborhood set V_patch and the query matrix; obtain the attention weighted features corresponding to the selected convolution kernel size by performing a dot product operation with the similarity score matrix and the value neighborhood set K_patch; accumulate the attention weighted features corresponding to all convolution kernel sizes in the convolution kernel size set to obtain the multi-scale attention weighted features.

[0008] Preferably, the similarity score matrix Scores is:

[0009] Scores=Softmax(Q×K_patch).

[0010] Preferably, the convolution kernel size set is: {1×1, 3×3, 5×5}.

[0011] Preferably, converting the image into an image feature tensor includes: extracting image features of the image using a CNN network, and using the image features as an image feature tensor, wherein the size of the image feature tensor is (C, H, W), wherein C represents the number of channels, H represents the number of rows, and W represents the number of columns.

[0012] Preferably, converting the point cloud into a point cloud feature tensor comprises:

[0013] Convert the point cloud into a plurality of discrete voxels, and obtain a plurality of attributes of each voxel, wherein the plurality of attributes include a center point coordinate, a number of points, a highest point position, and an average position;

[0014] The VoxelNet feature learning network is used to convert each voxel or voxel group into a feature vector, and all feature vectors are combined to obtain the point cloud feature tensor. The size of the point cloud feature tensor is (C, H, W).

[0015] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a multi-sensor feature fusion device based on multi-scale local cross-attention, which is used to implement a multi-sensor feature fusion method based on multi-scale local cross-attention described in the first aspect of the present invention, including: an acquisition module, which acquires the image and point cloud to be fused; a conversion module, which converts the image into an image feature tensor and converts the point cloud into a point cloud feature tensor; an image multi-scale attention weighted feature acquisition module, which uses the image feature tensor as the query matrix source data of the multi-scale local cross-attention module, and the point cloud feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross-attention module, respectively, and uses the multi-scale local cross-attention module to obtain the image multi-scale attention weighted feature; a point cloud multi-scale attention weighted feature acquisition module, which uses the point cloud feature tensor as the query matrix source data of the multi-scale local cross-attention module, and uses the image feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross-attention module, respectively, and uses the multi-scale local cross-attention module to obtain the point cloud multi-scale attention weighted feature; a fusion module, which fuses the image multi-scale attention weighted feature and the point cloud multi-scale attention weighted feature to obtain the modal fusion feature.

[0016] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a multi-sensor feature fusion method based on multi-scale local cross-attention described in the first aspect of the present invention.

[0017] In order to achieve the above-mentioned purpose of the present invention, according to the fourth aspect of the present invention, the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a multi-sensor feature fusion method based on multi-scale local cross-attention as described in the first aspect of the present invention.

[0018] The present invention realizes the multi-scale feature interaction of images and point clouds through a multi-scale local cross-attention algorithm / module, extracts the image multi-scale attention weighted features and the point cloud multi-scale attention weighted features respectively, and then fuses the image multi-scale attention weighted features and the point cloud multi-scale attention weighted features to obtain the modal fusion features, realizes the alignment and weighting of different modal features, makes the pixels and the point cloud fully match, reduces the information loss in multi-modal fusion, uses the similarity score matrix Scores as the weight of the image feature tensor or the point cloud feature tensor, realizes weighted fusion, and improves the perception performance of the modal fusion features finally obtained; wherein, in the multi-scale local cross-attention algorithm / module, in the interactive process of obtaining the multi-scale attention weighted features, the query The matrix only pays attention to the key value information in the vicinity of the corresponding position. In addition to reducing the amount of calculation, this method is also conducive to the alignment and weighting of features. The neighborhood convolution kernel uses different convolution kernel sizes to obtain attention weighted features, which can help capture features at different levels and achieve it without significantly increasing the computational complexity. The local attention mechanism can make the image and point cloud only focus on the features of the local area when interacting. Compared with the global attention, it can effectively reduce the computational complexity and make it easier to achieve feature alignment. In the multi-scale local cross-attention algorithm / module, the query matrix, value matrix and key matrix are calculated in the convolution method of the first convolution, the second convolution and the third convolution, which can better capture the spatial relationship of the features and can be better suitable for image applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a multi-sensor feature fusion method based on multi-scale local cross attention in a preferred embodiment of the present invention;

[0020] Figure 2 It is an overall flow chart of a multi-sensor feature fusion method based on multi-scale local cross attention in a specific application scenario of the present invention;

[0021] Figure 3 It is an execution flow chart of a multi-scale local cross attention algorithm / module in a preferred embodiment of the present invention;

[0022] Figure 4 It is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0023] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0024] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0025] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0026] The execution subject of the multi-sensor feature fusion method based on multi-scale local cross-attention includes but is not limited to at least one of the electronic devices such as the server and the terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the multi-sensor feature fusion method based on multi-scale local cross-attention can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0027] Traditional cross attention is an important structure in transformer, which is applied between decoder and encoder, and is mainly used to obtain encoder information related to the current decoding position. Therefore, cross attention will calculate the attention of two different sequences. Unlike the self-attention mechanism, cross attention calculates the query matrix Q based on one of the sequences and the key matrix K and value matrix V based on the other sequence, so as to realize the interaction between the two sequences. Let the two input tensors be X1 and X2, X1 as the query set and X2 as the key value set.

[0028] Let Q = X1W Q ,K=X2W K ,V=X2W V , then the calculation formula of the traditional cross attention is as follows:

[0029]

[0030] Where d represents the size of the feature vector. For two-dimensional images (H, W) and point cloud features (H, W), d = H × W. H represents the number of rows and W represents the number of columns. Q represents the query weight matrix; W K represents the key weight matrix; W V Represents the value weight matrix.

[0031] The above-mentioned traditional cross-book attention method is mostly used for serialized data such as text, which has a moderate size. For two-dimensional image features (image feature tensors), the features need to be expanded into one dimension. At this time, the size of the feature vector will be too large, resulting in an increase in calculation time. Therefore, the present invention provides a multi-sensor feature fusion method based on multi-scale local cross-attention, which uses a local cross-attention method to calculate the attention score (i.e., the similarity score matrix), that is, during the interaction process, the query matrix only pays attention to the key value information in the vicinity of the corresponding position. In addition to reducing the amount of calculation, this method is also conducive to the alignment and weighting of features; at the same time, the method uses a convolution method (first convolution, second convolution, third convolution) to calculate the query matrix Q, key matrix K, and value matrix V, while the non-traditional cross-attention mechanism calculates each matrix through a linear layer, which can better capture the spatial relationship of features and is suitable for image applications.

[0032] The present invention discloses a multi-sensor feature fusion method based on multi-scale local cross attention. In a preferred embodiment, Figure 1 As shown, the method includes:

[0033] Step S1, obtaining the image and point cloud to be fused.

[0034] Specifically, the image is preferably, but not limited to, an RGB image taken by an RGB camera, or an infrared image taken by an infrared camera. The point cloud data is preferably, but not limited to, point cloud data obtained by a lidar or a structured light scanner or a time-of-flight (ToF) sensor or a binocular / multi-camera stereo vision device or an ultrasonic sensor or an RGB-D camera (using its depth map). In order to improve environmental perception capabilities, the image and point cloud to be fused may be data collected from the surface or area of ​​the same object. In some examples, the image and point cloud to be fused are respectively derived from a camera and a point cloud device (such as a lidar) for environmental perception mounted on a mobile target (robot, vehicle, drone, etc.).

[0035] Step S2: convert the image into an image feature tensor and convert the point cloud into a point cloud feature tensor. The image feature tensor and the point cloud feature tensor have the same size, which is (C, H, W), where C represents the number of channels, H represents the number of rows, and W represents the number of columns.

[0036] Step S3, using the image feature tensor as the query matrix source data of the preset multi-scale local cross-attention algorithm, and using the point cloud feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross-attention algorithm respectively, and using the multi-scale local cross-attention algorithm to obtain the image multi-scale attention weighted features.

[0037] Specifically, refer to Figure 3 As shown in the figure, the image feature tensor is used as tensor X1 to obtain the query matrix Q, and the point cloud feature tensor is used as tensor X2 to obtain the key matrix K and the value matrix V. The multi-scale local cross attention algorithm obtains the image multi-scale attention weighted feature WF based on the obtained query matrix Q, key matrix K and value matrix V. img , the process can be expressed as:

[0038] WF img =MSLCA(F img ,F pts ). img represents the image feature tensor, F pts represents the point cloud feature tensor, and MSLCA(,) represents the multi-scale local cross attention algorithm.

[0039] Step S4, using the point cloud feature tensor as the query matrix source data of the multi-scale local cross-attention algorithm, and using the image feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross-attention algorithm respectively, and using the multi-scale local cross-attention algorithm to obtain the point cloud multi-scale attention weighted features.

[0040] Specifically, refer to Figure 3 , the point cloud feature tensor is used as tensor X1 to obtain the query matrix Q, and the image feature tensor is used as tensor X2 to obtain the key matrix K and value matrix V. The multi-scale local cross attention algorithm obtains the point cloud multi-scale attention weighted feature WF based on the obtained query matrix Q, key matrix K and value matrix V. pts , the process can be expressed as:

[0041] WF pts =MSLCA(F pts ,F img ). img represents the image feature tensor, F pts Represents a point cloud feature tensor.

[0042] Step S5, fusing the image multi-scale attention weighted features and the point cloud multi-scale attention weighted features to obtain the modality fusion feature F (pts,img) .

[0043] For ease of calculation, preferably, F (pts,img) =WFpts +WF img .

[0044] It should be noted that, in this embodiment, steps S3 and S4 can be executed in parallel to increase the processing speed.

[0045] Figure 2 It is an overall flow chart of a multi-sensor feature fusion method based on multi-scale local cross attention in a specific application scenario of the present invention. After obtaining the image feature vector and the point cloud feature vector, the image multi-scale attention weighted feature WF is obtained in parallel. img And point cloud multi-scale attention weighted feature WF pts , and finally fuse the image multi-scale attention weighted feature WF img And point cloud multi-scale attention weighted feature WF pts Obtain modal fusion feature F (pts,img) , which can improve processing speed.

[0046] In a preferred embodiment, in order to capture features at different levels, facilitate the alignment of image and point cloud features, and improve the modality fusion feature F (pts,img) Perceived performance, see Figure 3 , the above multi-scale local cross-attention algorithm includes:

[0047] Step 01, use the first convolution to convolve the query matrix source data to obtain the query matrix Q:

[0048] Q = Conv(X1, kernel_size = 1)

[0049] The kernel size kernel_size of the first convolution is 1, that is, the size is 1×1.

[0050] The key matrix K is obtained by convolving the key matrix source data using the second convolution:

[0051] K = Conv(X2, kernel_size = 1)

[0052] The kernel size kernel_size of the second convolution is 1, that is, the size is 1×1.

[0053] The third convolution is used to convolve the value matrix source data to obtain the value matrix V:

[0054] V = Conv(X2, kernel_size = 1)

[0055] The kernel size kernel_size of the third convolution is 1, that is, the size is 1×1.

[0056] In step 01, the linear layer calculation method used in the traditional method is changed to the above-mentioned convolutional layer calculation method. By adjusting the convolution kernel size, convolution step size and other parameters of the first convolutional layer, the second convolutional layer and the third convolutional layer, the sizes of the obtained Q, K, and V matrices are all (C, H, W), which are consistent with the input tensors X1 and X2. Then, the size of the query matrix Q is adjusted to ((H×W), 1, C) to facilitate the subsequent attention score (i.e., similarity score) calculation.

[0057] Step 02, traverse the preset convolution kernel size set to select the convolution kernel size of the neighborhood convolution kernel. After each traversal, perform the following steps to obtain the attention weighted features corresponding to the selected convolution kernel size:

[0058] Step 021, using the neighborhood convolution kernel to traverse the key matrix to obtain the key neighborhood that needs to be paid attention to for each pixel in the query matrix in the key matrix, and all key neighborhoods constitute a key neighborhood set V_patch;

[0059] The neighborhood convolution kernel is used to traverse the value matrix to obtain the value neighborhood that needs to be paid attention to for each pixel in the query matrix in the value matrix. All value neighborhoods constitute the value neighborhood set K_patch.

[0060] In step 021, after each traversal, a fixed-size neighborhood convolution kernel is used to traverse the key matrix K and the value matrix V respectively to obtain the corresponding neighborhood (patch, such as key neighborhood and value neighborhood). The patch is the set of pixels in the neighborhood that each pixel needs to pay attention to, and the convolution kernel size is the size of the local neighborhood of interest. Since the size of the key matrix K and the value matrix V is (C×H×W), the size of the corresponding patch is (C×kw×kh, H×W), where (kw, kh) represents the size of the neighborhood convolution kernel. The size of K_patch is adjusted to ((H×W), C, (kw×kh)), and the size of V_patch is adjusted to ((H×W), (kw×kh), C) to facilitate the subsequent calculation of the similarity score matrix, that is, the local similarity and attention score.

[0061] Step 022, obtain the similarity score matrix Scores between the query matrix and the key matrix by performing a dot product operation on the key neighborhood set V_patch and the query matrix.

[0062] Specifically, since the size of the query matrix Q is ((H×W), 1, C) and the size of the key neighborhood set K_patch is ((H×W), C, (kw×kh)), the query matrix Q and the key neighborhood set K_patch are matrix-producted along the last two dimensions to obtain the local region similarity score, that is, the (1, C) two-dimensional data of the query matrix Q and the (C, (kw×kh)) two-dimensional data of the key neighborhood set K_patch are matrix-producted to obtain the initial similarity score matrix, whose size is ((H×W), 1, (kw×kh)). Then, the Softmax operation is performed on the last dimension of the initial similarity score matrix to achieve weighting, and the final similarity score matrix Scores is obtained:

[0063] Scores=Softmax(Q×K_patch).

[0064] Step 023, the attention weighted features corresponding to the selected convolution kernel size are obtained by performing a dot product operation on the similarity score matrix Scores and the value neighborhood set V_patch. Specifically, the similarity score matrix Scores and the value neighborhood set V_patch are matrix-producted along the last two dimensions to implement attention weighting and obtain the attention weighted features corresponding to the convolution kernel size selected in this traversal:

[0065] LCA i =Scores×V_patch;

[0066] i represents the index of the convolution kernel size selected in this traversal in the preset convolution kernel size set.

[0067] Specifically, since the size of Scores is ((H×W),1,(kw×kh)) and the size of V_patch is ((H×W),(kw×kh),C), the weighted summation result is the attention weighted feature LCA corresponding to the selected convolution kernel size i The size of is ((H×W), 1, C), which is adjusted to (C, H, W) size, consistent with the input tensor, which is the final weighted result.

[0068] Step 03: Accumulate the attention weighted features corresponding to all convolution kernel sizes in the convolution kernel size set to obtain the multi-scale attention weighted feature ∑LCA i .

[0069] In this embodiment, in order to improve the accuracy of cross-attention, local areas of different scales (convolution kernel size of neighborhood convolution kernel) are used to calculate attention, and then these attention results of different scales are combined to help the model capture features at different levels. Preferably, the convolution kernel size set is: {1×1, 3×3, 5×5}. It means that neighborhood convolution kernels of sizes 1×1, 3×3, and 5×5 are used to slide on the key matrix K and the value matrix V, respectively, to obtain local areas of different sizes, and to obtain the weighted results of the corresponding local sizes according to the local cross-attention, and finally the weighted results corresponding to all neighborhood convolution kernels are accumulated to obtain the final weighted result.

[0070] In a preferred embodiment, in order to extract image features more accurately, the image is converted into an image feature tensor, specifically including: using a CNN network to extract image features of the image, using the image features as an image feature tensor, and changing the image feature tensor size to (C, H, W), where C represents the number of channels, H represents the number of rows, and W represents the number of columns. The image features are extracted through the convolution layer and pooling layer of the CNN network, and the convolution kernel parameters are finally determined through back propagation to obtain the final image feature tensor.

[0071] In a preferred embodiment, in order to extract point cloud features more accurately and efficiently, the point cloud is converted into a point cloud feature tensor, including:

[0072] Convert the point cloud into multiple discrete voxels, and obtain multiple attributes of each voxel, including the center point coordinates, the number of points, the highest point position, and the average position;

[0073] The VoxelNet feature learning network is used to convert each voxel or voxel group (each voxel group includes more than one voxel) into a high-dimensional feature vector, and all feature vectors are combined to obtain the point cloud feature tensor. The size of the point cloud feature tensor is (C, H, W).

[0074] The present invention also discloses a multi-sensor feature fusion device based on multi-scale local cross attention, which is used to implement the multi-sensor feature fusion method based on multi-scale local cross attention provided by the present invention. In a preferred embodiment, the device includes:

[0075] Acquisition module, acquiring the image and point cloud to be fused;

[0076] The conversion module converts the image into an image feature tensor and the point cloud into a point cloud feature tensor;

[0077] The image multi-scale attention weighted feature acquisition module uses the image feature tensor as the query matrix source data of the multi-scale local cross attention module, and uses the point cloud feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross attention module, and uses the multi-scale local cross attention module to obtain the image multi-scale attention weighted features;

[0078] The point cloud multi-scale attention weighted feature acquisition module uses the point cloud feature tensor as the query matrix source data of the multi-scale local cross attention module, and uses the image feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross attention module, and uses the multi-scale local cross attention module to obtain the point cloud multi-scale attention weighted features;

[0079] The fusion module fuses the multi-scale attention weighted features of the image and the multi-scale attention weighted features of the point cloud to obtain the modal fusion features.

[0080] In this embodiment, the acquisition module, the conversion module, the image multi-scale attention weighted feature acquisition module, the point cloud multi-scale attention weighted feature acquisition module and the fusion module respectively correspond to steps S1, S2, S3, S4 and S5 of the above-mentioned multi-sensor feature fusion method based on multi-scale local cross-attention, and will not be repeated here.

[0081] In a preferred embodiment, the multi-scale local criss-cross attention module includes:

[0082] A first convolution unit, performing convolution processing on the query matrix source data using a first convolution to obtain a query matrix;

[0083] A second convolution unit, performing convolution processing on the key matrix source data using a second convolution to obtain a key matrix;

[0084] A third convolution unit, performing convolution processing on the value matrix source data using a third convolution to obtain a value matrix;

[0085] The attention weighted feature acquisition unit traverses the preset convolution kernel size set to select the convolution kernel size of the neighborhood convolution kernel, and performs the following steps after each traversal to obtain the attention weighted feature corresponding to the selected convolution kernel size:

[0086] The neighborhood convolution kernel is used to traverse the key matrix to obtain the key neighborhood that needs to be paid attention to in the key matrix for each pixel in the query matrix. All key neighborhoods constitute the key neighborhood set V_patch;

[0087] Use the neighborhood convolution kernel to traverse the value matrix to obtain the value neighborhood that needs to be paid attention to in the value matrix for each pixel in the query matrix. All value neighborhoods constitute the value neighborhood set K_patch;

[0088] A similarity score matrix between the query matrix and the key matrix is ​​obtained by performing a dot product operation on the key neighborhood set V_patch and the query matrix; an attention weighted feature corresponding to the selected convolution kernel size is obtained by performing a dot product operation on the similarity score matrix and the value neighborhood set K_patch;

[0089] The accumulation unit accumulates the attention weighted features corresponding to all convolution kernel sizes in the convolution kernel size set to obtain multi-scale attention weighted features.

[0090] In this embodiment, the first convolution unit, the second convolution unit and the third convolution unit correspond to step 01 of the above-mentioned multi-sensor feature fusion method based on multi-scale local cross-attention, and the attention weighted feature acquisition unit and the accumulation unit correspond one-to-one to steps 02 and 03 of the above-mentioned multi-sensor feature fusion method based on multi-scale local cross-attention, respectively, and are not repeated here.

[0091] The present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned multi-sensor feature fusion method based on multi-scale local cross-attention provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0092] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a multi-sensor feature fusion method based on multi-scale local cross-attention provided by the present invention.

[0093] like Figure 4 , which is a schematic diagram of the structure of an electronic device of a multi-sensor feature fusion method based on multi-scale local cross attention provided by an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a multi-sensor feature fusion method program based on multi-scale local cross attention.

[0094] Among them, the processor 10 may be composed of an integrated circuit in some embodiments, for example, it may be composed of a single packaged integrated circuit, or it may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a multi-sensor feature fusion method based on multi-scale local cross attention, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0095] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as a code of a multi-sensor feature fusion method program based on multi-scale local cross attention, but also can be used to temporarily store data that has been output or is to be output.

[0096] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0097] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0098] Figure 4 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) for powering each component. Preferably, the power supply may be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0099] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0100] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0102] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A multi-sensor feature fusion method based on multi-scale local cross attention, characterized in that: include: Obtain the image and point cloud to be fused; Convert the image into an image feature tensor and the point cloud into a point cloud feature tensor; The image feature tensor is used as the query matrix source data of the preset multi-scale local cross-attention algorithm, and the point cloud feature tensor is used as the key matrix source data and value matrix source data of the multi-scale local cross-attention algorithm respectively, and the multi-scale local cross-attention algorithm is used to obtain the image multi-scale attention weighted features; The point cloud feature tensor is used as the query matrix source data of the multi-scale local cross-attention algorithm, and the image feature tensor is used as the key matrix source data and value matrix source data of the multi-scale local cross-attention algorithm respectively, and the multi-scale local cross-attention algorithm is used to obtain the multi-scale attention weighted features of the point cloud; The modality fusion features are obtained by fusing the multi-scale attention weighted features of the image and the multi-scale attention weighted features of the point cloud.

2. A multi-sensor feature fusion method based on multi-scale local cross attention as claimed in claim 1, characterized in that: The multi-scale local criss-cross attention algorithm includes: Using the first convolution to convolve the query matrix source data to obtain the query matrix; Using the second convolution to convolve the key matrix source data to obtain the key matrix; Using the third convolution to perform convolution processing on the value matrix source data to obtain the value matrix; Traverse the preset convolution kernel size set to select the convolution kernel size of the neighborhood convolution kernel. After each traversal, perform the following steps to obtain the attention weighted features corresponding to the selected convolution kernel size: The neighborhood convolution kernel is used to traverse the key matrix to obtain the key neighborhood that needs to be paid attention to in the key matrix for each pixel in the query matrix. All key neighborhoods constitute the key neighborhood set V_patch; Use the neighborhood convolution kernel to traverse the value matrix to obtain the value neighborhood that needs to be paid attention to in the value matrix for each pixel in the query matrix. All value neighborhoods constitute the value neighborhood set K_patch; The similarity score matrix between the query matrix and the key matrix is ​​obtained by performing a dot product operation on the key neighborhood set V_patch and the query matrix; The attention weighted feature corresponding to the selected convolution kernel size is obtained by performing a dot product operation on the similarity score matrix and the value neighborhood set K_patch; The attention weighted features corresponding to all convolution kernel sizes in the convolution kernel size set are accumulated to obtain multi-scale attention weighted features.

3. A multi-sensor feature fusion method based on multi-scale local cross attention as claimed in claim 2, characterized in that: The similarity score matrix Scores is: Scores=Softmax(Q×K_patch).

4. The multi-sensor feature fusion method based on multi-scale local cross attention as claimed in claim 2, characterized in that: The convolution kernel size set is: {1×1, 3×3, 5×5}.

5. A multi-sensor feature fusion method based on multi-scale local cross attention as described in any one of claims 1 to 4, characterized in that: The converting of the image into an image feature tensor includes: extracting image features of the image using a CNN network, and using the image features as an image feature tensor, wherein the size of the image feature tensor is (C, H, W), wherein C represents the number of channels, H represents the number of rows, and W represents the number of columns.

6. A multi-sensor feature fusion method based on multi-scale local cross attention as claimed in claim 5, characterized in that: The step of converting the point cloud into a point cloud feature tensor includes: Convert the point cloud into a plurality of discrete voxels, and obtain a plurality of attributes of each voxel, wherein the plurality of attributes include a center point coordinate, a number of points, a highest point position, and an average position; The VoxelNet feature learning network is used to convert each voxel or voxel group into a feature vector, and all feature vectors are combined to obtain the point cloud feature tensor. The size of the point cloud feature tensor is (C, H, W).

7. A multi-sensor feature fusion device based on multi-scale local cross attention, used to implement a multi-sensor feature fusion method based on multi-scale local cross attention as described in any one of claims 1-6, characterized in that: include: Acquisition module, acquiring the image and point cloud to be fused; The conversion module converts the image into an image feature tensor and the point cloud into a point cloud feature tensor; The image multi-scale attention weighted feature acquisition module uses the image feature tensor as the query matrix source data of the multi-scale local cross attention module, and uses the point cloud feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross attention module, and uses the multi-scale local cross attention module to obtain the image multi-scale attention weighted features; The point cloud multi-scale attention weighted feature acquisition module uses the point cloud feature tensor as the query matrix source data of the multi-scale local cross attention module, and uses the image feature tensor as the key matrix source data and value matrix source data of the multi-scale local cross attention module, and uses the multi-scale local cross attention module to obtain the point cloud multi-scale attention weighted features; The fusion module fuses the multi-scale attention weighted features of the image and the multi-scale attention weighted features of the point cloud to obtain the modal fusion features.

8. A multi-sensor feature fusion device based on multi-scale local cross attention as claimed in claim 7, characterized in that: The multi-scale local criss-cross attention module includes: A first convolution unit, performing convolution processing on the query matrix source data using a first convolution to obtain a query matrix; A second convolution unit, performing convolution processing on the key matrix source data using a second convolution to obtain a key matrix; A third convolution unit, performing convolution processing on the value matrix source data using a third convolution to obtain a value matrix; The attention weighted feature acquisition unit traverses the preset convolution kernel size set to select the convolution kernel size of the neighborhood convolution kernel, and performs the following steps after each traversal to obtain the attention weighted feature corresponding to the selected convolution kernel size: The neighborhood convolution kernel is used to traverse the key matrix to obtain the key neighborhood that needs to be paid attention to in the key matrix for each pixel in the query matrix. All key neighborhoods constitute the key neighborhood set V_patch; Use the neighborhood convolution kernel to traverse the value matrix to obtain the value neighborhood that needs to be paid attention to in the value matrix for each pixel in the query matrix. All value neighborhoods constitute the value neighborhood set K_patch; A similarity score matrix between the query matrix and the key matrix is ​​obtained by performing a dot product operation on the key neighborhood set V_patch and the query matrix; an attention weighted feature corresponding to the selected convolution kernel size is obtained by performing a dot product operation on the similarity score matrix and the value neighborhood set K_patch; The accumulation unit accumulates the attention weighted features corresponding to all convolution kernel sizes in the convolution kernel size set to obtain multi-scale attention weighted features.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a multi-sensor feature fusion method based on multi-scale local cross-attention as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Point cloud segmentation method, device and equipment based on voxel and point set fusion

    CN118135220A

  • Image point cloud fusion method and device, electronic equipment and storage medium

    CN118230111A

  • Target tracking method and system based on multi-scale aggregation attention feature extraction network

    CN118429389A

  • Three-dimensional target reconstruction method based on non-calibration single view

    CN118470221A

Cited By

  • Cross-modal large aircraft point cloud completion method

    CN120147190A