Multi-modal data fusion method and system

Abstract

The invention relates to the technical field of point cloud data processing, in particular to a multi-modal data fusion method and system, and aims to avoid information loss caused by inaccurate projection through a dynamic cross-modal attention mechanism and accurate capture of a mutual relation between an image and point cloud features, and particularly, the multi-modal data fusion method and system are excellent in performance in a long-distance area; according to the method, through an adaptive attention mechanism, dynamic interaction of the image and the point cloud features can be realized, deep modal cooperation is realized, and each scale feature of the two modals can be effectively utilized; according to the method, through an adaptive weight generation mechanism, contributions of different modes in fusion are balanced, and the problem of information imbalance is avoided; meanwhile, the fusion process is optimized, the calculation complexity is reduced through multi-scale dynamic fusion, and the real-time performance and the resource efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud data processing, and in particular to a multimodal data fusion method and system. Background Art

[0002] The multimodal data fusion task of 3D point cloud and 2D image is mainly used in fields such as autonomous driving and environmental perception. It aims to combine the 3D point cloud collected by LiDAR with the 2D image data captured by the camera to improve the performance of tasks such as target detection and semantic segmentation. Existing technical solutions can be divided into four modes. Early fusion realizes the spatial alignment of the two modalities at the raw data level through an external calibration matrix. For example, Frustum PointNet performs 3D target detection by projecting the region of interest in the image to the point cloud. The mid-term fusion method performs fusion after multimodal feature extraction, such as PointPainting and FusionPainting by projecting 2D semantic information onto the point cloud and then processing it using a 3D network. The late fusion method processes the multimodal branches separately and then fuses the output, retaining the independent processing process of each modality. Multi-phase fusion combines the advantages of early, mid-term and late fusion, and designs a multi-stage fusion module, such as PMF by projecting the point cloud to the image plane for fusion, and 2D3DNet by jointly optimizing the 2D segmentation and 3D point cloud model. These technical solutions provide different implementation paths for the fusion of 3D point cloud and 2D image.

[0003] The existing multimodal fusion technology of 3D point cloud and 2D image still has some objective shortcomings in practical applications. Early fusion methods are prone to information loss during the projection process due to the sparsity of point clouds, especially in long-distance areas, and are limited by the field of view of the camera. Although mid-term fusion methods process at the feature level, they usually ignore the deep synergy between different modalities, or the information of one modality dominates due to resolution differences, affecting the overall fusion effect. Although late fusion retains the independent processing of each modality, it fails to fully utilize the complementary information between modalities, resulting in limited fusion effect. In addition, although multi-phase fusion methods attempt to combine the advantages of multiple fusion stages, they are complex in design, have high computational costs, and may increase the complexity of the model, affecting real-time performance and resource efficiency. Summary of the invention

[0004] In view of this, the present invention proposes a multimodal data fusion method and system, which reduces information loss in the point cloud projection process through a cross-modal collaborative attention mechanism and enhances the deep synergy between different modalities. Through multi-scale feature extraction and adaptive fusion weights, the dominant problem caused by the difference in resolution between different modalities is effectively solved, and the weight distribution between modalities is dynamically adjusted to improve the adaptability and flexibility of the model. In addition, this method optimizes the overall computing efficiency by intelligently allocating computing resources, solving the problem of excessive computing burden in the prior art.

[0005] In order to achieve the above object, a multimodal data fusion method is provided, wherein the multimodal data includes three-dimensional point cloud data and two-dimensional image data, and the method includes the following steps: S1: performing multi-scale feature extraction on the multimodal data to obtain three-dimensional point cloud data features and two-dimensional image data features; S2: dynamically calculating the relationship between the three-dimensional point cloud features and the image data features based on a cross-modal collaborative attention mechanism; S3: dynamically reconstructing the image features and the point cloud features; S4: Perform multimodal fusion on the reconstructed image features and the point cloud features to obtain the final fusion features .

[0006] Preferably, performing multi-scale feature extraction on the multimodal data includes performing feature extraction on the two-dimensional image data and performing feature extraction on the three-dimensional point cloud data.

[0007] Preferably, the feature extraction of the two-dimensional image data is specifically performed as follows: a convolutional neural network model (CNN) is used to extract multi-scale image features of the two-dimensional image data, and after the two-dimensional image data passes through different convolutional layers of the convolutional neural network, a series of feature maps are generated, and the convolutional neural network model captures the local information and global information of the two-dimensional image data respectively; the image features of each scale are represented as , , where n is the number of feature vectors, d is the dimension of the feature, and s is the index of multi-scale processing, which is used to distinguish features extracted at different scales.

[0008] Preferably, the feature extraction of the three-dimensional point cloud data is specifically performed by: using the point cloud processing network of PointNet++, performing hierarchical sampling and feature extraction on the three-dimensional point cloud data, generating geometric features of point clouds at different scales, and the point cloud features of each scale are represented as , , where m is the number of points, d is the feature dimension, and s is the index of multi-scale processing, which is used to distinguish features extracted at different scales.

[0009] Preferably, in S2, the dynamic calculation of the relationship between the three-dimensional point cloud features and the image data features based on the cross-modal collaborative attention mechanism is specifically: S2.1: The image features Mapping to query vector ; S2.2: The point cloud features Mapping to key vector ; S2.3: The point cloud features Mapping to a vector of values ; S2.4: Calculating the relationship between the image features and the point cloud features; S2.5: Using the attention weight matrix Perform weighted summation on the value vector to obtain image features combined with point cloud information .

[0010] Preferably, S3 is specifically: according to the attention weight matrix Dynamic feature reconstruction is performed on the image features and the point cloud features, that is, is the image feature reconstructed by dynamic features, Point cloud features reconstructed from dynamic features.

[0011] Preferably, S4 is specifically: S4.1: Calculate adaptive fusion weights; An adaptive weight generation module is introduced to calculate the weight of each modality at different scales based on the reconstructed image features and point cloud features. ,in is an MLP neural network module, the input of the MLP neural network module is the reconstructed image features and the point cloud features, and the output of the MLP neural network module is the fusion weight; S4.2: Based on the adaptive fusion weight, the reconstructed image features and the point cloud features are multimodally fused to obtain the final fusion features. ; Wherein, the S4.2 is expressed by the formula: .

[0012] Preferably, the S2.1 is specifically: transforming the image features by matrix multiplication Mapping to query vector ;Right now ;in, is a learnable matrix for mapping the image features to a query vector, is the dimension of the query vector after mapping; S2.2 is specifically: transforming the image features into Mapping to key vector ;Right now ;in, is a learnable matrix for mapping the point cloud features to key vectors, is the dimension of the query vector after mapping; S2.3 is specifically: transforming the image features into Mapping to a vector of values ;Right now ;in, is a learnable matrix for mapping the point cloud features to a value vector, is the dimension of the mapped value vector.

[0013] Preferably, in S2.4, the relationship between the image features and the point cloud features is represented by an attention weight matrix; based on the query vector and key vector Calculating the attention weight matrix; The query vector and key vector The calculation of the attention weight matrix is ​​specifically as follows: using the dot product attention mechanism, the calculation based on the query vector and key vector The dot product is normalized by the softmax function to obtain the attention weight matrix ; in, , .

[0014] According to another aspect of the present invention, a multimodal data fusion system is provided, wherein the system adopts the multimodal data fusion method described above, and the system comprises: A feature extraction module, used to perform multi-scale feature extraction on the multimodal data to obtain three-dimensional point cloud data features and two-dimensional image data features; A cross-modal collaborative attention mechanism module, used for dynamically calculating the relationship between the three-dimensional point cloud features and the image data features based on the cross-modal collaborative attention mechanism; A dynamic feature reconstruction module, used for dynamically reconstructing the image features and the point cloud features; The multimodal fusion module is used to perform multimodal fusion on the reconstructed image features and the point cloud features to obtain the final fusion features. The advantages and beneficial effects of the present invention are: The present invention uses a dynamic cross-modal attention mechanism to accurately capture the relationship between image and point cloud features, avoiding information loss due to inaccurate projection, especially in long-distance areas; the present invention uses an adaptive attention mechanism to enable dynamic interaction between image and point cloud features, achieve deep modal collaboration, and ensure that all scale features of the two modalities can be effectively utilized; the present invention uses an adaptive weight generation mechanism to balance the contributions of different modalities in fusion and avoid the problem of information imbalance; at the same time, the present invention optimizes the fusion process, reduces the computational complexity through multi-scale dynamic fusion, and improves real-time performance and resource efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the description of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of a multimodal data fusion method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a multimodal data fusion system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Attached Figure 1 A flowchart of a multimodal data fusion method is shown in the attached Figure 1 As shown, a multimodal data fusion method, wherein the multimodal data includes three-dimensional point cloud data and two-dimensional image data, and the method includes the following steps: S1: performing multi-scale feature extraction on the multimodal data to obtain three-dimensional point cloud data features and two-dimensional image data features; Wherein, performing multi-scale feature extraction on the multimodal data includes performing feature extraction on the two-dimensional image data and performing feature extraction on the three-dimensional point cloud data; The ‌Convolutional Neural Network Model (CNN)‌ is a neural network model widely used in the field of image recognition and processing. Its basic structure includes input layer, convolution layer, pooling layer and fully connected layer. Among them, the ‌input layer‌ is used to receive the original image data and convert it into a form that the neural network can process. The convolution layer‌ is used to extract the features of the image through convolution operation. The convolution operation obtains the feature map by convolving the image with a set of convolution kernels. The convolution kernel is a small matrix that is multiplied element by element with the image through a sliding window and summed to obtain each pixel value in the feature map. The convolution operation can preserve the spatial relationship and local features of the image. Through different convolution kernels, CNN can learn different features, such as edges, textures and shapes. The pooling layer‌ is used to reduce the size of the feature map and improve computational efficiency; common pooling operations include maximum pooling and average pooling. The purpose of the pooling operation is to reduce the size of the feature map, improve computational efficiency, and have a certain degree of translation invariance. Through the pooling operation, CNN can abstract the details of the image, thereby better capturing the overall features of the image; the fully connected layer is used to map the feature map to the output layer to achieve image classification or regression. Each neuron in the fully connected layer is connected to all neurons in the previous layer, and the discrimination of different categories is achieved by learning weight parameters. The fully connected layer plays a decision-making role in CNN. Through the learned weight parameters, the information of the feature map can be converted into a prediction of the image category; Therefore, the feature extraction of the two-dimensional image data is specifically performed as follows: a convolutional neural network model (CNN) is used to extract multi-scale image features of the two-dimensional image data. After the two-dimensional image data passes through different convolutional layers of the convolutional neural network, a series of feature maps are generated. The convolutional neural network model captures the local information and global information of the two-dimensional image data respectively; the image features of each scale are represented as , , where n is the number of feature vectors, d is the dimension of the feature, and s is the index of multi-scale processing, which is used to distinguish features extracted at different scales.

[0019] PointNet++ is a network model used in deep learning to process 3D point cloud data. PointNet++ is an improvement on PointNet, which solves some limitations of PointNet in processing point cloud data. The main improvements of PointNet++ are: Local feature extraction: PointNet++ introduces sampling and grouping to consider local domain features, while PointNet only considers single-point features and cannot represent local structures well.

[0020] ‌Hierarchical structure‌: PointNet++ adopts a hierarchical structure, which can effectively extract local features of different regions based on different receptive field sizes. However, the global feature of PointNet is directly obtained by max pooling, which is prone to information loss.

[0021] ‌Rotation invariance‌: PointNet++ uses local relative coordinates for feature extraction, eliminating the TNet network, while PointNet uses TNet to ensure the invariance of point cloud feature rotation.

[0022] ‌Sample unevenness problem‌: PointNet++ proposed the multi-scale method MSG and the multi-level method MRG to solve the sample unevenness problem, which PointNet did not deal with.

[0023] ‌Segmentation network structure‌: PointNet++ adopts the Encoder-Decoder structure, and the features are connected through skip link concatenation, while PointNet directly integrates the global feature and local embedding features.

[0024] Therefore, the feature extraction of 3D point cloud data is as follows: using the point cloud processing network of PointNet++, layered sampling and feature extraction are performed on the 3D point cloud data, and geometric features of point clouds are generated from different scales. The point cloud features of each scale are represented as , , where m is the number of points, d is the feature dimension, and s is the index of multi-scale processing, which is used to distinguish features extracted at different scales; S2: dynamically calculating the relationship between the three-dimensional point cloud features and the image data features based on a cross-modal collaborative attention mechanism; The dynamic calculation of the relationship between the three-dimensional point cloud features and the image data features based on the cross-modal collaborative attention mechanism is specifically: S2.1: The image features Mapping to query vector ; Among them, the S2.1 is specifically: by matrix multiplication, the image features Mapping to query vector ;Right now ;in, is a learnable matrix for mapping the image features to a query vector, is the dimension of the query vector after mapping; S2.2: The point cloud features Mapping to key vector ; Among them, the S2.2 is specifically: by matrix multiplication, the image features Mapping to key vector ;Right now ;in, is a learnable matrix for mapping the point cloud features to key vectors, is the dimension of the query vector after mapping; S2.3: The point cloud features Mapping to a vector of values ; Among them, the S2.3 is specifically: by matrix multiplication, the image features Mapping to a vector of values ;Right now ;in, is a learnable matrix for mapping the point cloud features to a value vector, is the dimension of the mapped value vector; S2.4: Calculating the relationship between the image features and the point cloud features; In this step, the relationship between the image features and the point cloud features is represented by the attention weight matrix; specifically, based on the query vector and key vector Calculating the attention weight matrix; Furthermore, the query vector based and key vector The calculation of the attention weight matrix is ​​specifically as follows: Use the dot product attention mechanism to calculate the query vector and key vector The dot product is normalized by the softmax function to obtain the attention weight matrix ; in, , ; S2.5: Using the attention weight matrix Perform weighted summation on the value vector to obtain image features combined with point cloud information ; Wherein, the formula of S2.5 is: ; Through the above steps, the image features after fusion of point cloud information are generated This feature not only contains the information of the original image, but also combines the point cloud feature information related to it.

[0025] S3: dynamically reconstructing the image features and the point cloud features; Among them, S3 is specifically: according to the attention weight matrix Dynamic feature reconstruction is performed on the image features and the point cloud features, that is, , is the image feature after dynamic feature reconstruction, The point cloud features are reconstructed from dynamic features. Through the attention mechanism, image features can be reconstructed in combination with point cloud information. Point cloud features also absorb image information. In the feature reconstruction process, complementary information between modalities is fully utilized, thereby reducing information loss and the limitations of single-modal features.

[0026] S4: Perform multimodal fusion on the reconstructed image features and the point cloud features to obtain the final fusion features ; Wherein, the S4 is specifically: S4.1: Calculate adaptive fusion weights; In order to balance the contribution of multimodal features, an adaptive weight generation module is introduced to calculate the weight of each modality at different scales according to the reconstructed image features and the point cloud features. ,in is an MLP neural network module, the input of the MLP neural network module is the reconstructed image features and the point cloud features, and the output of the MLP neural network module is the fusion weight; Through adaptive adjustment, features of different scales can be flexibly assigned different weights according to task requirements, preventing a certain modality from dominating the fusion process.

[0027] S4.2: Based on the adaptive fusion weight, the reconstructed image features and the point cloud features are multimodally fused to obtain the final fusion features. ; Wherein, the S4.2 is expressed by the formula: The fused features represent the deep interaction and complementary information of images and point clouds at different scales. Multi-scale fusion ensures the global and local integrity of the fused features, avoids information loss, and makes the features more expressive.

[0028] Through the above solution, this embodiment has the following advantages: 1) Avoid information loss: Existing early fusion methods are prone to information loss due to point cloud sparsity and field of view limitations. This technology uses a dynamic cross-modal attention mechanism to accurately capture the relationship between image and point cloud features, avoiding information loss caused by inaccurate projection, especially in long-distance areas.

[0029] 2) Enhanced modal collaboration: Existing mid-term fusion methods lack deep synergy between different modalities. This technology uses an adaptive attention mechanism to enable dynamic interaction between image and point cloud features, achieve deep modal collaboration, and ensure that all scale features of the two modalities can be effectively utilized.

[0030] 3) Balance the dominant effect of modalities: In existing methods, due to differences in resolution, one modality may dominate, affecting the fusion effect. This technology balances the contribution of different modalities in fusion through an adaptive weight generation mechanism, avoiding the problem of information imbalance.

[0031] 4) Simplify calculation and improve real-time performance: Although multi-phase fusion technology is comprehensive, it is complex in design and has high calculation cost. This technology optimizes the fusion process, reduces the calculation complexity through multi-scale dynamic fusion, and improves real-time performance and resource efficiency.

[0032] Embodiment 2: This embodiment includes a multi-modal data fusion system, Figure 2 A structural diagram of a multimodal data fusion system is shown in the attached Figure 2 As shown, the system adopts the multimodal data fusion method of embodiment 1, and the system includes: A feature extraction module, used to perform multi-scale feature extraction on the multimodal data to obtain three-dimensional point cloud data features and two-dimensional image data features; A cross-modal collaborative attention mechanism module, used for dynamically calculating the relationship between the three-dimensional point cloud features and the image data features based on the cross-modal collaborative attention mechanism; A dynamic feature reconstruction module, used for dynamically reconstructing the image features and the point cloud features; The multimodal fusion module is used to perform multimodal fusion on the reconstructed image features and the point cloud features to obtain the final fusion features. Embodiment 3: This embodiment includes a computer-readable storage medium, on which a data processing program is stored, and the data processing program is executed by a processor to implement a multimodal data fusion method according to embodiment 1.

[0033] It should be understood by those skilled in the art that the embodiments of this article can be provided as methods, devices (equipment), or computer program products. Therefore, this article can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Including but not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0034] This document is described with reference to flowcharts and / or block diagrams of methods, apparatuses (devices), and computer program products according to the embodiments of this document. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0035] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0036] The embodiments and / or implementation methods described above are only used to illustrate the preferred embodiments and / or implementation methods of the technology of the present invention, and are not intended to impose any formal restrictions on the implementation methods of the technology of the present invention. Any person skilled in the art may make some changes or modifications to other equivalent embodiments without departing from the scope of the technical means disclosed in the content of the present invention, but they should still be regarded as the technology or embodiments that are essentially the same as the present invention. Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described here, and that various obvious changes, readjustments and substitutions can be made to those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention is described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for fusion of multimodal data, wherein the multimodal data includes three-dimensional point cloud data and two-dimensional image data, characterized in that: The method comprises the following steps: S1: performing multi-scale feature extraction on the multimodal data to obtain three-dimensional point cloud data features and two-dimensional image data features; S2: dynamically calculating the relationship between the three-dimensional point cloud features and the image data features based on a cross-modal collaborative attention mechanism; S3: dynamically reconstructing the image features and the point cloud features; S4: Perform multimodal fusion on the reconstructed image features and the point cloud features to obtain the final fusion features .

2. A multimodal data fusion method according to claim 1, characterized in that: Performing multi-scale feature extraction on the multimodal data includes performing feature extraction on the two-dimensional image data and performing feature extraction on the three-dimensional point cloud data.

3. A multimodal data fusion method according to claim 2, characterized in that: The feature extraction of the two-dimensional image data is specifically performed as follows: a convolutional neural network model (CNN) is used to extract multi-scale image features of the two-dimensional image data. After the two-dimensional image data passes through different convolutional layers of the convolutional neural network, a series of feature maps are generated. The convolutional neural network model captures the local information and global information of the two-dimensional image data respectively; the image features of each scale are represented as , , where n is the number of feature vectors, d is the dimension of the feature, and s is the index of multi-scale processing, which is used to distinguish features extracted at different scales.

4. A multimodal data fusion method according to claim 2 or 3, characterized in that: The feature extraction of 3D point cloud data is as follows: using the point cloud processing network of PointNet++, performing hierarchical sampling and feature extraction on the 3D point cloud data, generating geometric features of point clouds at different scales, and representing the point cloud features of each scale as follows: , , where m is the number of points, d is the feature dimension, and s is the index of multi-scale processing, which is used to distinguish features extracted at different scales.

5. The multimodal data fusion method according to claim 1, characterized in that: In S2, the dynamic calculation of the relationship between the three-dimensional point cloud features and the image data features based on the cross-modal collaborative attention mechanism is specifically: S2.1: The image features Mapping to query vector ; S2.2: The point cloud features Mapping to key vector ; S2.3: The point cloud features Mapping to a vector of values ; S2.4: Calculating the relationship between the image features and the point cloud features; S2.5: Using the attention weight matrix Perform weighted summation on the value vector to obtain image features combined with point cloud information .

6. A multimodal data fusion method according to claim 5, characterized in that: The S3 is specifically: according to the attention weight matrix The image features and the point cloud features are dynamically reconstructed, that is, is the image feature reconstructed by dynamic features, Point cloud features reconstructed from dynamic features.

7. The multimodal data fusion method according to claim 1, characterized in that: The S4 is specifically: S4.1: Calculate adaptive fusion weights; An adaptive weight generation module is introduced to calculate the weight of each modality at different scales based on the reconstructed image features and point cloud features. ,in is an MLP neural network module, the input of the MLP neural network module is the reconstructed image features and the point cloud features, and the output of the MLP neural network module is the fusion weight; S4.2: Based on the adaptive fusion weight, the reconstructed image features and the point cloud features are multimodally fused to obtain the final fusion features. ; Wherein, the S4.2 is expressed by the formula: 。 8. The multimodal data fusion method according to claim 5, characterized in that: The S2.1 is specifically: transforming the image features by matrix multiplication Mapping to query vector ;Right now ;in, is a learnable matrix for mapping the image features to a query vector, is the dimension of the query vector after mapping; S2.2 is specifically: transforming the image features into Mapping to key vector ;Right now ;in, is a learnable matrix for mapping the point cloud features to key vectors, is the dimension of the query vector after mapping; S2.3 is specifically: transforming the image features into Mapping to a vector of values ;Right now ;in, is a learnable matrix for mapping the point cloud features to a value vector, is the dimension of the mapped value vector.

9. The multimodal data fusion method according to claim 5, characterized in that: In S2.4, the relationship between the image features and the point cloud features is represented by an attention weight matrix; based on the query vector and key vector Calculating the attention weight matrix; The query vector and key vector The calculation of the attention weight matrix is ​​specifically as follows: using the dot product attention mechanism, the calculation based on the query vector and key vector The dot product is normalized by the softmax function to obtain the attention weight matrix ; in, , .

10. A multimodal data fusion system for three-dimensional point cloud data processing, characterized in that: The system adopts a multimodal data fusion method according to any one of claims 1 to 9, and the system comprises: A feature extraction module, used to perform multi-scale feature extraction on the multimodal data to obtain three-dimensional point cloud data features and two-dimensional image data features; A cross-modal collaborative attention mechanism module, used for dynamically calculating the relationship between the three-dimensional point cloud features and the image data features based on the cross-modal collaborative attention mechanism; A dynamic feature reconstruction module, used for dynamically reconstructing the image features and the point cloud features; The multimodal fusion module is used to perform multimodal fusion on the reconstructed image features and the point cloud features to obtain final fused features.

Images