Multi-modal data enhancement and fusion method and device, equipment and medium

Abstract

The invention relates to an image processing technology, and discloses a multi-modal data enhancement and fusion method, which comprises the following steps: acquiring multi-modal data, and calculating a Gaussian noise vector of each modal data; inputting the Gaussian noise vector of each type of modal data into a multi-modal joint generation network for data generation to obtain a multi-modal data set; extracting modal features of the RGB image data and the Thermal image data in the multi-modal data set, and fusing the modal features of the RGB image data and the Thermal image data in a cross-modal manner to obtain initial RGBThermal fusion features; extracting LiDAR point cloud features in the multi-modal data set; and carrying out feature fusion on the LiDAR point cloud feature and the initial RGBThermal fusion feature to obtain a multi-mode BEV visual angle feature. The invention further provides a multi-modal data enhancement and fusion device, equipment and a medium. According to the invention, the accuracy of environment detection and recognition can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a multimodal data enhancement and fusion method, device, equipment and medium. Background Art

[0002] One of the core tasks of autonomous driving technology is environmental perception, that is, accurately identifying surrounding objects, obstacles and road conditions. In order to achieve efficient and safe perception in complex driving environments, autonomous driving systems usually rely on the fusion of multiple sensor modalities, such as LiDAR (laser radar), RGB images (visible light images), thermal images (Thermal) and radar. LiDAR can provide accurate depth information to help identify the geometry and distance of objects, but its point cloud data is susceptible to interference in bad weather or strong light, affecting accuracy. Cameras provide rich visual information, such as color and texture, but due to the lack of depth information and poor performance in low light, at night or in strong light, their application in three-dimensional object detection is limited.

[0003] Existing technical solutions use a multimodal data fusion framework based on a bird's-eye view (BEV). The core goal is to uniformly map LiIDAR point cloud data and other modal data into the BEV space, improving environmental perception through feature fusion. This typically involves three stages: sensor data encoding, BEV space conversion, feature fusion, and task-driven implementation.

[0004] (1) Multimodal feature extraction and perspective conversion

[0005] In multimodal data fusion, feature extraction is the basis for improving the system's perception capabilities. The BEVFusion method achieves effective cross-modal data integration by extracting features from LiDAR point clouds and camera images. BEVFusion extracts image features through a deep convolutional network and performs perspective conversion on this basis, mapping the image features to BEV space. This perspective conversion module uses two common technologies: one is Inverse Perspective Mapping (IPM) based on geometric rules. This method reversely projects the pixels in the image onto a plane through geometric transformation, maintaining the geometric relationship of the object; the other is a deformable attention mechanism based on deep learning, such as LSS (Learnable Spatial Sampling) or BEVFormer. These methods optimize the mapping from image to BEV space through deep learning models and can automatically adapt to different scenes and perspectives, thereby providing BEV features with geometric consistency. LiDAR point clouds use laser beams to obtain distance information in the environment and form dense three-dimensional point cloud data. To effectively utilize LiDAR data in the BEV space, a common processing method is to discretize the 3D point cloud data into voxels using voxelization techniques (such as VoxelNet and PointPillars). Voxelization divides LiDAR point cloud data into small cubic grid units (voxels) and calculates eigenvalues ​​for each voxel, effectively reducing the amount of computation required while preserving 3D spatial information.

[0006] (2) BEV spatial feature fusion

[0007] In the BEV space, the features of the camera and LiDAR need to be aligned in the same space. BEVFusion uses feature splicing or weighted addition to fuse the BEV features of the LiDAR and camera into a unified feature vector. Specifically, the number of channels of the LiDAR and camera features are merged to form a multi-channel BEV feature map. Next, these fused features are further processed by the BEV Encoder (such as ResNet+FPN) to generate an enhanced BEV feature representation. This fusion method effectively combines the geometric information of the LiDAR and the semantic information of the camera image, so that the final BEV feature map not only has depth information, but also contains rich semantic information, thereby improving the perception accuracy.

[0008] Existing technologies lack a multimodal data generation mechanism, which limits data generalization capabilities. The global contextual association between multiple modalities is not considered in the BEV multimodal fusion strategy, resulting in unstable fusion information for distant targets and complex environments, and problems such as missing key target features or mismatched modal information, which in turn leads to low accuracy in environmental detection and recognition. Summary of the Invention

[0009] The present invention provides a multimodal data enhancement and fusion method, device, equipment and medium, which can improve the accuracy of environmental detection and recognition.

[0010] To achieve the above objectives, the present invention provides a multimodal data enhancement and fusion method, comprising:

[0011] Acquire multimodal data and calculate the Gaussian noise vector for each modal data, where the multimodal data includes RGB image data, Thermal image data, and LiDAR point cloud data;

[0012] Inputting the Gaussian noise vector of the multimodal data into a multimodal joint generation network to obtain a multimodal data set, wherein the multimodal data set includes RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data;

[0013] Extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and cross-modally fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain the initial RGB_Thermal fusion features;

[0014] Convert LiDAR point cloud generated data into LiDAR point cloud features;

[0015] The LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain the multimodal BEV perspective features, and the environment is recognized based on the multimodal BEV perspective features.

[0016] Optionally, inputting the Gaussian noise vector of the multimodal data into a multimodal joint generation network to obtain a multimodal data set includes:

[0017] The Gaussian noise vector of each modal data is normalized using the fully connected layer of the multimodal joint generation network to obtain the normalized features corresponding to each modal data;

[0018] The convolutional layer of the multimodal joint generation network is used to upsample the normalized features corresponding to each modality data to obtain a high-resolution feature map corresponding to each modality data;

[0019] A graph neural network is used to convert the high-resolution feature map corresponding to each modality data into a heterogeneous graph, and a linear activation function is used to update the nodes in the heterogeneous graph to obtain an updated heterogeneous graph. The nodes in the heterogeneous graph are the feature points of different modalities, and the edges in the heterogeneous graph are the connections between different modalities.

[0020] An adaptive attention mechanism is calculated based on the updated heterogeneous graph, and the adaptive attention mechanism is used to enhance the feature expression of each modality data to obtain enhanced features of each modality data;

[0021] The enhanced features of each modality data are transposed and convolved using the convolutional layer corresponding to the modality in the multimodal joint generation network to obtain generated data of different modal data;

[0022] The generated data of different modal data are integrated to obtain a multimodal data set.

[0023] Optionally, after integrating the generated data of different modal data to obtain a multimodal data group, the method further includes using a discriminator of a multimodal joint generation network to perform coordination and consistency processing on the multimodal data group.

[0024] Optionally, the step of performing coordination and consistency processing on the multimodal data set using a discriminator of the multimodal joint generative network includes:

[0025] Input the multimodal data set and the real multimodal data into the discriminator to obtain the probability values ​​of different modes corresponding to the multimodal data set and the real multimodal data;

[0026] Calculate modal embedding codes of different modalities based on the probability values ​​of different modalities corresponding to the multimodal data set and the real multimodal data;

[0027] Multimodal data sets are harmonized and consistent based on modal embedding coding.

[0028] Optionally, extracting modal features corresponding to the RGB image generation data and modal features corresponding to the Thermal image generation data includes:

[0029] Converting RGB image generation data and Thermal image generation data into RGB image feature matrix and Thermal image feature matrix;

[0030] Perform image convolution processing according to the number of channels of the RGB image feature matrix to obtain preliminary RGB image features;

[0031] Perform image convolution processing according to the number of channels of the Thermal image feature matrix to obtain preliminary Thermal image features;

[0032] The preliminary RGB image features are expanded using a dilated convolutional layer with a first expansion rate to obtain modal features corresponding to the RGB image generation data;

[0033] The preliminary thermal image features are expanded using the dilated convolutional layer with the second expansion rate to obtain the modal features corresponding to the thermal image generation data.

[0034] Optionally, the cross-modal fusion of the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain an initial RGB_Thermal fusion feature includes:

[0035] The cross-attention mechanism is used to calculate the correlation between the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, respectively, to obtain the correlation matrix between the modalities corresponding to the RGB image data and the correlation matrix between the modalities corresponding to the Thermal image data;

[0036] The modal features corresponding to the RGB image generation data are extracted using a preset number of convolutional layers to obtain a preset number of RGB image feature matrices;

[0037] The modal features corresponding to the thermal image generation data are extracted using a preset number of convolutional layers to obtain a preset number of thermal image feature matrices;

[0038] Multiply the inter-modality correlation matrix corresponding to the RGB image data by a preset number of RGB image feature matrices element by element to obtain the RGB image global features;

[0039] Multiply the correlation matrix between the modalities corresponding to the thermal image data by a preset number of thermal image feature matrices element by element to obtain the global features of the thermal image;

[0040] Multiply the RGB image global features and the modal features corresponding to the RGB image generation data element by element to obtain the final RGB image features;

[0041] The final Thermal image feature is obtained by multiplying the global feature of the Thermal image with the modal feature corresponding to the Thermal image generation data element by element;

[0042] The final RGB image features are adaptively fused with the final Thermal image features to obtain the initial RGB_Thermal fusion features.

[0043] Optionally, the step of fusing the LiDAR point cloud features with the initial RGB_Thermal fusion features to obtain the multimodal BEV viewing angle features includes:

[0044] Map the LiDAR point cloud features into the query vector in the dual attention mechanism, and map the initial RGB_Thermal fusion features into the key vector and value vector in the dual attention mechanism;

[0045] The cross attention in the dual attention mechanism is used to calculate the attention degree of the LiDAR point cloud features and the initial RGB_Thermal fusion features and perform preliminary fusion to obtain the initial multimodal fusion features;

[0046] Use the self-attention in the dual attention mechanism to calculate the self-attention features in the initial multimodal fusion features;

[0047] The initial multimodal fusion features and self-attention features are concatenated to obtain multimodal joint features;

[0048] In order to solve the above problems, the present invention further provides a multimodal data enhancement and fusion device, the device comprising:

[0049] A data generation module is configured to acquire multimodal data and calculate a Gaussian noise vector for each modal data, wherein the multimodal data includes RGB image data, Thermal image data, and LiDAR point cloud data; input the Gaussian noise vector of the multimodal data into a multimodal joint generation network to obtain a multimodal data set, wherein the multimodal data set includes RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data;

[0050] A feature fusion module is used to extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and to fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data across modalities to obtain the initial RGB_Thermal fusion features;

[0051] The environment recognition module is used to convert LiDAR point cloud generated data into LiDAR point cloud features; the LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain multimodal BEV perspective features, and the environment recognition is performed based on the multimodal BEV perspective features.

[0052] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0053] at least one processor; and,

[0054] a memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the multimodal data enhancement and fusion method described above.

[0056] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one computer program. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned multimodal data enhancement and fusion method.

[0057] The present invention generates multimodal data sets by inputting Gaussian noise vectors from each modal data set into a multimodal joint generative network for data generation. This generates physically consistent multimodal data, significantly improving data diversity and coverage. Furthermore, the present invention extracts modal features corresponding to the RGB image generation data and the thermal image generation data from the multimodal data set, and cross-modally fuses these modal features to generate initial RGB-Thermal fusion features. This achieves a fusion of the complementary advantages of visible light and thermal imaging, improving the robustness of the visual modality under challenging lighting conditions. Furthermore, the present invention extracts LiDAR point cloud generation data from the multimodal data set and converts the point cloud generation data into LiDAR point cloud features. This effectively extracts precise geometric structure information from the generated point cloud, providing basic depth information for the BEV space. Furthermore, the LiDAR point cloud features are fused with the initial RGB-Thermal fusion features to generate multimodal BEV perspective features. Environmental recognition based on the multimodal BEV perspective features can improve the accuracy of environmental detection and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of a multimodal data enhancement and fusion method provided by an embodiment of the present invention;

[0059] Figure 2 A network structure diagram of a multimodal data enhancement and fusion method provided in one embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a feature fusion structure of the RGB_Thermal fusion feature of the multimodal data enhancement and fusion method provided in one embodiment of the present invention;

[0061] Figure 4 A schematic diagram of a cross-modal complementary structure of a multimodal data enhancement and fusion method provided in one embodiment of the present invention;

[0062] Figure 5 A schematic diagram of LiDAR point cloud feature generation for a multimodal data enhancement and fusion method provided in one embodiment of the present invention;

[0063] Figure 6A schematic diagram of the cross-attention mechanism structure of a multimodal data enhancement and fusion method provided in one embodiment of the present invention;

[0064] Figure 7 A functional module diagram of a multimodal data enhancement and fusion device provided by one embodiment of the present invention;

[0065] Figure 8 A schematic structural diagram of an electronic device for implementing a multimodal data enhancement and fusion method provided in one embodiment of the present invention.

[0066] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] The embodiment of the present application provides a method for multimodal data enhancement and fusion thereof. The execution subject of the multimodal data enhancement and fusion method thereof includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the multimodal data enhancement and fusion method thereof can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0069] Reference Figure 1 FIG. 1 is a flow chart of a multimodal data enhancement and fusion method according to an embodiment of the present invention. In this embodiment, the multimodal data enhancement and fusion method includes:

[0070] S1. Acquire multimodal data and calculate the Gaussian noise vector of each modal data, wherein the multimodal data includes RGB image data, Thermal image data, and LiDAR point cloud data.

[0071] In the embodiment of the present invention, the Gaussian noise vector refers to a one-dimensional vector generated by a multimodal joint generation network and obeys a standard normal distribution.

[0072] S2. Input the Gaussian noise vector of the multimodal data into the multimodal joint generation network to obtain a multimodal data group, wherein the multimodal data group includes RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data.

[0073] In an embodiment of the present invention, the multimodal joint generation network may be a DCGAN framework network.

[0074] As an embodiment of the present invention, a Gaussian noise vector of multimodal data is input into a multimodal joint generation network to obtain a multimodal data set, including:

[0075] The Gaussian noise vector of each modal data is normalized using the fully connected layer of the multimodal joint generation network to obtain the normalized features corresponding to each modal data;

[0076] The convolutional layer of the multimodal joint generation network is used to upsample the normalized features corresponding to each modality data to obtain a high-resolution feature map corresponding to each modality data;

[0077] A graph neural network is used to convert the high-resolution feature map corresponding to each modality data into a heterogeneous graph, and a linear activation function is used to update the nodes in the heterogeneous graph to obtain an updated heterogeneous graph. The nodes in the heterogeneous graph are the feature points of different modalities, and the edges in the heterogeneous graph are the connections between different modalities.

[0078] An adaptive attention mechanism is calculated based on the updated heterogeneous graph, and the adaptive attention mechanism is used to enhance the feature expression of each modality data to obtain enhanced features of each modality data;

[0079] The enhanced features of each modality data are transposed and convolved using the convolutional layer corresponding to the modality in the multimodal joint generation network to obtain generated data of different modal data;

[0080] The generated data of different modal data are integrated to obtain a multimodal data set.

[0081] Exemplarily, the Gaussian noise vector of each modal data is input into the multimodal joint generation network to generate data to obtain a multimodal data set, which can be achieved by the following implementation steps:

[0082] a. First, the Gaussian noise vector of each modality data is processed through a fully connected layer and batch normalization, as follows:

[0083]

[0084] Among them, v mThe index subscript includes {RGB, Thermal, LiDAR}. The calculation formula for each modality is the same, but the subscript is different. In fact, it is:

[0085] Where W is the weight matrix; b is the bias vector; H is the feature representation after processing by the fully connected layer.

[0086]

[0087] where u B is the batch mean, σ B is the batch standard deviation; γ is the first learnable parameter and β is the second learnable parameter, which is used to control the scaling and translation after normalization.

[0088] Then go through three layers of convolutional network:

[0089]

[0090] v m ∈{RGB,Therml,LiDAR}, where ConvT2d represents a two-dimensional transposed convolution operation; ReLU(·) is used as the activation function; and the output size is H. out =S(H in -1)+K-2P; S is the stride (Stride=2), K is the convolution kernel size (KernelSize=4); P is padding (Padding=1).

[0091] b. To achieve complementary information transmission between modalities, GNN is used for feature propagation to construct a heterogeneous graph G = (V, E), where nodes V represent feature points of different modalities and edges E represent the association between modalities. The adjacency matrix between modalities is defined as:

[0092]

[0093] Among them, σ is used to control the scale of the adjacency matrix, and Represents features of two different modes.

[0094] Each node is updated as follows:

[0095]

[0096] Among them, sigmoid(·) is a nonlinear activation function, W is the first learning parameter, and t represents the number of updates. For the adaptive attention mechanism:

[0097]

[0098] Among them, W is the first learning parameter, a is the second learning parameter, || represents the vector splicing operation, and Representing the features of two different modes, LeakyRelu(·) is used as the activation function to allow negative numbers to pass, as shown below:

[0099]

[0100] After repeating step b three times, the modality group is output to the self-attention mechanism to enhance the information expression ability of each modality.

[0101] c. The three modalities then enter independent convolutional layers. The RGB and Therml image features enter the 2D transposed convolution, while the LiDAR point cloud requires the use of 3D transposed convolution to generate three modal data.

[0102]

[0103] Tanh(·) and sigmoid(·) are activation functions; represents the RGB features after three iterations, Represents the Therml feature after three iterations, Represents the LiDAR modal features after three iterations.

[0104] Furthermore, after the generation data of different modal data are integrated to obtain a multimodal data group, it also includes using a discriminator of a multimodal joint generation network to perform coordination and consistency processing on the multimodal data group.

[0105] Furthermore, the discriminator of the multimodal joint generation network is used to coordinate and coordinate the multimodal data set, including:

[0106] Input the multimodal data set and the real multimodal data into the discriminator to obtain the probability values ​​of different modes corresponding to the multimodal data set and the real multimodal data;

[0107] Calculate modal embedding codes of different modalities based on the probability values ​​of different modalities corresponding to the multimodal data set and the real multimodal data;

[0108] Multimodal data sets are harmonized and consistent based on modal embedding coding.

[0109] For example, the discriminator of the multimodal joint generative network is used to perform coordination and consistency processing on the multimodal data set, and the following implementation steps can be adopted:

[0110] Step 1. Generate multimodal data set and real multimodal data The improved discriminator obtains the corresponding probability value. The improved discriminator adds a self-attention mechanism. The improved discriminator includes four convolutional layers, increasing the number of channels layer by layer, and using convolution with a step size of 2 for downsampling. For example, the first convolution operation is: BatchNorm is added after each convolution layer. The last layer does not use BatchNorm and directly uses Sigmoid to output the probability:

[0111]

[0112] The above formula is the sigmoid activation function, where W represents the weight matrix of the fully connected layer, b is the bias term of the fully connected layer, To generate the features extracted by the fourth layer of convolution, the features extracted from the real data are expressed as

[0113] The goal of the discriminator is to make the real data Generate data

[0114] Step 2. Get features after the fourth convolution layer The modal embedding code is dynamically calculated based on the real data and generated data. The characteristics of each modality are the dynamic weighting of the real data and the generated data.

[0115]

[0116] where λ r is the weighting coefficient, λ f =1-λ r ; T represents the total number of iterations, which is the current number of iterations. Then calculate the modal embedding code:

[0117]

[0118] in is the attention weight, As the third learnable parameter, exp(·) ensures that the attention score is positive.

[0119] S3. Extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and cross-modally fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain the initial RGB_Thermal fusion features.

[0120] As an embodiment of the present invention, extracting modal features corresponding to RGB image generation data and modal features corresponding to Thermal image generation data in a multimodal data group includes:

[0121] Converting RGB image generation data and Thermal image generation data into RGB image feature matrix and Thermal image feature matrix;

[0122] Perform image convolution processing according to the number of channels of the RGB image feature matrix to obtain preliminary RGB image features;

[0123] Perform image convolution processing according to the number of channels of the Thermal image feature matrix to obtain preliminary Thermal image features;

[0124] The preliminary RGB image features are expanded using a dilated convolutional layer with a first expansion rate to obtain modal features corresponding to the RGB image generation data;

[0125] The preliminary thermal image features are expanded using the dilated convolutional layer with the second expansion rate to obtain the modal features corresponding to the thermal image generation data.

[0126] Exemplarily, extracting the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data in the multimodal data group can be achieved by the following implementation steps:

[0127] The RGB image and thermal imaging image are represented by feature matrices, where H and W are the height and width of the image respectively, and C RGB and C Thermal are the number of channels of RGB images and thermal images, respectively. Typically, RGB images have 3 channels, while thermal images typically have 1 channel. During the preprocessing phase, preliminary feature extraction is performed on each modality image through convolution. Specifically, a convolution operation is performed on each channel of each modality image to obtain preliminary convolution features:

[0128]

[0129] Among them, Conv(·) represents the convolution operation, and ReLU(·) is the activation function used to increase the nonlinear expression ability of the network.

[0130] To fuse local information at multiple scales, the initially extracted RGB and thermal image features are processed through dilated convolution layers with different expansion rates to capture information at different scales. To further extract multi-scale features, the RGB and thermal image features are convolved using multiple convolution kernels of different sizes (3×3, 5×5, and 7×7). The steps for obtaining convolution features at different scales are as follows:

[0131] a. RGB / thermal imaging branch feature extraction:

[0132] First, the input RGB image features are convolved and multi-scale feature information is obtained by dilating the convolution layer, v′ m ∈{RGB,Therml} process is as follows:

[0133] Among them, because v′ m Instead of RGB and Thermal symbols, for or It has been obtained through the above calculation. All the following formulas use v′ m The principles of the representation are the same. The expanded description is:

[0134]

[0135] Among them, D1_Conv(·) is the dilation convolution operation with a dilation rate of 1, and cat(·) is the splicing operation. Represents the output features after the first layer of dilated convolution.

[0136] Furthermore, the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data are cross-modally fused to obtain the initial RGB_Thermal fusion features, including:

[0137] The cross-attention mechanism is used to calculate the correlation between the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, respectively, to obtain the correlation matrix between the modalities corresponding to the RGB image data and the correlation matrix between the modalities corresponding to the Thermal image data;

[0138] The modal features corresponding to the RGB image generation data are extracted using a preset number of convolutional layers to obtain a preset number of RGB image feature matrices;

[0139] The modal features corresponding to the thermal image generation data are extracted using a preset number of convolutional layers to obtain a preset number of thermal image feature matrices;

[0140] Multiply the inter-modality correlation matrix corresponding to the RGB image data by a preset number of RGB image feature matrices element by element to obtain the RGB image global features;

[0141] Multiply the correlation matrix between the modalities corresponding to the thermal image data by a preset number of thermal image feature matrices element by element to obtain the global features of the thermal image;

[0142] Multiply the RGB image global features and the modal features corresponding to the RGB image generation data element by element to obtain the final RGB image features;

[0143] The final Thermal image feature is obtained by multiplying the global feature of the Thermal image with the modal feature corresponding to the Thermal image generation data element by element;

[0144] The final RGB image features are adaptively fused with the final Thermal image features to obtain the initial RGB_Thermal fusion features.

[0145] For example, cross-modal fusion of the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain the initial RGB_Thermal fusion features can be implemented using the following steps:

[0146] Step 1: Multi-scale fusion of RGB and thermal imaging image features:

[0147] To capture richer features, the multi-scale features of RGB and thermal images are further expanded through dilated convolution. The process steps are as follows:

[0148] The second layer of dilated convolution (expansion rate is 2):

[0149]

[0150] The second layer of dilated convolution (expansion rate is 4):

[0151]

[0152] Finally, the multi-scale features of RGB and thermal imaging are fused through the dilated convolution layer to output the final multi-scale feature representation:

[0153]

[0154] Step 2: Cross-modal information complementation:

[0155] The cross-attention mechanism is used to obtain the correlation matrix between RGB images and thermal images at each perspective, which helps the model recognize the complementarity of information in different environments and thus better combine the two modalities in the fusion process.

[0156] First, the multi-scale features of the extracted RGB and thermal imaging images are input into three different 1×1 convolutional layers to generate three feature matrices:

[0157]

[0158] In the above formula, Represents the convolution operation from multi-scale features Different features are extracted from . These features capture the color distribution and spatial details of the RGB image.

[0159] Similarly:

[0160]

[0161] The extracted features can provide key information about the location of the heat source and its intensity.

[0162] Calculate the correlation between features and obtain the correlation matrix between modalities

[0163]

[0164] And the correlation matrix With the characteristic matrix Perform element-by-element multiplication to generate a global context representation, and use the Softmax function to calculate the normalized weights to enhance the fused feature representation.

[0165] Finally, the global features are combined with the multi-scale features and a weighted sum (element-wise addition) is used to fuse the global features into the multi-scale features of RGB and thermal imaging. To avoid feature loss, a residual connection is used to preserve the information of the original features.

[0166]

[0167] Similarly:

[0168]

[0169] Step 3: Adaptive multimodal fusion:

[0170] By calculating the importance weights of features from each modality, we ensure that more important modalities are assigned higher weights during the fusion process, while relatively unimportant modalities are assigned lower weights, thereby achieving adaptive fusion of different modalities.

[0171] First, RGB features and thermal imaging features ( and ) After the global average pooling operation (GlobalAverage Pooling, GAP), the one-dimensional convolution (1D Convolution) is used to extract higher-level information from the global feature representation to obtain the global feature G RGB and G Thermal .

[0172]

[0173] Here, the GAP operation compresses the features of each modality into a fixed-size global feature vector, thereby reducing the complexity of the feature space while retaining important global information. By applying the Softmax function, the global features are converted into adaptive weight vectors corresponding to RGB and thermal imaging features.

[0174]

[0175] ω Thermal =1-ω RGB

[0176] Among them, G RGB and G Thermal are the global features of RGB and Thermal images respectively; ω RGB represents the weight of the RGB image, ω Thermal Represents the weight of the Thermal image. The Softmax function ensures the normalization of the weight vector so that the sum of all weights is 1, which facilitates subsequent weighting operations.

[0177] The obtained weight vector is multiplied element-by-element by the features of the corresponding modality to generate weighted features, and then the weighted RGB features and thermal imaging features are added together to obtain the final fusion feature representation:

[0178]

[0179] Among them, ω RGB 、ω Thermal is the weight corresponding to RGB and Thermal, and They are the features of RGB and Thermal after multi-scale convolution. It is the initial RGB_Thermal fusion feature of RGB and Thermal.

[0180] S4. Convert the LiDAR point cloud generated data into LiDAR point cloud features.

[0181] For example, the conversion of LiDAR point cloud generated data into LiDAR point cloud features can be achieved by the following implementation steps:

[0182] In the LiDAR point cloud feature extraction module, the point cloud is processed by voxelization. The point cloud is voxelized into a 3D voxel representation:

[0183]

[0184] The LiDAR point cloud is then fed into the LiDAR encoder, where it undergoes a multi-layer perceptron (MLP) and maximum pooling (MaxPool) operation to extract features. The input is assumed to be n×4 point cloud data, and the output is an n×512 feature representation.

[0185] A LiDAR =MaxPool(MLP(P LiDAR ))

[0186] Among them, P LiDAR Represents the input LiDAR point cloud data.

[0187] After projecting to the BEV perspective, we use a 2D convolutional neural network (CNN) to extract high-level features to provide a unified BEV perspective feature for subsequent fusion with RGB-Thermal.

[0188]

[0189] 105: A point (u, v) in the RGB-Thermal image coordinate system is associated with the LiDAR point cloud coordinates (X, Y, Z) through the camera intrinsic parameter matrix K and the extrinsic parameter T (camera to LiDAR transformation matrix):

[0190]

[0191] in, is the camera intrinsic parameter matrix, is the rotation matrix, is the translation vector and s is the scale factor.

[0192] The RGB-Thermal view feature is projected onto BEV, and the RGB_Thermal image feature is obtained after pre-fusion. It is a multi-view feature, each feature point corresponds to a pixel point (u, v), and uses the depth information Z provided by LiDAR LiDAR Calculate the corresponding BEV coordinates (X, Y).

[0193]

[0194] Among them, (c x, c y ) is the optical center of the camera, (f x , f y ) is the focal length.

[0195] LiDAR cannot cover all pixels of the RGB-Thermal image, so the LiDAR BEV feature is used for depth compensation:

[0196] Z′=αZ LiDAR +(1-αZ pred )

[0197] where Z pred is the depth estimated by RGB-Thermal, and α is the weight coefficient, which is obtained based on the consistency of LiDAR point cloud density and depth:

[0198]

[0199] where N LiDAR Represents the number of points in the LiDAR point cloud within a grid cell (X, Y) in the BEV's field of view. λ is a smoothing factor that prevents the denominator from being zero and is usually set to 1. β is a hyperparameter (usually between 1 and 10) that controls the rate at which α decreases.

[0200] Then construct a mapping function based on LiDAR features to project the RGB-Thermal features into the BEV space:

[0201]

[0202] in, It is a projection mapping operation that maps the features of the FV view to the BEV view based on the depth Z. Characteristics of RGB-Thermal at BEV perspective.

[0203] S5. Fuse the LiDAR point cloud features with the initial RGB_Thermal fusion features to obtain multimodal BEV perspective features, and perform environment recognition based on the multimodal BEV perspective features.

[0204] As an embodiment of the present invention, the LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain multimodal BEV view features, including:

[0205] Map the LiDAR point cloud features into the query vector in the dual attention mechanism, and map the initial RGB_Thermal fusion features into the key vector and value vector in the dual attention mechanism;

[0206] The cross attention in the dual attention mechanism is used to calculate the attention degree of the LiDAR point cloud features and the initial RGB_Thermal fusion features and perform preliminary fusion to obtain the initial multimodal fusion features;

[0207] Use the self-attention in the dual attention mechanism to calculate the self-attention features in the initial multimodal fusion features;

[0208] The initial multimodal fusion features and self-attention features are concatenated to obtain multimodal joint features;

[0209] The spatial features in the multimodal joint features are extracted to obtain the multimodal BEV perspective features.

[0210] For example, the LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain the multimodal BEV view features, which can be achieved by the following steps:

[0211] It is implemented using a dual attention mechanism, cross attention and self-attention, which is specifically implemented as follows:

[0212] First, BEV cross-modal attention fusion, the feature representation of LiDAR modality is The feature representation of RGB-Thermal mode is Where n is the length of the feature sequence and m is the feature dimension. Mapping to query vector The RGB-Thermal feature Mapped to key vectors Sum vector

[0213] In order to reduce computational complexity and enhance local feature association, cross attention only calculates the attention weights of the keys and values ​​within the query position i and its neighborhood ρ(i). The neighborhood ρ(i) is a region of size k centered at the query position i. For the query position i, the attention degree of LiDAR to RGB-Thermal is calculated, and the initial multimodal fusion feature as follows:

[0214]

[0215] in, In the above formula, x i Score attention is the query vector at LiDAR query position i; is the key vector of RGB-Thermal, which is mapped using the domain p(i) of query point i during calculation; B (i,p(i)) is the position bias, used to introduce position information, d k is a scaling factor used to stabilize the gradient, It is the value vector corresponding to RGB-Thermal, representing the information provided by this modality; where d represents the destination modality, i.e. RGB-Thermal.

[0216] On the basis of the cross-attention mechanism, the self-attention mechanism is used to further strengthen the feature correlation within the fusion modality. The specific implementation is as follows:

[0217] Output the initial multimodal fusion feature A from the cross attention c As input, the query vector Q is generated by linear projection Self , key vector K Self Sum value vector V Self , the self-attention feature representation of query position i The calculation is as follows:

[0218]

[0219] in, is the query vector at query position i; is the corresponding key vector; B (i) is the position offset; d k is the scaling factor; is the corresponding value vector, and self represents the linear transformation of the query, key, and value vectors from the unified input.

[0220] Output the initial multimodal fusion feature A from the cross attention c and self-attention output A s Splicing on the feature dimension to obtain the joint feature representation A joint

[0221] A joint =[A c ,A s ]

[0222] Among them, [·] represents the splicing operation.

[0223] Then the joint feature A joint Input convolution layer to further extract spatial features and generate the final dual attention output F output .

[0224] F output =Conv(A joint )

[0225] F output The final output multimodal BEV view feature map.

[0226] Furthermore, the embodiment of the present invention uses edge computing optimization technology to improve the computational efficiency of the multimodal joint generation network, and adopts the following implementation steps:

[0227] Step 1: Structured Pruning: Use the L1 norm pruning method to calculate the L1 norm of the weight of each convolution channel or attention head, and remove redundant convolution channels and connections in Adaptive Multi-Scale Feature Fusion and DualAttention Fusion. For a given convolution kernel weight W, the L1 norm is the sum of the absolute values ​​of its elements.

[0228]

[0229] Where W ij Represents each element in the weight matrix. By calculating the L1 norm of each convolutional layer or attention layer, the channels with L1 norm less than the threshold λ are pruned:

[0230] pruned channels={W i |L1(W i )<λ}

[0231] Where W represents the weight matrix of the convolution kernel or attention head

[0232] Low-bit quantization effectively reduces memory usage and computing requirements by converting network models from 32-bit floating-point numbers to 8-bit integers (INT8 quantization), ensuring efficient operation of the model on edge devices.

[0233] Step 2: Knowledge Distillation: Use the lightweight MobileNet model to learn from the original model (Teacher Model) through knowledge distillation, reducing the model size and improving the inference speed while maintaining high accuracy. Distillation loss function:

[0234] L KD =αL CE (y,p stu )+(1-α)T 2 L CE (q te ,q stu )

[0235] Among them L CE(y,p) represents the standard cross entropy loss function, y is the true label, p is the predicted value; T is the temperature parameter, which adjusts the smoothness of the soft label; q te ,q stu are the output probability distributions of the teacher model and the student model, respectively, and α is a hyperparameter that controls the relative importance of the teacher loss and the student loss.

[0236] Step 3: TensorRT acceleration. To speed up the inference process, the optimized quantized model is accelerated by TensorRT, and the hardware acceleration function of TensorRT is used to further improve real-time performance.

[0237] The present invention generates multimodal data sets by inputting Gaussian noise vectors from each modal data set into a multimodal joint generative network for data generation. This generates physically consistent multimodal data, significantly improving data diversity and coverage. Furthermore, the present invention extracts modal features corresponding to the RGB image generation data and the thermal image generation data from the multimodal data set, and cross-modally fuses these modal features to generate initial RGB-Thermal fusion features. This achieves a fusion of the complementary advantages of visible light and thermal imaging, improving the robustness of the visual modality under challenging lighting conditions. Furthermore, the present invention extracts LiDAR point cloud generation data from the multimodal data set and converts the point cloud generation data into LiDAR point cloud features. This effectively extracts precise geometric structure information from the generated point cloud, providing basic depth information for the BEV space. Furthermore, the LiDAR point cloud features are fused with the initial RGB-Thermal fusion features to generate multimodal BEV perspective features. Environmental recognition based on the multimodal BEV perspective features can improve the accuracy of environmental detection and recognition.

[0238] Reference Figure 2 FIG. 1 is a network structure diagram of a multimodal data enhancement and fusion method provided by an embodiment of the present invention.

[0239] Reference Figure 3 , which is a schematic diagram of the feature fusion structure of the RGB_Thermal fusion feature of the multimodal data enhancement and fusion method provided in one embodiment of the present invention.

[0240] Reference Figure 4 , which is a schematic diagram of a cross-modal complementary structure of a multimodal data enhancement and fusion method thereof provided by an embodiment of the present invention.

[0241] Reference Figure 5 , which is a schematic diagram of LiDAR point cloud feature generation for a multimodal data enhancement and fusion method provided by an embodiment of the present invention.

[0242] Reference Figure 6 , which is a schematic diagram of the cross-attention mechanism structure of the multimodal data enhancement and fusion method provided by one embodiment of the present invention.

[0243] like Figure 7 , which is a functional module diagram of a multimodal data enhancement and fusion device provided by one embodiment of the present invention.

[0244] The multimodal data enhancement and fusion device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the multimodal data enhancement and fusion device 100 can include a data generation module 101, a feature fusion module 102, and an environment recognition module 103.

[0245] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0246] In this embodiment, the functions of each module / unit are as follows:

[0247] The data generation module 101 is used to obtain multimodal data and calculate the Gaussian noise vector of each modal data, wherein the multimodal data includes RGB image data, Thermal image data and LiDAR point cloud data; input the Gaussian noise vector of the multimodal data into the multimodal joint generation network to obtain a multimodal data group, wherein the multimodal data group includes RGB image generation data, Thermal image generation data and LiDAR point cloud generation data.

[0248] In the embodiment of the present invention, the Gaussian noise vector refers to a one-dimensional vector generated by a multimodal joint generation network and obeys a standard normal distribution.

[0249] In an embodiment of the present invention, the multimodal joint generation network may be a DCGAN framework network.

[0250] As an embodiment of the present invention, a Gaussian noise vector of multimodal data is input into a multimodal joint generation network to obtain a multimodal data set, including:

[0251] The Gaussian noise vector of each modal data is normalized using the fully connected layer of the multimodal joint generation network to obtain the normalized features corresponding to each modal data;

[0252] The convolutional layer of the multimodal joint generation network is used to upsample the normalized features corresponding to each modality data to obtain a high-resolution feature map corresponding to each modality data;

[0253] A graph neural network is used to convert the high-resolution feature map corresponding to each modality data into a heterogeneous graph, and a linear activation function is used to update the nodes in the heterogeneous graph to obtain an updated heterogeneous graph. The nodes in the heterogeneous graph are the feature points of different modalities, and the edges in the heterogeneous graph are the connections between different modalities.

[0254] An adaptive attention mechanism is calculated based on the updated heterogeneous graph, and the adaptive attention mechanism is used to enhance the feature expression of each modality data to obtain enhanced features of each modality data;

[0255] The enhanced features of each modality data are transposed and convolved using the convolutional layer corresponding to the modality in the multimodal joint generation network to obtain generated data of different modal data;

[0256] The generated data of different modal data are integrated to obtain a multimodal data set.

[0257] Exemplarily, the Gaussian noise vector of each modal data is input into the multimodal joint generation network to generate data to obtain a multimodal data set, which can be achieved by the following implementation steps:

[0258] a. First, the Gaussian noise vector of each modality data is processed through a fully connected layer and batch normalization, as follows:

[0259]

[0260] Among them, v m The index subscript includes {RGB, Thermal, LiDAR}. The calculation formula for each modality is the same, but the subscript is different. In fact, it is:

[0261] Where W is the weight matrix; b is the bias vector; H is the feature representation after processing by the fully connected layer.

[0262]

[0263] where u B is the batch mean, σ B is the batch standard deviation; γ is the first learnable parameter and β is the second learnable parameter, which is used to control the scaling and translation after normalization.

[0264] Then go through three layers of convolutional network:

[0265]

[0266] v m∈{RGB,Therml,LiDAR}, where ConvT2d represents a two-dimensional transposed convolution operation; ReLU(·) is used as the activation function; and the output size is H. out =S(H in -1)+K-2P; S is the stride (Stride=2), K is the convolution kernel size (KernelSize=4); P is padding (Padding=1).

[0267] b. To achieve complementary information transmission between modalities, GNN is used for feature propagation to construct a heterogeneous graph G = (V, E), where nodes V represent feature points of different modalities and edges E represent the association between modalities. The adjacency matrix between modalities is defined as:

[0268]

[0269] Among them, σ is used to control the scale of the adjacency matrix, and Represents features of two different modes.

[0270] Each node is updated as follows:

[0271]

[0272] Among them, sigmoid(·) is a nonlinear activation function, W is the first learning parameter, and t represents the number of updates. For the adaptive attention mechanism:

[0273]

[0274] Among them, W is the first learning parameter, a is the second learning parameter, || represents the vector splicing operation, and Representing the features of two different modes, LeakyRelu(·) is used as the activation function to allow negative numbers to pass, as shown below:

[0275]

[0276] After repeating step b three times, the modality group is output to the self-attention mechanism to enhance the information expression ability of each modality.

[0277] c. The three modalities then enter independent convolutional layers. The RGB and Therml image features enter the 2D transposed convolution, while the LiDAR point cloud requires the use of 3D transposed convolution to generate three modal data.

[0278]

[0279] Tanh(·) and sigmoid(·) are activation functions; represents the RGB features after three iterations, Represents the Therml feature after three iterations, Represents the LiDAR modal features after three iterations.

[0280] Furthermore, after the generation data of different modal data are integrated to obtain a multimodal data group, it also includes using a discriminator of a multimodal joint generation network to perform coordination and consistency processing on the multimodal data group.

[0281] Furthermore, the discriminator of the multimodal joint generation network is used to coordinate and coordinate the multimodal data set, including:

[0282] Input the multimodal data set and the real multimodal data into the discriminator to obtain the probability values ​​of different modes corresponding to the multimodal data set and the real multimodal data;

[0283] Calculate modal embedding codes of different modalities based on the probability values ​​of different modalities corresponding to the multimodal data set and the real multimodal data;

[0284] Multimodal data sets are harmonized and consistent based on modal embedding coding.

[0285] For example, the discriminator of the multimodal joint generative network is used to perform coordination and consistency processing on the multimodal data set, and the following implementation steps can be adopted:

[0286] Step 1. Generate multimodal data set and real multimodal data The improved discriminator obtains the corresponding probability value. The improved discriminator adds a self-attention mechanism. The improved discriminator includes four convolutional layers, increasing the number of channels layer by layer, and using convolution with a step size of 2 for downsampling. For example, the first convolution operation is: BatchNorm is added after each convolution layer. The last layer does not use BatchNorm and directly uses Sigmoid to output the probability:

[0287]

[0288] The above formula is the sigmoid activation function, where W represents the weight matrix of the fully connected layer, b is the bias term of the fully connected layer, To generate the features extracted by the fourth layer of convolution, the features extracted from the real data are expressed as

[0289] The goal of the discriminator is to make the real data Generate data

[0290] Step 2. Get features after the fourth convolution layer The modal embedding code is dynamically calculated based on the real data and generated data. The characteristics of each modality are the dynamic weighting of the real data and the generated data.

[0291]

[0292] where λ r is the weighting coefficient, λ f =1-λ r ; T represents the total number of iterations, which is the current number of iterations. Then calculate the modal embedding code:

[0293]

[0294] in is the attention weight, As the third learnable parameter, exp(·) ensures that the attention score is positive.

[0295] The feature fusion module 102 is used to extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and cross-modally fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain an initial RGB_Thermal fusion feature.

[0296] As an embodiment of the present invention, extracting modal features corresponding to RGB image generation data and modal features corresponding to Thermal image generation data in a multimodal data group includes:

[0297] Converting RGB image generation data and Thermal image generation data into RGB image feature matrix and Thermal image feature matrix;

[0298] Perform image convolution processing according to the number of channels of the RGB image feature matrix to obtain preliminary RGB image features;

[0299] Perform image convolution processing according to the number of channels of the Thermal image feature matrix to obtain preliminary Thermal image features;

[0300] The preliminary RGB image features are expanded using a dilated convolutional layer with a first expansion rate to obtain modal features corresponding to the RGB image generation data;

[0301] The preliminary thermal image features are expanded using the dilated convolutional layer with the second expansion rate to obtain the modal features corresponding to the thermal image generation data.

[0302] Exemplarily, extracting the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data in the multimodal data group can be achieved by the following implementation steps:

[0303] The RGB image and thermal imaging image are represented by feature matrices, where H and W are the height and width of the image respectively, and C RGB and C Thermal are the number of channels of RGB images and thermal images, respectively. Typically, RGB images have 3 channels, while thermal images typically have 1 channel. During the preprocessing phase, preliminary feature extraction is performed on each modality image through convolution. Specifically, a convolution operation is performed on each channel of each modality image to obtain preliminary convolution features:

[0304]

[0305] Among them, Conv(·) represents the convolution operation, and ReLU(·) is the activation function used to increase the nonlinear expression ability of the network.

[0306] To fuse local information at multiple scales, the initially extracted RGB and thermal image features are processed through dilated convolution layers with different expansion rates to capture information at different scales. To further extract multi-scale features, the RGB and thermal image features are convolved using multiple convolution kernels of different sizes (3×3, 5×5, and 7×7). The steps for obtaining convolution features at different scales are as follows:

[0307] a. RGB / thermal imaging branch feature extraction:

[0308] First, the input RGB image features are convolved and multi-scale feature information is obtained by dilating the convolution layer, v′ m ∈{RGB,Therml} process is as follows:

[0309] Among them, because v′ m Instead of RGB and Thermal symbols, for or It has been obtained through the above calculation. All the following formulas use v′ m The principles of the representation are the same. The expanded description is:

[0310]

[0311] Among them, D1_Conv(·) is the dilation convolution operation with a dilation rate of 1, and cat(·) is the splicing operation. Represents the output features after the first layer of dilated convolution.

[0312] Furthermore, the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data are cross-modally fused to obtain the initial RGB_Thermal fusion features, including:

[0313] The cross-attention mechanism is used to calculate the correlation between the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, respectively, to obtain the correlation matrix between the modalities corresponding to the RGB image data and the correlation matrix between the modalities corresponding to the Thermal image data;

[0314] The modal features corresponding to the RGB image generation data are extracted using a preset number of convolutional layers to obtain a preset number of RGB image feature matrices;

[0315] The modal features corresponding to the thermal image generation data are extracted using a preset number of convolutional layers to obtain a preset number of thermal image feature matrices;

[0316] Multiply the inter-modality correlation matrix corresponding to the RGB image data by a preset number of RGB image feature matrices element by element to obtain the RGB image global features;

[0317] Multiply the correlation matrix between the modalities corresponding to the thermal image data by a preset number of thermal image feature matrices element by element to obtain the global features of the thermal image;

[0318] Multiply the RGB image global features and the modal features corresponding to the RGB image generation data element by element to obtain the final RGB image features;

[0319] The final Thermal image feature is obtained by multiplying the global feature of the Thermal image with the modal feature corresponding to the Thermal image generation data element by element;

[0320] The final RGB image features are adaptively fused with the final Thermal image features to obtain the initial RGB_Thermal fusion features.

[0321] For example, cross-modal fusion of the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain the initial RGB_Thermal fusion features can be implemented using the following steps:

[0322] Step 1: Multi-scale fusion of RGB and thermal imaging image features:

[0323] To capture richer features, the multi-scale features of RGB and thermal images are further expanded through dilated convolution. The process steps are as follows:

[0324] The second layer of dilated convolution (expansion rate is 2):

[0325]

[0326] The second layer of dilated convolution (expansion rate is 4):

[0327]

[0328] Finally, the multi-scale features of RGB and thermal imaging are fused through the dilated convolution layer to output the final multi-scale feature representation:

[0329]

[0330] Step 2: Cross-modal information complementation:

[0331] The cross-attention mechanism is used to obtain the correlation matrix between RGB images and thermal images at each perspective, which helps the model recognize the complementarity of information in different environments and thus better combine the two modalities in the fusion process.

[0332] First, the multi-scale features of the extracted RGB and thermal imaging images are input into three different 1×1 convolutional layers to generate three feature matrices:

[0333]

[0334] In the above formula, Represents the convolution operation from multi-scale features Different features are extracted from . These features capture the color distribution and spatial details of the RGB image.

[0335] Similarly:

[0336]

[0337] The extracted features can provide key information about the location of the heat source and its intensity.

[0338] Calculate the correlation between features and obtain the correlation matrix between modalities

[0339]

[0340] And the correlation matrix With the characteristic matrix Perform element-by-element multiplication to generate a global context representation, and use the Softmax function to calculate the normalized weights to enhance the fused feature representation.

[0341] Finally, the global features are combined with the multi-scale features and a weighted sum (element-wise addition) is used to fuse the global features into the multi-scale features of RGB and thermal imaging. To avoid feature loss, a residual connection is used to preserve the information of the original features.

[0342]

[0343] Similarly:

[0344]

[0345] Step 3: Adaptive multimodal fusion:

[0346] By calculating the importance weights of features from each modality, we ensure that more important modalities are assigned higher weights during the fusion process, while relatively unimportant modalities are assigned lower weights, thereby achieving adaptive fusion of different modalities.

[0347] First, RGB features and thermal imaging features ( and ) After the global average pooling operation (GlobalAverage Pooling, GAP), the one-dimensional convolution (1D Convolution) is used to extract higher-level information from the global feature representation to obtain the global feature G RGB and G Thermal .

[0348]

[0349] Here, the GAP operation compresses the features of each modality into a fixed-size global feature vector, thereby reducing the complexity of the feature space while retaining important global information. By applying the Softmax function, the global features are converted into adaptive weight vectors corresponding to RGB and thermal imaging features.

[0350]

[0351] ω Thermal =1-ω RGB

[0352] Among them, G RGB and G THermal are the global features of RGB and Thermal images respectively; ω RGB represents the weight of the RGB image, ω Thermal Represents the weight of the Thermal image. The Softmax function ensures the normalization of the weight vector so that the sum of all weights is 1, which facilitates subsequent weighting operations.

[0353] The obtained weight vector is multiplied element-by-element by the features of the corresponding modality to generate weighted features, and then the weighted RGB features and thermal imaging features are added together to obtain the final fusion feature representation:

[0354]

[0355] Among them, ω RGB 、ω Thermal is the weight corresponding to RGB and Thermal, and They are the features of RGB and Thermal after multi-scale convolution. It is the initial RGB_Thermal fusion feature of RGB and Thermal.

[0356] The environment recognition module 103 is used to convert LiDAR point cloud generated data into LiDAR point cloud features; fuse the LiDAR point cloud features with the initial RGB_Thermal fusion features to obtain multimodal BEV perspective features, and perform environment recognition based on the multimodal BEV perspective features.

[0357] For example, extracting LiDAR point cloud generated data from a multimodal data set and converting the point cloud generated data into LiDAR point cloud features can be achieved by the following implementation steps:

[0358] In the LiDAR point cloud feature extraction module, the point cloud is processed by voxelization. The point cloud is voxelized into a 3D voxel representation:

[0359]

[0360] The LiDAR point cloud is then fed into the LiDAR encoder, where it undergoes a multi-layer perceptron (MLP) and maximum pooling (MaxPool) operation to extract features. The input is assumed to be n×4 point cloud data, and the output is an n×512 feature representation.

[0361] A LiDAR =MaxPool(MLP(P LiDAR ))

[0362] Among them, P LiDAR Represents the input LiDAR point cloud data.

[0363] After projecting to the BEV perspective, we use a 2D convolutional neural network (CNN) to extract high-level features to provide a unified BEV perspective feature for subsequent fusion with RGB-Thermal.

[0364]

[0365] 105: A point (u, v) in the RGB-Thermal image coordinate system is associated with the LiDAR point cloud coordinates (X, Y, Z) through the camera intrinsic parameter matrix K and the extrinsic parameter T (camera to LiDAR transformation matrix):

[0366]

[0367] in, is the camera intrinsic parameter matrix, is the rotation matrix, is the translation vector and s is the scale factor.

[0368] The RGB-Thermal view feature is projected onto BEV, and the RGB_Thermal image feature is obtained after pre-fusion. It is a multi-view feature, each feature point corresponds to a pixel point (u, v), and uses the depth information Z provided by LiDAR LiDAR Calculate the corresponding BEV coordinates (X, Y).

[0369]

[0370] Among them, (c x , c y ) is the optical center of the camera, (f x , f y ) is the focal length.

[0371] LiDAR cannot cover all pixels of the RGB-Thermal image, so the LiDAR BEV feature is used for depth compensation:

[0372] Z′=αZ LiDAR +(1-αZ pred )

[0373] where Z pred is the depth estimated by RGB-Thermal, and α is the weight coefficient, which is obtained based on the consistency of LiDAR point cloud density and depth:

[0374]

[0375] where N LiDAR Represents the number of points in the LiDAR point cloud within a grid cell (X, Y) in the BEV's field of view. λ is a smoothing factor that prevents the denominator from being zero and is usually set to 1. β is a hyperparameter (usually between 1 and 10) that controls the rate at which α decreases.

[0376] Then construct a mapping function based on LiDAR features to project the RGB-Thermal features into the BEV space:

[0377]

[0378] in, It is a projection mapping operation that maps the features of the FV view to the BEV view based on the depth Z. Characteristics of RGB-Thermal at BEV perspective.

[0379] As an embodiment of the present invention, the LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain multimodal BEV view features, including:

[0380] Map the LiDAR point cloud features into the query vector in the dual attention mechanism, and map the initial RGB_Thermal fusion features into the key vector and value vector in the dual attention mechanism;

[0381] The cross attention in the dual attention mechanism is used to calculate the attention degree of the LiDAR point cloud features and the initial RGB_Thermal fusion features and perform preliminary fusion to obtain the initial multimodal fusion features;

[0382] Use the self-attention in the dual attention mechanism to calculate the self-attention features in the initial multimodal fusion features;

[0383] The initial multimodal fusion features and self-attention features are concatenated to obtain multimodal joint features;

[0384] The spatial features in the multimodal joint features are extracted to obtain the multimodal BEV perspective features.

[0385] For example, the LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain the multimodal BEV view features, which can be achieved by the following steps:

[0386] It is implemented using a dual attention mechanism, cross attention and self-attention, which is specifically implemented as follows:

[0387] First, BEV cross-modal attention fusion, the feature representation of LiDAR modality is The feature representation of RGB-Thermal mode is Where n is the length of the feature sequence and m is the feature dimension. Mapping to query vector The RGB-Thermal feature Mapped to key vectors Sum vector

[0388] In order to reduce computational complexity and enhance local feature association, cross attention only calculates the attention weights of the keys and values ​​within the query position i and its neighborhood ρ(i). The neighborhood ρ(i) is a region of size k centered at the query position i. For the query position i, the attention degree of LiDAR to RGB-Thermal is calculated, and the initial multimodal fusion feature as follows:

[0389]

[0390] in, In the above formula, x i Score attention is the query vector at LiDAR query position i; is the key vector of RGB-Thermal, which is mapped using the domain p(i) of query point i during calculation; B (i,p(i)) is the position bias, used to introduce position information, d k is a scaling factor used to stabilize the gradient, It is the value vector corresponding to RGB-Thermal, representing the information provided by this modality; where d represents the destination modality, i.e. RGB-Thermal.

[0391] On the basis of the cross-attention mechanism, the self-attention mechanism is used to further strengthen the feature correlation within the fusion modality. The specific implementation is as follows:

[0392] Output the initial multimodal fusion feature A from the cross attention c As input, the query vector Q is generated by linear projection Self , key vector K Self Sum value vector V Self , the self-attention feature representation of query position i The calculation is as follows:

[0393]

[0394] in, is the query vector at query position i; is the corresponding key vector; B (i) is the position offset; d k is the scaling factor; is the corresponding value vector, and self represents the linear transformation of the query, key, and value vectors from the unified input.

[0395] Output the initial multimodal fusion feature A from the cross attention c and self-attention output A s Splicing on the feature dimension to obtain the joint feature representation Ajoint

[0396] A joint =[A c ,A s ]

[0397] Among them, [·] represents the splicing operation.

[0398] Then the joint feature A joint Input convolution layer to further extract spatial features and generate the final dual attention output F output .

[0399] F output =Conv(A joint )

[0400] F output The final output multimodal BEV view feature map.

[0401] Furthermore, the embodiment of the present invention uses edge computing optimization technology to improve the computational efficiency of the multimodal joint generation network, and adopts the following implementation steps:

[0402] Step 1: Structured Pruning: Use the L1 norm pruning method to calculate the L1 norm of the weight of each convolution channel or attention head, and remove redundant convolution channels and connections in Adaptive Multi-Scale Feature Fusion and DualAttention Fusion. For a given convolution kernel weight W, the L1 norm is the sum of the absolute values ​​of its elements.

[0403]

[0404] Where W ij Represents each element in the weight matrix. By calculating the L1 norm of each convolutional layer or attention layer, the channels with L1 norm less than the threshold λ are pruned:

[0405] pruned channels={W i |L1(W i )<λ}

[0406] Where W represents the weight matrix of the convolution kernel or attention head

[0407] Low-bit quantization effectively reduces memory usage and computing requirements by converting network models from 32-bit floating-point numbers to 8-bit integers (INT8 quantization), ensuring efficient operation of the model on edge devices.

[0408] Step 2: Knowledge Distillation: Use the lightweight MobileNet model to learn from the original model (Teacher Model) through knowledge distillation, reducing the model size and improving the inference speed while maintaining high accuracy. Distillation loss function:

[0409] L KD =αL CE (y,p stu )+(1-α)T 2 L CE (q te ,q stu )

[0410] Among them L CE (y,p) represents the standard cross entropy loss function, y is the true label, p is the predicted value; T is the temperature parameter, which adjusts the smoothness of the soft label; q te ,q stu are the output probability distributions of the teacher model and the student model, respectively, and α is a hyperparameter that controls the relative importance of the teacher loss and the student loss.

[0411] Step 3: TensorRT acceleration. To speed up the inference process, the optimized quantized model is accelerated by TensorRT, and the hardware acceleration function of TensorRT is used to further improve real-time performance.

[0412] like Figure 8 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a multimodal data enhancement and fusion method provided by an embodiment of the present invention.

[0413] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a multimodal data enhancement and fusion method program.

[0414] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the memory 11 (for example, executing a multimodal data enhancement and fusion method program, etc.), as well as calling data stored in the memory 11, to perform various functions of the electronic device and process data.

[0415] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as a code of a multimodal data enhancement and fusion method program, but can also be used to temporarily store data that has been output or is to be output.

[0416] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0417] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0418] Figure 8 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 8 The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0419] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0420] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0421] The multimodal data enhancement and fusion method program stored in the memory 11 of the electronic device is a combination of multiple instructions. When executed in the processor 10, it can achieve the following:

[0422] Acquire multimodal data and calculate the Gaussian noise vector for each modal data, where the multimodal data includes RGB image data, Thermal image data, and LiDAR point cloud data;

[0423] Inputting the Gaussian noise vector of the multimodal data into a multimodal joint generation network to obtain a multimodal data set, wherein the multimodal data set includes RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data;

[0424] Extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and cross-modally fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain the initial RGB_Thermal fusion features;

[0425] Convert LiDAR point cloud generated data into LiDAR point cloud features;

[0426] The LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain the multimodal BEV perspective features, and the environment is recognized based on the multimodal BEV perspective features.

[0427] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0428] Furthermore, if the modules / units integrated into the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0429] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor of an electronic device, the computer program can implement:

[0430] Acquire multimodal data and calculate the Gaussian noise vector for each modal data, where the multimodal data includes RGB image data, Thermal image data, and LiDAR point cloud data;

[0431] Inputting the Gaussian noise vector of the multimodal data into a multimodal joint generation network to obtain a multimodal data set, wherein the multimodal data set includes RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data;

[0432] Extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and cross-modally fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain the initial RGB_Thermal fusion features;

[0433] Convert LiDAR point cloud generated data into LiDAR point cloud features;

[0434] The LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain the multimodal BEV perspective features, and the environment is recognized based on the multimodal BEV perspective features.

[0435] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0436] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0437] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0438] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0439] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0440] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0441] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0442] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

[0443] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal data enhancement and fusion method, characterized in that: The method comprises: Acquire multimodal data and calculate the Gaussian noise vector for each modal data, where the multimodal data includes RGB image data, Thermal image data, and LiDAR point cloud data; Inputting the Gaussian noise vector of the multimodal data into a multimodal joint generation network to obtain a multimodal data set, wherein the multimodal data set includes RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data; Extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and cross-modally fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data to obtain the initial RGB_Thermal fusion features; Convert LiDAR point cloud generated data into LiDAR point cloud features; The LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain the multimodal BEV perspective features, and the environment is recognized based on the multimodal BEV perspective features.

2. The multimodal data enhancement and fusion method according to claim 1, wherein: The Gaussian noise vector of the multimodal data is input into the multimodal joint generation network to obtain a multimodal data group, including: The Gaussian noise vector of each modal data is normalized using the fully connected layer of the multimodal joint generation network to obtain the normalized features corresponding to each modal data; The convolutional layer of the multimodal joint generation network is used to upsample the normalized features corresponding to each modality data to obtain a high-resolution feature map corresponding to each modality data; A graph neural network is used to convert the high-resolution feature map corresponding to each modality data into a heterogeneous graph, and a linear activation function is used to update the nodes in the heterogeneous graph to obtain an updated heterogeneous graph. The nodes in the heterogeneous graph are the feature points of different modalities, and the edges in the heterogeneous graph are the connections between different modalities. An adaptive attention mechanism is calculated based on the updated heterogeneous graph, and the adaptive attention mechanism is used to enhance the feature expression of each modality data to obtain enhanced features of each modality data; The enhanced features of each modality data are transposed and convolved using the convolutional layer corresponding to the modality in the multimodal joint generation network to obtain generated data of different modal data; The generated data of different modal data are integrated to obtain a multimodal data set.

3. The multimodal data enhancement and fusion method according to claim 1, wherein: After the generation data of the different modal data are integrated to obtain the multimodal data group, the method further includes using a discriminator of the multimodal joint generation network to perform coordination and consistency processing on the multimodal data group.

4. The multimodal data enhancement and fusion method according to claim 3, wherein: The method of using the discriminator of the multimodal joint generation network to perform coordination and consistency processing on the multimodal data group includes: Input the multimodal data set and the real multimodal data into the discriminator to obtain the probability values ​​of different modes corresponding to the multimodal data set and the real multimodal data; Calculate modal embedding codes of different modalities based on the probability values ​​of different modalities corresponding to the multimodal data set and the real multimodal data; Multimodal data sets are harmonized and consistent based on modal embedding coding.

5. The multimodal data enhancement and fusion method according to claim 1, wherein: The extracting of modal features corresponding to the RGB image generation data and modal features corresponding to the Thermal image generation data includes: Converting RGB image generation data and Thermal image generation data into RGB image feature matrix and Thermal image feature matrix; Perform image convolution processing according to the number of channels of the RGB image feature matrix to obtain preliminary RGB image features; Perform image convolution processing according to the number of channels of the Thermal image feature matrix to obtain preliminary Thermal image features; The preliminary RGB image features are expanded using a dilated convolutional layer with a first expansion rate to obtain modal features corresponding to the RGB image generation data; The preliminary thermal image features are expanded using the dilated convolutional layer with the second expansion rate to obtain the modal features corresponding to the thermal image generation data.

6. The multimodal data enhancement and fusion method according to claim 1, wherein: The cross-modal fusion of the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data obtains the initial RGB_Thermal fusion features, including: The cross-attention mechanism is used to calculate the correlation between the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, respectively, to obtain the correlation matrix between the modalities corresponding to the RGB image data and the correlation matrix between the modalities corresponding to the Thermal image data; The modal features corresponding to the RGB image generation data are extracted using a preset number of convolutional layers to obtain a preset number of RGB image feature matrices; The modal features corresponding to the thermal image generation data are extracted using a preset number of convolutional layers to obtain a preset number of thermal image feature matrices; Multiply the inter-modality correlation matrix corresponding to the RGB image data by a preset number of RGB image feature matrices element by element to obtain the RGB image global features; Multiply the correlation matrix between the modalities corresponding to the thermal image data by a preset number of thermal image feature matrices element by element to obtain the global features of the thermal image; Multiply the RGB image global features and the modal features corresponding to the RGB image generation data element by element to obtain the final RGB image features; The final Thermal image feature is obtained by multiplying the global feature of the Thermal image with the modal feature corresponding to the Thermal image generation data element by element; The final RGB image features are adaptively fused with the final Thermal image features to obtain the initial RGB_Thermal fusion features.

7. The multimodal data enhancement and fusion method according to claim 1, wherein: The LiDAR point cloud features and the initial RGB_Thermal fusion features are fused to obtain the multimodal BEV perspective features, including: Map the LiDAR point cloud features into the query vector in the dual attention mechanism, and map the initial RGB_Thermal fusion features into the key vector and value vector in the dual attention mechanism; The cross attention in the dual attention mechanism is used to calculate the attention degree of the LiDAR point cloud features and the initial RGB_Thermal fusion features and perform preliminary fusion to obtain the initial multimodal fusion features; Use the self-attention in the dual attention mechanism to calculate the self-attention features in the initial multimodal fusion features; The initial multimodal fusion features and self-attention features are concatenated to obtain multimodal joint features; The spatial features in the multimodal joint features are extracted to obtain the multimodal BEV perspective features.

8. A multimodal data enhancement and fusion device, characterized in that: The device implements the multimodal data enhancement and fusion method according to any one of claims 1 to 7, and the device includes: A data generation module is configured to acquire multimodal data and calculate a Gaussian noise vector for each modal data, wherein the multimodal data includes RGB image data, Thermal image data, and LiDAR point cloud data; input the Gaussian noise vector of the multimodal data into a multimodal joint generation network to obtain a multimodal data set, wherein the multimodal data set includes RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data; A feature fusion module is used to extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data, and to fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data across modalities to obtain the initial RGB_Thermal fusion features; The environment recognition module is used to convert LiDAR point cloud generated data into LiDAR point cloud features; the LiDAR point cloud features are fused with the initial RGB_Thermal fusion features to obtain multimodal BEV perspective features, and the environment recognition is performed based on the multimodal BEV perspective features.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the multimodal data enhancement and fusion method thereof as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal data enhancement and fusion method thereof is implemented as described in any one of claims 1 to 7.

Images