A multi-modal data augmentation and fusion method, device, equipment and medium

Abstract

The present application relates to image processing technology, disclose a kind of multimodal data enhancement and its fusion method, comprising: obtaining multimodal data, and the Gaussian noise vector of each modality data is calculated;Each modality data Gaussian noise vector is input into multimodal joint generation network to carry out data generation, obtain multimodal data group;Extract the modal feature of RGB image data and Thermal image data in multimodal data group, and the modal feature of RGB image data and Thermal image data is fused across modal to obtain initial RGB_Thermal fusion feature;Extract LiDAR point cloud feature in multimodal data group;LiDAR point cloud feature and initial RGB_Thermal fusion feature are fused to obtain multimodal BEV perspective feature.The present application also proposes a kind of multimodal data enhancement and its fusion device, equipment and medium.The present application can improve the accuracy of environment detection and identification.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a multi-modal data enhancement and fusion method, device, equipment and medium. BACKGROUND

[0002] One of the core tasks of autonomous driving technology is environment perception, i.e., accurately identifying surrounding objects, obstacles and road conditions. In order to achieve efficient and safe perception in complex driving environments, autonomous driving systems usually rely on the fusion of multiple sensor modalities, such as LiDAR (laser radar), RGB images (visible light images), thermal imaging images (Thermal) and radar, etc. LiDAR can provide accurate depth information to help identify the geometry and distance of objects, but its point cloud data is easily disturbed in bad weather or strong light, affecting accuracy. Cameras provide rich visual information such as color and texture, but due to the lack of depth information and poor performance in low light, night or strong light, their application in three-dimensional object detection is limited.

[0003] The prior art scheme adopts a multi-modal data fusion framework based on bird's eye view (BEV), and the core goal is to map LiIDAR point cloud data and other modal data uniformly to the BEV space, and to improve the environment perception capability through feature fusion. It usually contains three stages of sensor data encoding, BEV space conversion, feature fusion and task driving.

[0004] (1) Multi-modal feature extraction and perspective conversion

[0005] In multi-modal data fusion, feature extraction is the basis for improving the system's perception ability. The BEVFusion method extracts features from LiDAR point clouds and camera images, effectively combining cross-modal data. BEVFusion extracts image features through a deep convolutional network and performs perspective transformation to map image features to the BEV space. This perspective transformation module uses two common techniques: one is Inverse Perspective Mapping (IPM) based on geometric rules, which projects image pixels inversely to a plane through geometric transformation, preserving the geometric relationship of objects; the other is a deformable attention mechanism based on deep learning, such as LSS (Learnable Spatial Sampling) or BEVFormer, which optimizes the mapping of images to BEV space through deep learning models, automatically adapting to different scenes and perspectives to provide BEV features with geometric consistency. LiDAR point clouds obtain distance information in the environment through laser beams and form dense three-dimensional point cloud data. To effectively use LiDAR data in the BEV space, common processing methods use voxelization techniques (such as VoxelNet, PointPillars) to discretize three-dimensional point cloud data into voxels. Voxelization technology divides LiDAR point cloud data into small cubic grid units (voxels) and calculates feature values for each voxel, effectively reducing data computation and preserving three-dimensional spatial information.

[0006] (2) BEV space feature fusion

[0007] In the BEV space, the features of the camera and LiDAR need to be aligned in the same space. BEVFusion uses feature stitching or weighted addition to fuse LiDAR and camera BEV features into a unified feature vector. Specifically, the channel numbers of LiDAR and camera features are merged to form a multi-channel BEV feature map. Next, the BEV Encoder (such as ResNet+FPN) further processes these fused features to generate an enhanced BEV feature representation. This fusion method effectively combines the geometric information of LiDAR and the semantic information of camera images, making the final BEV feature map not only have depth information but also contain rich semantic information, improving perception accuracy.

[0008] The existing technology lacks a multi-modal data generation mechanism, resulting in limited data generalization ability, and does not consider the global context relationship between multi-modal in the BEV multi-modal fusion strategy, making the fusion information unstable in long-distance targets and complex environments, resulting in key target feature missing or modal information mismatching, and thus leading to low accuracy of environment detection and recognition. SUMMARY

[0009] The application provides a multi-modal data enhancement and fusion method, device, equipment and medium, which can improve the accuracy of environment detection and recognition.

[0010] To achieve the above-mentioned purpose, the application provides a multi-modal data enhancement and fusion method, which comprises:

[0011] Obtain multi-modal data and calculate the Gaussian noise vector of each modal data, wherein the multi-modal data comprises RGB image data, thermal image data and LiDAR point cloud data;

[0012] Input the Gaussian noise vector of the multi-modal data into a multi-modal joint generation network to obtain a multi-modal data group, wherein the multi-modal data group comprises RGB image generation data, thermal image generation data and LiDAR point cloud generation data;

[0013] Extract the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data, and cross-modally fuse the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data to obtain initial RGB_Thermal fusion features;

[0014] Convert the LiDAR point cloud generation data into LiDAR point cloud features;

[0015] Fuse the LiDAR point cloud features with the initial RGB_Thermal fusion features to obtain multi-modal BEV perspective features, and perform environment recognition according to the multi-modal BEV perspective features.

[0016] Optionally, the inputting of the Gaussian noise vector of the multi-modal data into the multi-modal joint generation network to obtain the multi-modal data group comprises:

[0017] Normalizing the Gaussian noise vector of each modal data by using the full connection layer of the multi-modal joint generation network to obtain the normalized features corresponding to each modal data;

[0018] Up-sampling the normalized features corresponding to each modal data by using the convolution layer of the multi-modal joint generation network to obtain the high-resolution feature maps corresponding to each modal data;

[0019] Converting the high-resolution feature maps corresponding to each modal data into a heterogeneous graph by using a graph neural network, and updating the nodes in the heterogeneous graph by using a linear activation function to obtain an updated heterogeneous graph, wherein the nodes in the heterogeneous graph are feature points of different modalities, and the edges in the heterogeneous graph are correlations between different modalities;

[0020] An adaptive attention mechanism is calculated according to the updated heterogeneous graph, and the adaptive attention mechanism is used to enhance the feature expression of each modality data, so as to obtain enhanced features of each modality data;

[0021] The enhanced features of each modality data are subjected to transposed convolution processing by the convolution layer corresponding to the modality in the multi-modal joint generation network, so as to obtain generated data of different modality data;

[0022] The generated data of different modality data are integrated to obtain a multi-modal data group.

[0023] Optionally, after the generated data of different modality data are integrated to obtain the multi-modal data group, the multi-modal data group is subjected to coordination consistency processing by a discriminator of the multi-modal joint generation network.

[0024] Optionally, the coordination consistency processing of the multi-modal data group by the discriminator of the multi-modal joint generation network comprises:

[0025] The multi-modal data group and real multi-modal data are input into the discriminator to obtain probability values of different modalities corresponding to the multi-modal data group and the real multi-modal data;

[0026] The modality embedding encoding of different modalities is calculated according to the probability values of different modalities corresponding to the multi-modal data group and the real multi-modal data;

[0027] The multi-modal data group is subjected to coordination consistency processing according to the modality embedding encoding.

[0028] Optionally, the extraction of the modality features corresponding to the RGB image generated data and the modality features corresponding to the Thermal image generated data comprises:

[0029] The RGB image generated data and the Thermal image generated data are converted into an RGB image feature matrix and a Thermal image feature matrix;

[0030] The RGB image feature matrix is subjected to image convolution processing according to the number of channels to obtain preliminary RGB image features;

[0031] The Thermal image feature matrix is subjected to image convolution processing according to the number of channels to obtain preliminary Thermal image features;

[0032] The preliminary RGB image features are subjected to inflation processing by an inflation convolution layer with a first expansion rate to obtain the modality features corresponding to the RGB image generated data;

[0033] The preliminary Thermal image features are subjected to inflation processing by an inflation convolution layer with a second expansion rate to obtain the modality features corresponding to the Thermal image generated data.

[0034] Optionally, the initial RGB_Thermal fusion feature is obtained by performing feature fusion on the modality features corresponding to the RGB image generation data and the modality features corresponding to the Thermal image generation data, and the feature fusion comprises the following steps:

[0035] Correlations between the modality features corresponding to the RGB image generation data and correlations between the modality features corresponding to the Thermal image generation data are calculated by using a cross-attention mechanism respectively, to obtain a correlation matrix between the modalities corresponding to the RGB image data and a correlation matrix between the modalities corresponding to the Thermal image data;

[0036] The modality features corresponding to the RGB image generation data are subjected to feature extraction by using a preset number of convolution layers, to obtain a preset number of RGB image feature matrices;

[0037] The modality features corresponding to the Thermal image generation data are subjected to feature extraction by using a preset number of convolution layers, to obtain a preset number of Thermal image feature matrices;

[0038] The correlation matrix between the modalities corresponding to the RGB image data is multiplied element by element with the preset number of RGB image feature matrices, to obtain RGB image global features;

[0039] The correlation matrix between the modalities corresponding to the Thermal image data is multiplied element by element with the preset number of Thermal image feature matrices, to obtain Thermal image global features;

[0040] The RGB image global features are multiplied element by element with the modality features corresponding to the RGB image generation data, to obtain final RGB image features;

[0041] The Thermal image global features are multiplied element by element with the modality features corresponding to the Thermal image generation data, to obtain final Thermal image features;

[0042] The final RGB image features and the final Thermal image features are subjected to adaptive feature fusion, to obtain the initial RGB_Thermal fusion feature.

[0043] Optionally, the multi-modal BEV perspective feature is obtained by performing feature fusion on the LiDAR point cloud feature and the initial RGB_Thermal fusion feature, and the feature fusion comprises the following steps:

[0044] The LiDAR point cloud feature is mapped into a query vector in a dual-attention mechanism, and the initial RGB_Thermal fusion feature is mapped into a key vector and a value vector in the dual-attention mechanism;

[0045] The cross-attention in the dual attention mechanism is used to calculate the attention degree of the LiDAR point cloud feature and the initial RGB_Thermal fusion feature, and initial multi-modal fusion features are obtained through preliminary fusion;

[0046] The self-attention in the dual attention mechanism is used to calculate the self-attention feature in the initial multi-modal fusion feature;

[0047] The initial multi-modal fusion feature and the self-attention feature are spliced to obtain a multi-modal joint feature;

[0048] In order to solve the above problems, the application further provides a multi-modal data enhancement and fusion device, the device comprises:

[0049] A data generation module is configured to obtain multi-modal data and calculate a Gaussian noise vector for each modality of data, wherein the multi-modal data comprises RGB image data, thermal image data and LiDAR point cloud data; the Gaussian noise vector of the multi-modal data is input into a multi-modal joint generation network to obtain a multi-modal data group, wherein the multi-modal data group comprises RGB image generation data, thermal image generation data and LiDAR point cloud generation data;

[0050] A feature fusion module is configured to extract a modality feature corresponding to the RGB image generation data and a modality feature corresponding to the thermal image generation data, and cross-modality fuse the modality feature corresponding to the RGB image generation data and the modality feature corresponding to the thermal image generation data to obtain initial RGB_Thermal fusion features;

[0051] An environment recognition module is configured to convert the LiDAR point cloud generation data into LiDAR point cloud features, fuse the LiDAR point cloud features with the initial RGB_Thermal fusion features to obtain multi-modal BEV perspective features, and recognize an environment according to the multi-modal BEV perspective features.

[0052] In order to solve the above problems, the application further provides an electronic device, the electronic device comprises:

[0053] At least one processor; and,

[0054] A memory in communication connection with the at least one processor; wherein,

[0055] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the multi-modal data enhancement and fusion method described above.

[0056] To solve the above problems, the application further provides a computer readable storage medium, wherein at least one computer program is stored in the computer readable storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the multi-modal data enhancement and fusion method.

[0057] The application can generate multi-modal data sets by inputting the Gaussian noise vector of each modal data into the multi-modal joint generation network for data generation, can generate generated multi-modal data with physical consistency, significantly improve the diversity and coverage of data, in addition, the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data in the multi-modal data set are extracted, and the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data are cross-modal fused to obtain initial RGB_Thermal fusion features, which can realize the complementary advantage fusion of visible light and thermal imaging, and improve the robustness of visual modal under challenging lighting conditions, in addition, the LiDAR point cloud generation data in the multi-modal data set is extracted, and the point cloud generation data is converted into LiDAR point cloud features, which can effectively extract the accurate geometric structure information of the generated point cloud, provide basic depth information for BEV space, in addition, the LiDAR point cloud features and the initial RGB_Thermal fusion features are fused to obtain multi-modal BEV view features, and environment recognition is performed according to the multi-modal BEV view features, which can improve the accuracy of environment detection and recognition. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of a multi-modal data enhancement and fusion method provided by an embodiment of the application is shown in the figure;

[0059] Figure 2 A network structure diagram of a multi-modal data enhancement and fusion method provided by an embodiment of the application is shown in the figure;

[0060] Figure 3 A feature fusion structure diagram of RGB_Thermal fusion features of a multi-modal data enhancement and fusion method provided by an embodiment of the application is shown in the figure;

[0061] Figure 4 A cross-modal complementary structure diagram of a multi-modal data enhancement and fusion method provided by an embodiment of the application is shown in the figure;

[0062] Figure 5 A LiDAR point cloud feature generation diagram of a multi-modal data enhancement and fusion method provided by an embodiment of the application is shown in the figure;

[0063] Figure 6A cross-attention mechanism structure schematic diagram of a multi-modal data enhancement and fusion method provided by an embodiment of the present application is shown in Fig.

[0064] Figure 7 A function module diagram of a multi-modal data enhancement and fusion device provided by an embodiment of the present application is shown in Fig.

[0065] Figure 8 A structure schematic diagram of an electronic device for implementing the multi-modal data enhancement and fusion method provided by an embodiment of the present application is shown in Fig.

[0066] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0068] Embodiments of the present application provide a multi-modal data enhancement and fusion method. The execution subject of the multi-modal data enhancement and fusion method includes, but is not limited to, at least one of electronic devices capable of being configured to execute the method provided by the embodiments of the present application, such as a server and a terminal. In other words, the multi-modal data enhancement and fusion method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0069] Referring to Fig. Figure 1 A flowchart of a multi-modal data enhancement and fusion method provided by an embodiment of the present application is shown in Fig. In this embodiment, the multi-modal data enhancement and fusion method includes:

[0070] S1, acquiring multi-modal data and calculating a Gaussian noise vector of each modality data, wherein the multi-modal data includes RGB image data, thermal image data and LiDAR point cloud data.

[0071] In the embodiments of the present application, the Gaussian noise vector refers to a one-dimensional vector subject to a standard normal distribution generated by a multi-modal joint generation network.

[0072] S2, input the Gaussian noise vector of the multi-modal data into the multi-modal joint generation network to obtain a multi-modal data set, wherein the multi-modal data set comprises RGB image generation data, thermal image generation data and LiDAR point cloud generation data.

[0073] In the embodiment of the application, the multi-modal joint generation network can be a DCGAN framework network.

[0074] As an embodiment of the application, the Gaussian noise vector of the multi-modal data is input into the multi-modal joint generation network to obtain a multi-modal data set, comprising:

[0075] The Gaussian noise vector of each modal data is normalized by using the full connection layer of the multi-modal joint generation network to obtain the normalized feature corresponding to each modal data;

[0076] The normalized feature corresponding to each modal data is up-sampled by using the convolution layer of the multi-modal joint generation network to obtain the high-resolution feature map corresponding to each modal data;

[0077] The high-resolution feature map corresponding to each modal data is converted into a heterogeneous graph by using the graph neural network, and the nodes in the heterogeneous graph are updated by using a linear activation function to obtain an updated heterogeneous graph, wherein the nodes in the heterogeneous graph are feature points of different modalities, and the edges in the heterogeneous graph are associations between different modalities;

[0078] An adaptive attention mechanism is calculated according to the updated heterogeneous graph, and the feature expression of each modal data is enhanced by using the adaptive attention mechanism to obtain the enhanced feature of each modal data;

[0079] The enhanced feature of each modal data is transposed and convoluted by using the convolution layer corresponding to the modal in the multi-modal joint generation network to obtain the generation data of different modal data;

[0080] The generation data of different modal data is integrated to obtain the multi-modal data set.

[0081] Exemplarily, the Gaussian noise vector of each modal data is input into the multi-modal joint generation network for data generation to obtain a multi-modal data set, which can be realized by the following implementation steps:

[0082] a. First, the Gaussian noise vector of each modal data is processed by a full connection (Fully Connected) layer and a batch normalization (Batch Normalization) layer, and the formula is as follows:

[0083]

[0084] wherein, v mis an index set including {RGB, Thermal, LiDAR}, the calculation formula of each modality is the same, the subscript is different, and actually it is:

[0085] where W is a weight matrix; b is a bias vector; H is a feature representation processed by a fully connected layer.

[0086]

[0087] where u B is the batch mean, σ B is the batch standard deviation; γ is the first learnable parameter, and β is the second learnable parameter, which is used to control the scaling and translation after normalization.

[0088] After passing through a three-layer convolutional network:

[0089]

[0090] v m ∈ {RGB, Thermal, LiDAR}, where ConvT2d represents a two-dimensional transpose convolution operation; ReLU(·) is an activation function; the output size: H out = S(H in -1) + K-2P; S is the step size (Stride = 2), K is the kernel size (KernelSize = 4); P is the padding (Padding = 1).

[0091] b. To realize the complementary information transmission between modalities, GNN is used for feature propagation, and a heterogeneous graph G = (V, E) is constructed, where node V represents the feature points of different modalities, and edge E represents the association between modalities. The adjacency matrix between modalities is defined as:

[0092]

[0093] where σ is used to control the scale of the adjacency matrix, and represent the features of two different modalities.

[0094] The update mode of each node is as follows:

[0095]

[0096] where sigmoid(·) is a nonlinear activation function, W is the first learning parameter, t represents the number of updates, is an adaptive attention mechanism:

[0097]

[0098] where W is a first learning parameter, a is a second learning parameter, and || represents a vector concatenation operation, and denote features of two different modalities, and LeakyRelu(·) is an activation function that allows negative numbers to pass, as follows:

[0099]

[0100] After repeating step b three times, the modal group is output to the self-attention mechanism to enhance the information expression ability of each modality.

[0101] c. Then the three modalities respectively enter independent convolutional layers, and the RGB and thermal image features enter 2D transpose convolution, while the LiDAR point cloud needs to use 3D transpose convolution to generate three modal data.

[0102]

[0103] Tanh(·) and sigmoid(·) are activation functions; denotes the RGB feature after three iterations, denotes the thermal feature after three iterations, denotes the LiDAR modality feature after three iterations.

[0104] Further, after the generated data of the integrated different modal data obtains the multi-modal data group, the multi-modal data group is further subjected to coordination consistency processing by using a discriminator of a multi-modal joint generation network.

[0105] Further, the coordination consistency processing of the multi-modal data group by using the discriminator of the multi-modal joint generation network includes:

[0106] The multi-modal data group and real multi-modal data are input into the discriminator to obtain probability values of different modalities corresponding to the multi-modal data group and the real multi-modal data;

[0107] The modality embedding encoding of different modalities is calculated according to the probability values of different modalities corresponding to the multi-modal data group and the real multi-modal data;

[0108] The multi-modal data group is subjected to coordination consistency processing according to the modality embedding encoding.

[0109] Exemplarily, the coordination consistency processing of the multi-modal data group by using the discriminator of the multi-modal joint generation network can adopt the following implementation steps:

[0110] Step 1. Generated multi-modal data group and real multi-modal data The improved discriminator obtains corresponding probability values. The improved discriminator adds a self-attention mechanism, and the improved discriminator includes four convolutional layers, the number of channels is increased layer by layer, and a convolution with a step of 2 is used for down-sampling. For example, the first layer of convolution operation is as follows: BatchNorm is added after each convolutional layer, and BatchNorm is not used in the last layer, and the probability is output directly by using Sigmoid:

[0111]

[0112] The above formula is a sigmoid activation function, where W represents the weight matrix of the full connection layer, b is the bias term of the full connection layer, To generate the features extracted by the fourth convolutional layer, the features extracted from the real data are represented as

[0113] The objective of the discriminator is to make the real data The generated data

[0114] Step 2. The features obtained after the fourth convolutional layer are According to the real data and the generated data, the modal embedding encoding is dynamically calculated, and the features of each modal are the dynamic weighting of the real data and the generated data,

[0115]

[0116] where λ r is a weighting coefficient, λ f = 1-λ r ; T represents the total number of iterations, and is the current iteration number. Then the modal embedding encoding is calculated:

[0117]

[0118] where is an attention weight, is a third learnable parameter, and exp(·) ensures that the attention score is positive.

[0119] S3, extract the modal features corresponding to the generated data of the RGB image and the modal features corresponding to the generated data of the thermal image, and cross-modally fuse the modal features corresponding to the generated data of the RGB image and the modal features corresponding to the generated data of the thermal image to obtain initial RGB_Thermal fusion features.

[0120] As an embodiment of the present application, the modal feature corresponding to the RGB image generation data and the modal feature corresponding to the thermal image generation data in the multi-modal data set are extracted, including:

[0121] The RGB image generation data and the thermal image generation data are converted into an RGB image feature matrix and a thermal image feature matrix;

[0122] The RGB image feature matrix is subjected to image convolution processing according to the number of channels to obtain a preliminary RGB image feature;

[0123] The thermal image feature matrix is subjected to image convolution processing according to the number of channels to obtain a preliminary thermal image feature;

[0124] The preliminary RGB image feature is subjected to inflation processing by using an inflation convolution layer with a first expansion rate to obtain the modal feature corresponding to the RGB image generation data;

[0125] The preliminary thermal image feature is subjected to inflation processing by using an inflation convolution layer with a second expansion rate to obtain the modal feature corresponding to the thermal image generation data.

[0126] Exemplarily, the modal feature corresponding to the RGB image generation data and the modal feature corresponding to the thermal image generation data in the multi-modal data set can be realized by using the following implementation steps:

[0127] The RGB image and the thermal image are respectively represented by a feature matrix, where H and W are the height and width of the image respectively, and C RGB and C Thermal are the number of channels of the RGB image and the thermal image respectively. Generally, the number of channels of the RGB image is 3, and the thermal image usually has 1 channel. In the preprocessing stage, preliminary feature extraction is performed on each modal image by convolution operation. Specifically, for each channel of each modal image, convolution operation is performed to obtain a preliminary convolution feature:

[0128]

[0129] where Conv(·) represents convolution operation, and ReLU(·) is an activation function, which is used to increase the nonlinear expression ability of the network.

[0130] In order to fuse the local information of multiple scales, the features of the preliminarily extracted RGB and thermal imaging images are processed through dilated convolution layers with different expansion rates respectively, so as to capture information of different scales. In order to further extract multi-scale features, the features of the RGB and thermal imaging images are subjected to convolution operation using multiple convolution kernels of different sizes (3x3, 5x5 and 7x7), so as to obtain convolution features of different scales. The steps are as follows:

[0131] a. RGB / thermal imaging branch feature extraction:

[0132] Firstly, the input RGB image features are subjected to convolution processing, and multi-scale feature information is obtained through dilated convolution layers, v' m ∈{RGB,Therml}processes are as follows:

[0133] Where, because v' m replaces the two symbols of RGB and Thermal, in fact = v' or has been obtained through the above calculation. The principles of all parts represented by v' m in the following formula are the same. The expansion description is as follows:

[0134]

[0135] Where, D1_Conv(·) is the dilated convolution operation with an expansion rate of 1, cat(·) is the concatenation operation, represents the output feature after the first layer of dilated convolution.

[0136] Further, the initial RGB_Thermal fusion features are obtained by fusing the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data, including:

[0137] The correlation between the modal features corresponding to the RGB image generation data and the correlation between the modal features corresponding to the thermal image generation data are calculated respectively by using the cross-attention mechanism, so as to obtain the correlation matrix between the modal features corresponding to the RGB image data and the correlation matrix between the modal features corresponding to the thermal image data.

[0138] The modal features corresponding to the RGB image generation data are subjected to feature extraction using a preset number of convolution layers, so as to obtain a preset number of RGB image feature matrices;

[0139] The modal features corresponding to the thermal image generation data are subjected to feature extraction using a preset number of convolution layers, so as to obtain a preset number of thermal image feature matrices;

[0140] The correlation matrix between the modalities corresponding to the RGB image data is multiplied element by element with a preset number of RGB image feature matrices to obtain RGB image global features;

[0141] The correlation matrix between the modalities corresponding to the Thermal image data is multiplied element by element with a preset number of Thermal image feature matrices to obtain Thermal image global features;

[0142] The RGB image global features are multiplied element by element with the modal features corresponding to the RGB image generation data to obtain final RGB image features;

[0143] The Thermal image global features are multiplied element by element with the modal features corresponding to the Thermal image generation data to obtain final Thermal image features;

[0144] The final RGB image features and the final Thermal image features are adaptively fused to obtain initial RGB_Thermal fusion features.

[0145] Exemplarily, the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data are fused across modalities to obtain initial RGB_Thermal fusion features, which can be realized by the following implementation steps:

[0146] Step 1, multi-scale fusion of RGB and thermal imaging image features:

[0147] In order to capture more rich features, the multi-scale features of RGB and thermal imaging images are further expanded through dilated convolution. The process steps are as follows:

[0148] Second layer dilated convolution (expansion rate is 2):

[0149]

[0150] Second layer dilated convolution (expansion rate is 4):

[0151]

[0152] Finally, the multi-scale features of RGB and thermal imaging are fused through the dilated convolution layer, and the final multi-scale feature representation is output:

[0153]

[0154] Step 2, complementary information across modalities:

[0155] The cross-attention mechanism is used to obtain the correlation matrix of the RGB image and the thermal image under each view, which helps the model to identify the complementarity of information in different environments, so as to better combine the two modalities in the fusion process.

[0156] First, the multi-scale features of the extracted RGB and thermal images are input into three different 1x1 convolutional layers to generate three feature matrices:

[0157]

[0158] In the above formula, represents different features extracted from the multi-scale features by convolutional operation. These features capture the color distribution and spatial details of the RGB image.

[0159] Similarly, we have:

[0160]

[0161] The extracted features can provide key information about the location and intensity of the heat source.

[0162] The correlation between the features is calculated to obtain the correlation matrix

[0163]

[0164] And the correlation matrix is element-wise multiplied with the feature matrix to generate a global context representation. The Softmax function is used to calculate the normalized weights, thereby strengthening the feature representation after fusion.

[0165] Finally, the global feature is combined with the multi-scale feature, and the weighted sum (element-wise addition) is used to fuse the global feature into the multi-scale feature of the RGB and thermal image. To avoid the loss of features, residual connection is used to maintain the information of the original features.

[0166]

[0167] Similarly, we have:

[0168]

[0169] Step 3, adaptive multi-modal fusion:

[0170] By calculating the importance weight from the features of each modality, it ensures that more important modalities are assigned higher weights in the fusion process, while less important modalities are reduced in weight, thereby achieving adaptive fusion of different modalities.

[0171] First, the RGB features and thermal imaging features ( and ) are subjected to a global average pooling operation (GAP), and then one-dimensional convolution (1D Convolution) is used to extract higher-level information from the global feature representation, obtaining global features G RGB and G Thermal .

[0172]

[0173] Here, the GAP operation compresses each modal feature into a fixed-size global feature vector, reducing the complexity of the feature space while preserving important global information. By applying the Softmax function, the global features are converted into an adaptive weight vector corresponding to the RGB and thermal imaging features.

[0174]

[0175] ω Thermal = 1-ω RGB

[0176] where G RGB and G Thermal are the global features of RGB and thermal images respectively; ω RGB represents the weight of the RGB image, and ω Thermal represents the weight of the thermal image. The Softmax function ensures the normalization of the weight vector, so that the sum of all weights is 1, facilitating subsequent weighted operations.

[0177] The resulting weight vector is multiplied element-wise with the corresponding modal features to generate weighted features, and then the weighted RGB features and thermal imaging features are added to obtain the final fusion feature representation:

[0178]

[0179] where ω RGB and ω Thermal are the weights corresponding to RGB and thermal, and are the features of RGB and thermal after multi-scale convolution. is the initial RGB_Thermal fusion feature of RGB and thermal.

[0180] S4, convert LiDAR point cloud generation data into LiDAR point cloud features.

[0181] Exemplarily, the LiDAR point cloud generation data is converted into LiDAR point cloud features, which can be achieved by the following implementation steps:

[0182] The LiDAR point cloud feature extraction module processes the point cloud by voxelization method. The point cloud is voxelized into a three-dimensional voxel representation:

[0183]

[0184] Then the LiDAR point cloud is input into the LiDAR encoder, and the features are extracted through multi-layer perception (MLP) and maximum pooling (MaxPool) operations. The input is set to n x 4 point cloud data, and the output is n x 512 feature representation.

[0185] A LiDAR =MaxPool(MLP(P LiDAR ))

[0186] Wherein, P LiDAR represents the input LiDAR point cloud data.

[0187] After projection into the BEV view, we use a 2D convolutional neural network (CNN) to extract high-level features to provide unified BEV view features for subsequent fusion with RGB-Thermal

[0188]

[0189] 105: A point (u, v) in the RGB-Thermal image coordinate system is associated with the LiDAR point cloud coordinates (X, Y, Z) through the camera intrinsic matrix K and the extrinsic T (the transformation matrix from the camera to the LiDAR):

[0190]

[0191] Wherein, is the camera intrinsic matrix, is the rotation matrix, is the translation vector, and s is the scale factor.

[0192] The RGB-Thermal view features are projected into the BEV, and the RGB_Thermal image features are obtained after pre-fusion is a multi-view feature, each feature point of which corresponds to a pixel point (u, v), and the depth information Z LiDAR provided by the LiDAR is used to calculate the corresponding BEV coordinates (X, Y).

[0193]

[0194] Wherein, (c x, c y ) is the camera optical center, (f x , f y ) is the focal length.

[0195] LiDAR cannot cover all pixels of the RGB-Thermal image, and the LiDAR BEV feature is used for depth compensation:

[0196] Z' = alpha Z LiDAR + (1-alpha) Z pred )

[0197] Where Z pred is the depth estimated by RGB-Thermal, alpha is the weight coefficient, which is obtained based on the consistency of LiDAR point cloud density and depth:

[0198]

[0199] Where N LiDAR represents the number of points of LiDAR point cloud in a grid element (X, Y) in the BEV view. Lambda is a smoothing factor to prevent the denominator from being zero, usually set to 1, and beta is a hyperparameter (usually between 1-10), which controls the descending speed of alpha.

[0200] Then a mapping function based on LiDAR features is constructed to project the RGB-Thermal features into the BEV space:

[0201]

[0202] Where, is the projection mapping operation, which maps the features in the FV view to the BEV view based on the depth Z. is the feature of RGB-Thermal in the BEV view.

[0203] S5, the LiDAR point cloud feature and the initial RGB_Thermal fusion feature are fused to obtain a multi-modal BEV view feature, and the environment is identified according to the multi-modal BEV view feature.

[0204] As an embodiment of the application, the LiDAR point cloud feature and the initial RGB_Thermal fusion feature are fused to obtain a multi-modal BEV view feature, which comprises:

[0205] The LiDAR point cloud feature is mapped into the query vector in the double attention mechanism, and the initial RGB_Thermal fusion feature is mapped into the key vector and value vector in the double attention mechanism;

[0206] The attention degree of the LiDAR point cloud feature and the initial RGB_Thermal fusion feature is calculated by cross attention in the dual attention mechanism, and initial multi-modal fusion features are obtained by preliminary fusion.

[0207] The self-attention feature in the initial multi-modal fusion feature is calculated by self-attention in the dual attention mechanism.

[0208] The initial multi-modal fusion feature and the self-attention feature are spliced to obtain a multi-modal joint feature.

[0209] The spatial feature in the multi-modal joint feature is extracted to obtain a multi-modal BEV perspective feature.

[0210] Exemplarily, the LiDAR point cloud feature and the initial RGB_Thermal fusion feature are fused to obtain a multi-modal BEV perspective feature, which can be realized by the following steps:

[0211] The dual attention mechanism is adopted, including cross attention and self-attention, and the specific implementation is as follows:

[0212] First, BEV cross-modal cross attention fusion, the feature representation of the LiDAR modality is The feature representation of the RGB-Thermal modality is Where n is the length of the feature sequence, and m is the feature dimension. By linear projection, the LiDAR feature is mapped to the query vector The RGB-Thermal feature is mapped to the key vector and the value vector

[0213] In order to reduce the computational complexity and enhance the local feature correlation, the cross attention only calculates the attention weight of the query position i and the keys and values within its neighborhood ρ(i). The neighborhood ρ(i) is a region centered on the query position i with a size of k. For the query position i, the attention degree of LiDAR to RGB-Thermal is calculated, and the initial multi-modal fusion feature is as follows:

[0214]

[0215] Where, In the above formula, x i is the attention score is the query vector at the query position i of the LiDAR; is the key vector of RGB-Thermal, which is mapped using the domain p(i) of the query point i when computing; B (i,p(i)) is the position bias, which is used to introduce position information, d k is the scaling factor, which is used to stabilize the gradient, is the value vector corresponding to RGB-Thermal, representing the information provided by this modality; where d represents the destination modality, i.e., RGB-Thermal.

[0216] On the basis of cross-attention mechanism, self-attention mechanism is used to further strengthen the feature association within the fused modalities, which is implemented as follows:

[0217] The cross-attention output initial multi-modal fusion feature A c is input, and query vector Q Self , key vector K Self and value vector V Self are generated by linear projection, and the self-attention feature representation of query position i is calculated as follows:

[0218]

[0219] where, is the query vector at query position i; is the corresponding key vector; B (i) is the position bias; d k is the scaling factor;, is the corresponding value vector, and self represents that the query, key, and value vectors all come from linear transformation of the unified input.

[0220] The cross-attention output initial multi-modal fusion feature A c and the self-attention output A s are spliced in the feature dimension to obtain the joint feature representation A joint

[0221] A joint =[A c ,A s ]

[0222] where [·] represents the splicing operation.

[0223] The joint feature A joint is then input into the convolution layer to further extract spatial features, and the final dual-attention output F output is generated.

[0224] F output =Conv(A joint )

[0225] ​F output The final output multi-modal BEV perspective feature map.

[0226] Further, the embodiment of the present application optimizes the computing efficiency of the multi-modal joint generation network through edge computing optimization technology, and adopts the following implementation steps:

[0227] Step 1, structured pruning, using L1 norm pruning method to calculate the L1 norm of the weight of each convolution channel or attention head, removing redundant convolution channels and connections in Adaptive Multi-Scale Feature Fusion and DualAttention Fusion. For a given convolution kernel weight W, the L1 norm is the sum of the absolute values of its elements

[0228]

[0229] where W ij represents each element in the weight matrix. By calculating the L1 norm of each convolution layer or attention layer, the channels with L1 norm less than the threshold λ are pruned:

[0230] pruned channels={W i |L1(W i )<λ}

[0231] where W represents the weight matrix of the convolution kernel or attention head

[0232] Low-bit quantization (Low-Bit Quantization), by converting the network model from 32-bit floating point number to 8-bit integer (INT8 quantization), effectively reducing the memory occupation and computing demand, ensuring the efficient operation of the model on the edge device.

[0233] Step 2, knowledge distillation (Knowledge Distillation), using a lightweight model MobileNet to learn from the original model (Teacher Model) through knowledge distillation, reducing the model size and improving the inference speed, while maintaining high accuracy. Distillation loss function:

[0234] L KD =αL CE (y,p stu )+(1-α)T 2 L CE (q te ,q stu )

[0235] where L CE(y, p) represents a standard cross-entropy loss function, y is a real label, p is a predicted value; T is a temperature parameter, adjusting the smoothing degree of the soft label; q te , q stu are output probability distributions of the teacher model and the student model respectively, and a is a hyperparameter controlling the relative importance of the teacher loss and the student loss.

[0236] Step 3, TensorRT acceleration, in order to accelerate the inference process, the optimized quantization model is accelerated by TensorRT, and the hardware acceleration function of TensorRT is used to further improve the real-time performance.

[0237] The application can generate generated multi-modal data with physical consistency by inputting the Gaussian noise vector of each modal data into the multi-modal joint generation network for data generation, significantly improve the diversity and coverage of the data, in addition, the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data in the multi-modal data group are extracted, and the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data are cross-modal fused to obtain initial RGB_Thermal fusion features, which can realize the complementary advantage fusion of visible light and thermal imaging, and improve the robustness of the visual modal under challenging lighting conditions, in addition, the LiDAR point cloud generation data in the multi-modal data group is extracted, and the point cloud generation data is converted into LiDAR point cloud features, which can effectively extract the accurate geometric structure information of the generated point cloud, and provide basic depth information for the BEV space, in addition, the LiDAR point cloud features and the initial RGB_Thermal fusion features are fused to obtain multi-modal BEV view features, and the environment is identified according to the multi-modal BEV view features, which can improve the accuracy of environment detection and identification.

[0238] Referring to Figure 2 , it is a network structure diagram of the multi-modal data enhancement and fusion method provided by an embodiment of the application.

[0239] Referring to Figure 3 , it is a feature fusion structure diagram of the RGB_Thermal fusion features of the multi-modal data enhancement and fusion method provided by an embodiment of the application.

[0240] Referring to Figure 4 , it is a cross-modal complementary structure diagram of the multi-modal data enhancement and fusion method provided by an embodiment of the application.

[0241] Referring to Figure 5 , it is a LiDAR point cloud feature generation diagram of the multi-modal data enhancement and fusion method provided by an embodiment of the application.

[0242] Referring to Figure 6 As shown in the figure, it is a cross attention mechanism structure diagram of the multi-modal data enhancement and fusion method provided by an embodiment of the application.

[0243] As Figure 7 As shown in the figure, it is a functional module diagram of a multi-modal data enhancement and fusion device provided by an embodiment of the application.

[0244] The multi-modal data enhancement and fusion device 100 can be installed in an electronic device. According to the functions to be implemented, the multi-modal data enhancement and fusion device 100 can include a data generation module 101, a feature fusion module 102, and an environment recognition module 103.

[0245] The module of the application can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete a fixed function, and is stored in the memory of the electronic device.

[0246] In this embodiment, the functions of each module / unit are as follows:

[0247] The data generation module 101 is configured to obtain multi-modal data and calculate a Gaussian noise vector of each modality data, wherein the multi-modal data includes RGB image data, thermal image data, and LiDAR point cloud data; and input the Gaussian noise vector of the multi-modal data into a multi-modal joint generation network to obtain a multi-modal data group, wherein the multi-modal data group includes RGB image generation data, thermal image generation data, and LiDAR point cloud generation data.

[0248] In the embodiment of the application, the Gaussian noise vector refers to a one-dimensional vector generated by the multi-modal joint generation network and subject to a standard normal distribution.

[0249] In the embodiment of the application, the multi-modal joint generation network can be a DCGAN framework network.

[0250] As an embodiment of the application, the Gaussian noise vector of the multi-modal data is input into the multi-modal joint generation network to obtain the multi-modal data group, including:

[0251] The full connection layer of the multi-modal joint generation network is used to perform normalization processing on the Gaussian noise vector of each modality data to obtain the normalized feature corresponding to each modality data;

[0252] The convolution layer of the multi-modal joint generation network is used to perform up-sampling on the normalized feature corresponding to each modality data to obtain the high-resolution feature map corresponding to each modality data;

[0253] The high-resolution feature map corresponding to each modality data is converted into a heterogeneous graph by using a graph neural network, and the nodes in the heterogeneous graph are updated by using a linear activation function, to obtain an updated heterogeneous graph, wherein the nodes in the heterogeneous graph are feature points of different modalities, and the edges in the heterogeneous graph are correlations between different modalities.

[0254] An adaptive attention mechanism is calculated according to the updated heterogeneous graph, and the adaptive attention mechanism is used to enhance the feature expression of each modality data, to obtain enhanced features of each modality data.

[0255] The enhanced features of each modality data are transposed and convoluted by using the convolution layer corresponding to the modality in the multi-modal joint generation network, to obtain generated data of different modalities.

[0256] The generated data of different modalities is integrated to obtain a multi-modal data group.

[0257] Exemplarily, the Gaussian noise vector of each modality data is input into the multi-modal joint generation network for data generation, to obtain a multi-modal data group, and the following implementation steps can be used to achieve this:

[0258] a. First, the Gaussian noise vector of each modality data is processed by a fully connected (FC) layer and a batch normalization (BN) layer, and the formula is as follows:

[0259]

[0260] wherein v m is an index subscript, the index set includes {RGB, Thermal, LiDAR}, the calculation formula of each modality is the same, the subscript is different, and actually it is:

[0261] wherein W is a weight matrix; b is a bias vector; and H is a feature representation processed by the fully connected layer.

[0262]

[0263] wherein u B is a batch mean, σ B is a batch standard deviation; γ is a first learnable parameter, and β is a second learnable parameter, used to control the scaling and translation after normalization.

[0264] Then, a three-layer convolutional network is used:

[0265]

[0266] v m∈ {RGB, Therml, LiDAR}, where ConvT2d denotes a two-dimensional transpose convolution operation; ReLU(·) is an activation function; the output size: H out = S(H in -1) + K-2P; S is the step size (Stride = 2), K is the kernel size (KernelSize = 4); P is the padding (Padding = 1).

[0267] b. To achieve the complementary information transmission between modalities, GNN is used for feature propagation, and a heterogeneous graph G = (V, E) is constructed, where the node V represents the feature points of different modalities, and the edge E represents the association between modalities, and the adjacency matrix between modalities is defined as:

[0268]

[0269] where σ is used to control the scale of the adjacency matrix, and represent the features of two different modalities.

[0270] The update mode of each node is as follows:

[0271]

[0272] where sigmoid(·) is a nonlinear activation function, W is the first learning parameter, t represents the number of updates, is an adaptive attention mechanism:

[0273]

[0274] where W is the first learning parameter, a is the second learning parameter, || represents the vector concatenation operation, and represent the features of two different modalities, and LeakyRelu(·) is an activation function that allows negative numbers to pass, as follows:

[0275]

[0276] After repeating step b three times, the modal group outputs to the self-attention mechanism to enhance the information expression ability of each modality.

[0277] c. Then the three modalities respectively enter independent convolution layers, and the RGB and Therml image features enter 2D transpose convolution, while the LiDAR point cloud needs to use 3D transpose convolution, to generate three modal data.

[0278]

[0279] Tanh(·), sigmoid(·) are activation functions; represents the RGB feature after three iterations, represents the Thermal feature after three iterations, represents the LiDAR modality feature after three iterations.

[0280] Further, after the generated data of the integrated different modal data obtains a multi-modal data set, a discriminator of a multi-modal joint generation network is further used to perform a coordinated consistency processing on the multi-modal data set.

[0281] Further, the coordinated consistency processing on the multi-modal data set by the discriminator of the multi-modal joint generation network comprises:

[0282] inputting the multi-modal data set and real multi-modal data into the discriminator to obtain probability values of different modalities corresponding to the multi-modal data set and the real multi-modal data;

[0283] calculating modality embedding encodings of different modalities according to the probability values of different modalities corresponding to the multi-modal data set and the real multi-modal data;

[0284] performing a coordinated consistency processing on the multi-modal data set according to the modality embedding encodings.

[0285] Exemplarily, the coordinated consistency processing on the multi-modal data set by the discriminator of the multi-modal joint generation network can adopt the following implementation steps:

[0286] Step 1. The generated multi-modal data set and real multi-modal data obtain corresponding probability values through the improved discriminator. The improved discriminator adds a layer of self-attention mechanism, and the improved discriminator comprises four convolutional layers, the number of channels is increased layer by layer, and a convolution with a step of 2 is used for down-sampling. For example, the first layer of convolution operation is: BatchNorm is added after each convolutional layer, and BatchNorm is not used in the last layer, and the probability is directly output by Sigmoid:

[0287]

[0288] The above formula is a sigmoid activation function, where W represents a weight matrix of a full connection layer, and b is a bias term of the full connection layer, is the feature extracted after the fourth convolution, and the feature extracted after the real data is represented as

[0289] The objective of the discriminator is to make the real data generated data

[0290] Step 2. Obtain features after the fourth layer convolution According to the real data and the generated data, the modal embedding encoding is dynamically calculated, and the features of each modal are dynamically weighted real data and generated data,

[0291]

[0292] Where λ r is a weighting coefficient, λ f =1-λ r ; T represents the total number of iterations, and T represents the current iteration number. Then the modal embedding encoding is calculated:

[0293]

[0294] Where is an attention weight, is a third learnable parameter, and exp(·) ensures that the attention score is positive.

[0295] The feature fusion module 102 is configured to extract modal features corresponding to the RGB image generated data and modal features corresponding to the Thermal image generated data, and cross-modal fusion of the modal features corresponding to the RGB image generated data and the modal features corresponding to the Thermal image generated data to obtain initial RGB_Thermal fusion features.

[0296] As an embodiment of the present application, the modal features corresponding to the RGB image generated data and the modal features corresponding to the Thermal image generated data in the multi-modal data set are extracted, comprising:

[0297] The RGB image generated data and the Thermal image generated data are converted into an RGB image feature matrix and a Thermal image feature matrix;

[0298] According to the channel number of the RGB image feature matrix, image convolution processing is performed to obtain preliminary RGB image features;

[0299] According to the channel number of the Thermal image feature matrix, image convolution processing is performed to obtain preliminary Thermal image features;

[0300] The preliminary RGB image features are dilated using a dilated convolution layer with a first expansion rate to obtain the modal features corresponding to the RGB image generated data;

[0301] The preliminary Thermal image features are dilated using a dilated convolution layer with a second expansion rate to obtain the modal features corresponding to the Thermal image generated data.

[0302] Exemplarily, the modal features corresponding to the RGB image generated data and the modal features corresponding to the thermal image generated data in the multi-modal data set are extracted, which can be realized by the following implementation steps:

[0303] The RGB image and the thermal image are represented by feature matrices, where H and W are the height and width of the image, respectively, and C RGB and C Thermal are the channel numbers of the RGB image and the thermal image, respectively. Generally, the channel number of the RGB image is 3, and the thermal image usually has 1 channel. In the preprocessing stage, preliminary feature extraction is performed on each modal image through convolution operation. Specifically, for each channel of each modal image, convolution operation is performed to obtain preliminary convolution features:

[0304]

[0305] where Conv(·) represents the convolution operation, and ReLU(·) is an activation function used to increase the non-linear expression ability of the network.

[0306] In order to fuse multi-scale local information, the preliminary extracted RGB and thermal image features are processed through dilated convolution layers with different expansion rates to capture information of different scales. In order to further extract multi-scale features, multiple convolution kernels of different sizes (3x3, 5x5 and 7x7) are used to perform convolution operation on the features of the RGB and thermal images to obtain convolution features of different scales. The steps are as follows:

[0307] a. RGB / thermal branch feature extraction:

[0308] First, the input RGB image features are processed by convolution, and multi-scale feature information is obtained through dilated convolution layers, v′ m ∈{RGB,Therml}process as follows:

[0309] where, because v′ m replaces the two symbols of RGB and Thermal, in fact is or has been obtained through the above calculation. All parts represented by v′ m in the following formula have the same principle. The expansion description is as follows:

[0310]

[0311] where D1_Conv(·) is a dilated convolution operation with a dilated rate of 1, and cat(·) is a concatenation operation, The output feature of the first layer after the expansion convolution is represented.

[0312] Further, the initial RGB_Thermal fusion feature is obtained by fusing the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data across modalities, including:

[0313] The correlation between the modal features corresponding to the RGB image generation data and the correlation between the modal features corresponding to the Thermal image generation data are calculated respectively by using the cross-attention mechanism, to obtain the correlation matrix between the modal corresponding to the RGB image data and the correlation matrix between the modal corresponding to the Thermal image data.

[0314] The modal features corresponding to the RGB image generation data are extracted by using a preset number of convolution layers to obtain a preset number of RGB image feature matrices.

[0315] The modal features corresponding to the Thermal image generation data are extracted by using a preset number of convolution layers to obtain a preset number of Thermal image feature matrices.

[0316] The correlation matrix between the modal corresponding to the RGB image data is multiplied element by element with the preset number of RGB image feature matrices to obtain the RGB image global feature.

[0317] The correlation matrix between the modal corresponding to the Thermal image data is multiplied element by element with the preset number of Thermal image feature matrices to obtain the Thermal image global feature.

[0318] The RGB image global feature is multiplied element by element with the modal features corresponding to the RGB image generation data to obtain the final RGB image feature.

[0319] The Thermal image global feature is multiplied element by element with the modal features corresponding to the Thermal image generation data to obtain the final Thermal image feature.

[0320] The final RGB image feature and the final Thermal image feature are adaptively fused to obtain the initial RGB_Thermal fusion feature.

[0321] Exemplarily, the initial RGB_Thermal fusion feature is obtained by fusing the modal features corresponding to the RGB image generation data and the modal features corresponding to the Thermal image generation data across modalities, which can be realized by the following implementation steps:

[0322] Step 1, multi-scale fusion of RGB and thermal imaging image features:

[0323] To capture richer features, the multi-scale features of RGB and thermal images are further expanded by dilated convolutions. The procedure steps are as follows:

[0324] Second layer dilated convolution (expansion rate is 2):

[0325]

[0326] Second layer dilated convolution (expansion rate is 4):

[0327]

[0328] Finally, the multi-scale features of RGB and thermal images are fused by dilated convolution layers, and the final multi-scale feature representation is output:

[0329]

[0330] Step 2, cross-modal information complementarity:

[0331] The cross-attention mechanism is used to obtain the correlation matrix of RGB images and thermal images under each view, helping the model to identify the complementarity of information in different environments, so as to better combine the two modalities in the fusion process.

[0332] First, the extracted multi-scale features of RGB and thermal images are input into three different 1x1 convolution layers to generate three feature matrices:

[0333]

[0334] In the above formula, represents the different features extracted from the multi-scale features by convolution operation. These features capture the color distribution and spatial details of the RGB image.

[0335] Similarly,

[0336]

[0337] The extracted features can provide key information about the location and intensity of the heat source.

[0338] The correlation between the features is calculated to obtain the correlation matrix

[0339]

[0340] And the correlation matrix is element-wise multiplied with the feature matrix to generate a global context representation. The Softmax function is used to calculate the normalized weight, so as to strengthen the feature representation after fusion.

[0341] Finally, the global feature is fused into the multi-scale features of RGB and thermal images by using the weighted sum (element-wise addition) method. In order to avoid the loss of features, residual connection is used to maintain the information of the original features.

[0342]

[0343] Similarly, we have:

[0344]

[0345] Step 3, adaptive multi-modal fusion:

[0346] By calculating the importance weight from each modality feature, it ensures that more important modalities are assigned higher weights in the fusion process, while less important modalities are reduced in weight, thus achieving adaptive fusion of different modalities.

[0347] First, the RGB feature and the thermal imaging feature ( and ) are subjected to global average pooling operation (GAP), and then one-dimensional convolution (1D Convolution) is used to extract higher-level information from the global feature representation, obtaining global features G RGB and G Thermal .

[0348]

[0349] Here, the GAP operation compresses each modality feature into a fixed-size global feature vector, reducing the complexity of the feature space while preserving important global information. By applying the Softmax function, the global feature is converted into an adaptive weight vector corresponding to the RGB and thermal imaging features.

[0350]

[0351] ω Thermal = 1-ω RGB

[0352] where G RGB and G THermal are the global features of RGB and thermal images respectively; ω RGB represents the weight of the RGB image, and ω Thermal represents the weight of the thermal image. The Softmax function ensures the normalization of the weight vector, so that the sum of all weights is 1, thus facilitating the subsequent weighted operation.

[0353] The obtained weight vector is multiplied with the features of the corresponding modalities element by element to generate weighted features, and the weighted RGB features and thermal imaging features are added to obtain the final fusion feature representation:

[0354]

[0355] wherein ω RGB , ω Thermal is the weight corresponding to RGB, Thermal, and are the features of RGB and Thermal after multi-scale convolution, respectively. is the initial RGB_Thermal fusion feature of RGB and Thermal.

[0356] The environment recognition module 103 is configured to convert the LiDAR point cloud generation data into LiDAR point cloud features; perform feature fusion on the LiDAR point cloud features and the initial RGB_Thermal fusion features to obtain multi-modal BEV perspective features, and perform environment recognition according to the multi-modal BEV perspective features.

[0357] Exemplarily, the LiDAR point cloud generation data in the multi-modal data set is extracted, and the point cloud generation data is converted into LiDAR point cloud features, which can be realized by the following implementation steps:

[0358] The LiDAR point cloud feature extraction module processes the point cloud by voxelization method. The point cloud is voxelized into a three-dimensional voxel representation:

[0359]

[0360] Then the LiDAR point cloud is input into the LiDAR encoder, and the features are extracted through multi-layer perception (MLP) and maximum pooling (MaxPool) operations. The input is set to n x 4 point cloud data, and the output is n x 512 feature representation.

[0361] A LiDAR =MaxPool(MLP(P LiDAR ))

[0362] wherein P LiDAR represents the input LiDAR point cloud data.

[0363] After projection to the BEV perspective, a 2D convolutional neural network (CNN) is used to extract high-level features to provide unified BEV perspective features for subsequent fusion with RGB-Thermal

[0364]

[0365] 105: A point (u, v) in the RGB-Thermal image coordinate system is associated with a LiDAR point cloud coordinate (X, Y, Z) through the camera intrinsic matrix K and the extrinsic T (the transformation matrix from camera to LiDAR):

[0366]

[0367] where, K is the camera intrinsic matrix, R is the rotation matrix, t is the translation vector, and s is the scale factor.

[0368] RGB-Thermal view features are projected into BEV, and pre-fusion is performed to obtain RGB_Thermal image features is a multi-view feature, each feature point of which corresponds to a pixel point (u, v), and the depth information Z LiDAR is calculated to obtain its corresponding BEV coordinate (X, Y).

[0369]

[0370] where, x c y is the camera optical center, and (f x , f y ) is the focal length.

[0371] LiDAR cannot cover all pixel points of the RGB-Thermal image, and depth compensation is performed using LiDAR BEV features:

[0372] Z′=αZ LiDAR +(1-αZ pred )

[0373] where Z pred is the estimated depth of the RGB-Thermal image, and a is a weight coefficient, which is obtained based on the consistency of LiDAR point cloud density and depth:

[0374]

[0375] where N LiDAR represents the number of points of the LiDAR point cloud in a grid element (X, Y) in the BEV view. Lambda is a smoothing factor to prevent the denominator from being zero, usually set to 1, and beta is a hyperparameter (usually between 1-10) that controls the descending speed of a.

[0376] Then a mapping function based on LiDAR features is constructed to project the RGB-Thermal features into the BEV space:

[0377]

[0378] wherein, is a projection mapping operation that maps the features of the FV perspective to the BEV perspective based on the depth Z. is the feature of RGB-Thermal in the BEV perspective.

[0379] As an embodiment of the present application, the LiDAR point cloud feature and the initial RGB_Thermal fusion feature are fused to obtain a multi-modal BEV perspective feature, including:

[0380] The LiDAR point cloud feature is mapped into a query vector in the double attention mechanism, and the initial RGB_Thermal fusion feature is mapped into a key vector and a value vector in the double attention mechanism.

[0381] The attention degree of the LiDAR point cloud feature and the initial RGB_Thermal fusion feature is calculated by using the cross attention in the double attention mechanism, and an initial multi-modal fusion feature is obtained by preliminary fusion.

[0382] The self-attention feature in the initial multi-modal fusion feature is calculated by using the self-attention in the double attention mechanism.

[0383] The initial multi-modal fusion feature and the self-attention feature are spliced to obtain a multi-modal joint feature.

[0384] The spatial feature in the multi-modal joint feature is extracted to obtain a multi-modal BEV perspective feature.

[0385] Exemplarily, the LiDAR point cloud feature and the initial RGB_Thermal fusion feature are fused to obtain a multi-modal BEV perspective feature, which can be realized by the following steps:

[0386] The implementation is realized by using the double attention mechanism, including cross attention (Cross Attention) and self-attention (Self-Attention), and the specific implementation is as follows:

[0387] First, the BEV cross-modal cross-attention fusion is performed, and the feature representation of the LiDAR modality is The feature representation of the RGB-Thermal modality is wherein n is the length of the feature sequence, and m is the feature dimension. By linear projection, the LiDAR feature is mapped into a query vector The RGB-Thermal feature is respectively mapped into a key vector and a value vector

[0388] To reduce the computational complexity and enhance the local feature correlation, cross-attention only computes the attention weights of query position i and the keys and values within its neighborhood p(i). The neighborhood p(i) is a region centered at query position i with size k. For query position i, the attention degree of LiDAR to RGB-Thermal is calculated, and the initial multi-modal fusion feature As follows:

[0389]

[0390] Wherein, x in the above formula i is the attention score is the query vector at the LiDAR query position i; is the key vector of RGB-Thermal, which is mapped using the field p(i) of the query point i when calculating; B (i,p(i)) is the position bias, used to introduce position information, d k is a scaling factor, used to stabilize the gradient, is the value vector corresponding to RGB-Thermal, representing the information provided by this modality; wherein d represents the destination (target) modality, i.e. RGB-Thermal.

[0391] On the basis of the cross-attention mechanism, the self-attention mechanism is used to further strengthen the feature correlation within the fused modal, and the specific implementation is as follows:

[0392] The cross-attention output initial multi-modal fusion feature A c is taken as input, and query vector Q Self , key vector K Self and value vector V Self are generated through linear projection, and the self-attention feature representation of query position i is The calculation is as follows:

[0393]

[0394] Wherein, is the query vector at the query position i; is the corresponding key vector; B (i) is the position bias; d k is the scaling factor;, is the corresponding value vector, and self represents that the query, key and value vectors all come from linear transformation of the unified input.

[0395] The cross-attention output initial multi-modal fusion feature A c and the self-attention output A s are spliced in the feature dimension to obtain the joint feature representation Ajoint

[0396] A joint =[A c ,A s ]

[0397] where [·] denotes the concatenation operation.

[0398] The joint feature A joint is input into the convolutional layer to further extract spatial features and generate the final dual-attention output F output .

[0399] F output =Conv(A joint )

[0400] F output is the final output multi-modal BEV perspective feature map.

[0401] Further, the embodiment of the present application optimizes the computing efficiency of the multi-modal joint generation network through edge computing optimization technology, and adopts the following implementation steps:

[0402] Step 1, structured pruning, using the L1 norm pruning method to calculate the L1 norm of the weight of each convolution channel or attention head, removing redundant convolution channels and connections in Adaptive Multi-Scale Feature Fusion and DualAttention Fusion. For a given convolution kernel weight W, the L1 norm is the sum of the absolute values of its elements

[0403]

[0404] where W ij represents each element in the weight matrix. By calculating the L1 norm of each convolutional layer or attention layer, the channels with L1 norm less than the threshold λ are pruned:

[0405] pruned channels={W i |L1(W i )<λ}

[0406] where W represents the weight matrix of the convolution kernel or attention head

[0407] Low-bit quantization (Low-Bit Quantization) effectively reduces memory occupation and computing demand by converting the network model from 32-bit floating point number to 8-bit integer (INT8 quantization), ensuring efficient operation of the model on edge devices.

[0408] Step 2, Knowledge Distillation, a lightweight model MobileNet is used to learn from the original model (Teacher Model) through knowledge distillation, reducing the model size and improving the inference speed while maintaining high accuracy. Distillation loss function:

[0409] L KD = aL CE (y, p stu ) + (1-a)T 2 L CE (q te , q stu )

[0410] Where L CE (y, p) represents the standard cross-entropy loss function, y is the true label, and p is the predicted value; T is the temperature parameter, adjusting the smoothing degree of the soft label; q te , q stu are the output probability distribution of the teacher model and the student model respectively, and a is the hyperparameter controlling the relative importance of the teacher loss and the student loss.

[0411] Step 3, TensorRT acceleration, to accelerate the inference process, the optimized quantization model is accelerated through TensorRT, and the hardware acceleration function of TensorRT is used to further improve the real-time performance.

[0412] As Figure 8 shown, it is a structural schematic diagram of an electronic device for implementing a multi-modal data enhancement and fusion method according to an embodiment of the present application.

[0413] The electronic device can include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and can also include a computer program stored in the memory 11 and executable on the processor 10, such as a multi-modal data enhancement and fusion method program.

[0414] The processor 10 may, in some embodiments, be composed of integrated circuits, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits of the same or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connects various components of the electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (for example, a multi-modal data augmentation and fusion method program), and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0415] The memory 11 includes at least one type of readable storage medium, including flash memories, mobile hard disks, multimedia cards, card-type memories (for example, SD or DX memories, etc.), magnetic memories, magnetic disks, optical disks, etc. The memory 11 may, in some embodiments, be an internal storage unit of the electronic device, for example, a mobile hard disk of the electronic device. The memory 11 may, in other embodiments, also be an external storage device of the electronic device, for example, a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 may include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, for example, a multi-modal data augmentation and fusion method program code, etc., but also to temporarily store data that has been or will be output.

[0416] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 11, the processor 10, etc.

[0417] The communication interface 13 is used for communication between the electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display information processed in the electronic device and to display a visualized user interface.

[0418] Figure 8 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 8 The structure shown does not constitute a limitation on the electronic device, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0419] For example, although not shown, the electronic device can also include a power supply (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so that functions such as charge management, discharge management, and power consumption management can be realized through the power management device. The power supply can also include one or more direct current or alternating current power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0420] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by this structure.

[0421] The multi-modal data enhancement and fusion method program stored in the memory 11 in the electronic device is a combination of multiple instructions, which, when running in the processor 10, can realize:

[0422] Obtaining multi-modal data and calculating a Gaussian noise vector for each modal data, wherein the multi-modal data includes RGB image data, thermal image data, and LiDAR point cloud data;

[0423] input the Gaussian noise vector of the multi-modal data into the multi-modal joint generation network to obtain a multi-modal data group, wherein the multi-modal data group comprises RGB image generation data, Thermal image generation data and LiDAR point cloud generation data;

[0424] extract the modal feature corresponding to the RGB image generation data and the modal feature corresponding to the Thermal image generation data, and cross-modal fusion is performed on the modal feature corresponding to the RGB image generation data and the modal feature corresponding to the Thermal image generation data to obtain initial RGB_Thermal fusion features;

[0425] convert the LiDAR point cloud generation data into LiDAR point cloud features;

[0426] perform feature fusion on the LiDAR point cloud features and the initial RGB_Thermal fusion features to obtain multi-modal BEV perspective features, and perform environment recognition according to the multi-modal BEV perspective features.

[0427] Specifically, the specific implementation method of the processor 10 on the above instructions can refer to the description of the related steps in the corresponding embodiments of the drawings, and will not be repeated here.

[0428] Further, the modules / units integrated in the electronic device 1 are implemented in the form of software function units and sold or used as independent products, which can be stored in a computer readable storage medium. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory).

[0429] The application also provides a computer readable storage medium, which stores a computer program, and the computer program can realize the following steps when executed by a processor of an electronic device:

[0430] obtain multi-modal data and calculate the Gaussian noise vector of each modal data, wherein the multi-modal data comprises RGB image data, Thermal image data and LiDAR point cloud data;

[0431] input the Gaussian noise vector of the multi-modal data into the multi-modal joint generation network to obtain a multi-modal data group, wherein the multi-modal data group comprises RGB image generation data, Thermal image generation data and LiDAR point cloud generation data;

[0432] The modal feature corresponding to the RGB image generation data and the modal feature corresponding to the thermal image generation data are extracted, and the modal feature corresponding to the RGB image generation data and the modal feature corresponding to the thermal image generation data are cross-modally fused to obtain initial RGB_Thermal fusion features;

[0433] The LiDAR point cloud generation data is converted into LiDAR point cloud features;

[0434] The LiDAR point cloud features and the initial RGB_Thermal fusion features are fused to obtain multi-modal BEV perspective features, and environment recognition is performed according to the multi-modal BEV perspective features.

[0435] In several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there can be another division manner in actual implementation.

[0436] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0437] In addition, each function module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of hardware plus software function modules.

[0438] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0439] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims involved.

[0440] The blockchain referred to in the present application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism and encryption algorithm. The blockchain is essentially a decentralized database, and is a series of data blocks associated using cryptographic methods. Each data block contains information of a batch of network transactions, and is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer and an application service layer.

[0441] Embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0442] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, not any particular order.

[0443] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A multi-modal data augmentation and fusion method, characterized in that, The method comprises: acquiring multi-modal data and calculating a Gaussian noise vector of each modal data, wherein the multi-modal data comprises RGB image data, thermal image data and LiDAR point cloud data; inputting the Gaussian noise vector of the multi-modal data into a multi-modal joint generation network to obtain a multi-modal data group, wherein the multi-modal data group comprises RGB image generation data, thermal image generation data and LiDAR point cloud generation data, and the inputting the Gaussian noise vector of the multi-modal data into the multi-modal joint generation network to obtain the multi-modal data group comprises: normalizing the Gaussian noise vector of each modal data by using a full connection layer of the multi-modal joint generation network to obtain a normalized feature corresponding to each modal data; performing up-sampling on the normalized feature corresponding to each modal data by using a convolution layer of the multi-modal joint generation network to obtain a high-resolution feature map corresponding to each modal data; converting the high-resolution feature map corresponding to each modal data into a heterogeneous graph by using a graph neural network, and updating nodes in the heterogeneous graph by using a linear activation function to obtain an updated heterogeneous graph, wherein the nodes in the heterogeneous graph are feature points of different modalities, and the edges in the heterogeneous graph are correlations between different modalities; calculating an adaptive attention mechanism according to the updated heterogeneous graph, and enhancing the feature expression of each modal data by using the adaptive attention mechanism to obtain enhanced features of each modal data; performing transpose convolution processing on the enhanced features of each modal data by using a convolution layer corresponding to the modal in the multi-modal joint generation network to obtain generation data of different modal data; integrating the generation data of different modal data to obtain the multi-modal data group, wherein after the integration, the method further comprises performing coordinated consistency processing on the multi-modal data group by using a discriminator of the multi-modal joint generation network, and the performing coordinated consistency processing on the multi-modal data group by using the discriminator of the multi-modal joint generation network comprises: inputting the multi-modal data group and real multi-modal data into the discriminator to obtain probability values of different modalities corresponding to the multi-modal data group and the real multi-modal data; calculating modal embedding encodings of different modalities according to the probability values of different modalities corresponding to the multi-modal data group and the real multi-modal data; and performing coordinated consistency processing on the multi-modal data group according to the modal embedding encodings; extracting modal features corresponding to the RGB image generation data and modal features corresponding to the thermal image generation data, and cross-modally fusing the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data to obtain initial RGB_Thermal fusion features; converting the LiDAR point cloud generation data into LiDAR point cloud features; performing feature fusion on the LiDAR point cloud features and the initial RGB_Thermal fusion features to obtain multi-modal BEV perspective features, and performing environment recognition according to the multi-modal BEV perspective features.

2. The multi-modal data augmentation and fusion method of claim 1, wherein, The extracting the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data comprises: The RGB image generation data and the thermal image generation data are converted into an RGB image feature matrix and a thermal image feature matrix; Preliminary RGB image features are obtained by performing image convolution processing according to the number of channels of the RGB image feature matrix; Preliminary thermal image features are obtained by performing image convolution processing according to the number of channels of the thermal image feature matrix; The preliminary RGB image features are dilated by using a dilated convolution layer with a first expansion rate to obtain modal features corresponding to the RGB image generation data; The preliminary thermal image features are dilated by using a dilated convolution layer with a second expansion rate to obtain modal features corresponding to the thermal image generation data.

3. The multi-modal data augmentation and fusion method of claim 1, wherein, The initial RGB_Thermal fusion features are obtained by fusing the modal features corresponding to the RGB image generation data and the modal features corresponding to the thermal image generation data across modalities, including: The correlation between the modal features corresponding to the RGB image generation data and the correlation between the modal features corresponding to the thermal image generation data are calculated by using a cross-attention mechanism, respectively, to obtain a modal correlation matrix corresponding to the RGB image data and a modal correlation matrix corresponding to the thermal image data; A preset number of RGB image feature matrices are obtained by performing feature extraction on the modal features corresponding to the RGB image generation data by using a preset number of convolution layers; A preset number of thermal image feature matrices are obtained by performing feature extraction on the modal features corresponding to the thermal image generation data by using a preset number of convolution layers; The RGB image global features are obtained by performing element-wise multiplication between the modal correlation matrix corresponding to the RGB image data and the preset number of RGB image feature matrices; The thermal image global features are obtained by performing element-wise multiplication between the modal correlation matrix corresponding to the thermal image data and the preset number of thermal image feature matrices; The final RGB image features are obtained by performing element-wise multiplication between the RGB image global features and the modal features corresponding to the RGB image generation data; The final thermal image features are obtained by performing element-wise multiplication between the thermal image global features and the modal features corresponding to the thermal image generation data; The initial RGB_Thermal fusion features are obtained by adaptively fusing the final RGB image features and the final thermal image features.

4. The multi-modal data augmentation and fusion method of claim 1, wherein, The multi-modal BEV perspective features are obtained by fusing the LiDAR point cloud features and the initial RGB_Thermal fusion features, including: The LiDAR point cloud features are mapped into query vectors in a dual attention mechanism, and the initial RGB_Thermal fusion features are mapped into key vectors and value vectors in the dual attention mechanism; The attention degree of the LiDAR point cloud features and the initial RGB_Thermal fusion features is calculated by using cross-attention in the dual attention mechanism, and preliminary multi-modal fusion features are obtained by preliminary fusion. The self-attention feature in the initial multi-modal fusion feature is calculated by using the self-attention in the double attention mechanism; The multi-modal joint feature is obtained by splicing the initial multi-modal fusion feature and the self-attention feature; The multi-modal BEV perspective feature is obtained by extracting the spatial feature in the multi-modal joint feature.

5. A multi-modal data augmentation and fusion apparatus, characterized in that, The device implements the multi-modal data augmentation and fusion method as claimed in any one of claims 1 to 4, and the device comprises: A data generation module is configured to obtain multi-modal data, and calculate a Gaussian noise vector of each modality data, wherein the multi-modal data comprises RGB image data, Thermal image data, and LiDAR point cloud data; and input the Gaussian noise vector of the multi-modal data into a multi-modal joint generation network to obtain a multi-modal data group, wherein the multi-modal data group comprises RGB image generation data, Thermal image generation data, and LiDAR point cloud generation data; A feature fusion module is configured to extract a modality feature corresponding to the RGB image generation data and a modality feature corresponding to the Thermal image generation data, and cross-modality fuse the modality feature corresponding to the RGB image generation data and the modality feature corresponding to the Thermal image generation data to obtain initial RGB_Thermal fusion features; An environment recognition module is configured to convert the LiDAR point cloud generation data into LiDAR point cloud features; fuse the LiDAR point cloud features with the initial RGB_Thermal fusion features to obtain multi-modal BEV perspective features, and recognize an environment according to the multi-modal BEV perspective features.

6. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the multi-modal data augmentation and fusion method as claimed in any one of claims 1 to 4.

7. A computer readable storage medium storing a computer program, wherein the computer program comprises program instructions configured to cause a processor to perform the method according to any one of claims 1 to 6. The computer program is executed by the processor to implement the multi-modal data augmentation and fusion method as claimed in any one of claims 1 to 4.

Images