A lightweight data fusion method based on dynamic multi-scale dual-channel attention

Abstract

The present application relates to a lightweight data fusion method based on dynamic multi-scale dual-channel attention, including: calculating the cross-modal similarity between the point cloud features of the first stage after alignment and the image features of the first stage and the second stage respectively, enhancing the point cloud features of the first stage after alignment based on the two cross-modal similarities, splicing the image features of the first stage, the image features of the second stage and the two enhanced point cloud feature maps, normalizing the splicing results by layers to obtain a first fused feature map; corresponding the image features of the remaining stages to the point cloud features of the remaining stages in sequence one by one, fusing the three corresponding groups of image features and point cloud features respectively to obtain a second fused feature map, a third fused feature map and a fourth fused feature map; passing the four fused feature maps through a global context cascade structure, passing the obtained four global context features through an attention mechanism respectively, and obtaining four levels of fused optimized data respectively.

Description

Technical Field

[0001] The present application relates to the technical field of lightweight data fusion, and in particular to a lightweight data fusion method based on dynamic multi-scale dual-channel attention. Background Art

[0002] In the era of digital information explosion, data is presented in multiple modal forms, including text, images, audio and video, each of which carries unique information value; images can intuitively display visual features, text accurately conveys semantics, and audio transmits sound characteristics. However, single-modal information processing can no longer meet the needs of many fields for high-precision and in-depth information mining. Traditional information fusion methods have obvious limitations. Early fusion methods based on simple rules simply mechanically spliced ​​or added different modal data with fixed weights, without fully considering the complex correlation between modalities, resulting in rough fusion effects and information loss. Although subsequent fusion methods based on shallow models have attempted to capture modal correlations, they are limited by the model's representation capabilities and are difficult to cope with large-scale, highly complex data scenarios. In particular, when processing massive amounts of information with high real-time requirements, it is difficult to strike a balance between efficiency and accuracy.

[0003] With the development of deep learning, convolutional neural networks have demonstrated outstanding performance in single-modal tasks. However, in multimodal fusion scenarios, standard convolutional neural networks lack dynamic and adaptive mechanisms for integrating features from different modalities. While the self-attention mechanism can focus on key parts of the data, its high computational complexity and low processing efficiency when applied directly to large-scale multimodal data make it unsuitable for scenarios with demanding real-time performance. Summary of the Invention

[0004] Based on this, it is necessary to provide a lightweight data fusion method based on dynamic multi-scale dual-channel attention, which includes:

[0005] S1: Acquire image data and point cloud data, and perform stage-by-stage feature extraction on the point cloud data and pre-processed image data, respectively, to obtain four-stage point cloud features and five-stage image features;

[0006] S2: Align the dimensions of the point cloud features in the first stage with the dimensions of the image features; based on the cosine similarity calculation formula, calculate the cross-modal similarity between the aligned point cloud features in the first stage and the image features in the first and second stages respectively, enhance the aligned point cloud features in the first stage based on the two cross-modal similarities, splice the image features in the first stage, the image features in the second stage and the two enhanced point cloud feature maps, and normalize the spliced ​​results by layers to obtain the first fused feature map;

[0007] S3: Match the image features of the remaining stages with the point cloud features of the remaining stages in sequence one by one, and fuse the three sets of corresponding image features and point cloud features through step S2 to obtain the second fused feature map, the third fused feature map, and the fourth fused feature map;

[0008] S4: The four fused feature maps are passed through the global context cascade structure to obtain the first global context feature, the second global context feature, the third global context feature, and the fourth global context feature respectively;

[0009] S5: The four global context features are passed through the attention mechanism respectively to obtain four levels of fusion optimization data.

[0010] Beneficial effects: This method extracts features from point cloud data and preprocessed image data in stages to obtain point cloud features of four stages and image features of five stages respectively; aligns the dimensions of the point cloud features of the first stage with the dimensions of the image features; calculates the cross-modal similarity between the aligned point cloud features of the first stage and the image features of the first and second stages respectively, enhances the aligned point cloud features of the first stage based on the two cross-modal similarities, splices the image features of the first stage, the image features of the second stage and the two enhanced point cloud feature maps, normalizes the splicing results by layers, and obtains a first fused feature map; matches the image features of the remaining stages with the point cloud features of the remaining stages in sequence one by one, and fuses the three corresponding groups of image features and point cloud features to obtain a second fused feature map, a third fused feature map, and a fourth fused feature map; passes the four fused feature maps through a global context cascade structure, and passes the obtained four global context features through an attention mechanism to obtain four levels of fused optimized data respectively. This method effectively addresses the shortcomings of existing technologies, meets the urgent needs of intelligent security, autonomous driving, multimedia content analysis and other fields for efficient and accurate multimodal information processing, and provides strong support for the industry's intelligent upgrade. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a flowchart of a lightweight data fusion method based on dynamic multi-scale dual-channel attention in an embodiment of the present application. DETAILED DESCRIPTION

[0013] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0014] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0015] like Figure 1 As shown, this embodiment provides a lightweight data fusion method based on dynamic multi-scale dual-channel attention, which includes:

[0016] S1: Obtain RGB image data and point cloud data, and perform stage-by-stage feature extraction on the point cloud data and preprocessed image data, respectively, to obtain four-stage point cloud features and five-stage image features.

[0017] In this embodiment, the preprocessing of the image data includes performing adaptive histogram equalization and Gaussian filtering on the acquired image data.

[0018] Furthermore, the pre-processed image data is subjected to stage-by-stage feature extraction, including:

[0019] The preprocessed image data is passed through the shallow and deep layers of the ShuffleNetV2 network, an EfficientNet-Lite network, and two MobileNetV2 networks to obtain the image features of each stage.

[0020] Specifically, the preprocessed image data is input into the ShuffleNetV2 network, and after passing through the first convolutional layer and the maximum pooling layer in the ShuffleNetV2 network, the image features of the first stage are output;

[0021] The image features of the first stage are passed through the subsequent convolutional layers and channel shuffling modules in the ShuffleNetV2 network to output the image features of the second stage.

[0022] ShuffleNetV2 is a lightweight convolutional neural network designed specifically for mobile and embedded devices. It optimizes computational efficiency through channel shuffling and point-by-point group convolution. Its core features include: channel shuffling, which addresses the "channel isolation" issue of group convolution and improves feature flow efficiency through cross-group information exchange; balanced computational complexity, using a uniform channel design to reduce memory access costs and improve inference speed; and a multi-branch structure combined with point-by-point group convolution to maintain feature expressiveness while minimizing computational effort.

[0023] The image features of the second stage are input into the EfficientNet-Lite network, and the image features of the third stage are output.

[0024] EfficientNet-Lite (Lightweight Efficient Network) is a lightweight version of Google's EfficientNet series. Based on a compound scaling strategy (simultaneously optimizing network depth, width, and resolution), it strikes a balance between accuracy and efficiency, making it suitable for edge device deployment. Its core features include compound scaling, which uniformly adjusts network dimensions to avoid the performance bottlenecks of single-dimensional optimization; a lightweight design with a simplified structure (e.g., reducing the number of layers), combining depthwise separable convolutions and squeeze-excitation attention mechanisms to enhance feature sensitivity; and multi-scale adaptability, which provides good generalization capabilities for input images of varying sizes.

[0025] The image features of the third stage are input into the MobileNetV2 network, and the image features of the fourth stage are output.

[0026] The image features of the fourth stage are input into the MobileNetV2 network, and the image features of the fifth stage are output.

[0027] MobileNetV2 (Mobile Neural Network V2) is a classic lightweight convolutional neural network proposed by Google. Based on an inverted residual block and linear bottleneck design, it addresses the information loss problem in low-dimensional feature extraction. Its core features include: inverted residual blocks, which first increase the dimensionality before reducing it, extract spatial features through deep convolution, avoiding the feature sparsity of shallow networks; linear bottlenecks, which use linear activation at the end of residual blocks to preserve the linear information of low-dimensional features and improve expressiveness; and efficient computation, based on depthwise separable convolutions, which significantly reduces parameters and computational complexity.

[0028] Furthermore, the staged feature extraction of point cloud data includes:

[0029] Based on the features of all voxels divided from the point cloud data, the point cloud features of the first stage are obtained; on the basis of the point cloud features of the first stage, three consecutive point cloud feature aggregations with gradually decreasing neighborhoods are performed to obtain the point cloud features of the remaining three stages.

[0030] Specifically, the three-dimensional space of the point cloud data is divided into multiple voxels of the same size; for each voxel, the average coordinates of all the points contained in it are calculated, and the average coordinates are used as the voxel feature of the corresponding voxel;

[0031] All voxel features are input into the convolutional neural network to output the first stage point cloud features.

[0032] This stage mainly extracts low-level features of point cloud data, such as the spatial distribution and local geometric shape of point clouds, etc. These features provide the basis for subsequent feature extraction and fusion.

[0033] For the point cloud features of the first stage, a set of core points (such as the center point of a local area or a key anchor point) are defined in the point cloud data, and the Euclidean distance between each core point and the remaining points is calculated. Based on the distance and the point cloud features of the first stage, the point cloud features of the second stage are calculated. The calculation formula is as follows:

[0034] ;

[0035] in, Indicates the i The second stage point cloud features of the kernel points; Indicates the i The neighborhood point set of a core point; Represents the Gaussian kernel function, which performs nonlinear mapping on the distance. The smaller the distance, the closer the mapping result is to 1. When the distance exceeds the preset threshold, the mapping result approaches 0, which can implicitly define the neighborhood range. Indicates the i The core point and its j The distance between neighboring points; Indicates the j The first-stage point cloud features of the neighboring points are obtained; all the core points in the point cloud data are traversed, and the corresponding first-stage point cloud features are replaced with the second-stage point cloud features of each core point to obtain the second-stage point cloud features.

[0036] This stage further deepens feature extraction, enhancing the expressive power and semantic information of features. The second stage's point cloud features further reduce resolution, but the semantic information becomes more concentrated and abstract. For example, in a complex scene containing multiple objects, the second stage's point cloud features can more accurately identify the local features of different objects.

[0037] For the point cloud features of the second stage, randomly sample the points in the point cloud data. For each sampling point, calculate the second distance between each sampling point and the corresponding neighboring point. Based on the second distance and the point cloud features of the second stage, calculate the point cloud features of the third stage. The calculation formula is as follows:

[0038] ;

[0039] in, Indicates the g The point cloud features of the third stage of sampling points; Indicates the g The neighborhood point set of the sampling point; represents the Gaussian kernel function; Indicates the g The sampling point and its h The second distance between adjacent points; Indicates the h The point cloud features of the second stage of each neighboring point are obtained; all sampling points are traversed, and the point cloud features of the corresponding second stage are replaced with the point cloud features of the third stage of each sampling point to obtain the point cloud features of the third stage.

[0040] The features extracted in this stage reach a higher semantic level, encompassing richer semantic information within the point cloud data. The point cloud features in the third stage have a lower resolution, but the semantic information contained within each feature point is even richer. For example, in a complex scene containing multiple objects, the point cloud features in the third stage can more accurately identify the categories and shape characteristics of different objects.

[0041] For the point cloud features of the third stage, all points in the point cloud data of the sampling points are randomly sampled again. For each second sampling point, the third distance between each second sampling point and the corresponding neighboring point is calculated. Based on the third distance and the point cloud features of the third stage, the point cloud features of the fourth stage are calculated. The calculation formula is as follows:

[0042] ;

[0043] in, Indicates the u Point cloud features of the fourth stage of sampling points; Indicates the u The neighborhood point set of the sampling point; represents the Gaussian kernel function; Indicates the u The sampling point and its v The third distance between adjacent points; Indicates the v The point cloud features of the third stage of the neighboring points are traversed; all the second sampling points are traversed, and the point cloud features of the corresponding third stage are replaced with the point cloud features of the fourth stage of each second sampling point to obtain the point cloud features of the fourth stage.

[0044] The features extracted in this stage reach the highest semantic level, encompassing the most core semantic information in the point cloud data. The point cloud features from the fourth stage have the lowest resolution, but each feature point embodies the most critical semantic information in the point cloud data. For example, in a complex scene containing multiple objects, the point cloud features from the fourth stage can extract the features that best represent the core semantics of the entire scene, providing critical support for final data fusion and decision-making.

[0045] In this embodiment, the point cloud features of the four stages all use the same type of Gaussian kernel function. Only the neighborhood range or kernel function parameters need to be adjusted to adapt to the feature scales of different stages. For example, in the first stage, fine geometric features (such as edges and surface normals) are extracted through a small neighborhood range, and in the subsequent stages, semantic-level features (such as object parts and scene structures) are aggregated through a large neighborhood range to achieve hierarchical expression of point cloud features from local to global.

[0046] S2: Align the dimensions of the point cloud features of the first stage with the dimensions of the image features; based on the cosine similarity calculation formula, calculate the cross-modal similarity between the point cloud features of the first stage after alignment and the image features of the first and second stages respectively, enhance the point cloud features of the first stage after alignment based on the two cross-modal similarities, splice the image features of the first stage, the image features of the second stage and the two enhanced point cloud feature maps, and normalize the splicing results by layer to obtain the first fusion feature map.

[0047] Specifically, the process of obtaining the first fusion feature map includes:

[0048] In the Cross-Modal Attention Module (CMAM),

[0049] Align the dimensions of the point cloud features from the first stage with the dimensions of the image features;

[0050] The image features of the first and second stages and the point cloud features of the first stage after alignment are all convolved through a convolution layer containing a 1×1 convolution kernel;

[0051] Based on the cosine similarity calculation formula, the cross-modal similarities between the point cloud features in the first stage after alignment and the image features in the first and second stages after convolution are calculated, and the first similarity matrix and the second similarity matrix are obtained respectively; the calculation formulas are:

[0052] ;

[0053] ;

[0054] in, represents the first similarity matrix; represents the second similarity matrix; Represents the first 1×1 convolution kernel; Represents the second 1×1 convolution kernel; Represents a module; Represents the first stage of image features a Row, No. b Elements of the column; Represents the image features in the second stage a Row, No. b Elements of the column; Represents the first stage of point cloud features after alignment k Row, No. l Elements of the column;

[0055] Based on the first similarity matrix and the point cloud features of the first stage after alignment, the first fused point cloud feature map is obtained, and the calculation formula is:

[0056] ;

[0057] in, Represents the point cloud feature map after the first fusion; represents the softmax activation function; C represents the number of channels of the point cloud features in the first stage; Represents the point cloud features of the first stage after alignment;

[0058] Based on the second similarity matrix and the point cloud features of the first stage after alignment, the second fused point cloud feature map is obtained. The calculation formula is:

[0059] ;

[0060] in, Represents the second fused point cloud feature map;

[0061] The image features of the first stage, the image features of the second stage, the first fused point cloud feature map and the second fused point cloud feature map are spliced ​​together, and the spliced ​​feature map is normalized layer by layer to obtain the first fused feature map.

[0062] By fusing the low-level and mid-level features of RGB images with the low-level features of point clouds through CMAM, we lay the foundation for subsequent feature extraction and fusion, ensure the consistency and completeness of basic details, and introduce some mid-level features to enhance the expression ability.

[0063] S3: Match the image features of the remaining stages with the point cloud features of the remaining stages in sequence one by one, and fuse the three sets of corresponding image features and point cloud features through step S2 to obtain the second fused feature map, the third fused feature map, and the fourth fused feature map.

[0064] Specifically, step S2 is repeatedly performed based on the image features of the remaining stages and the point cloud features of the remaining stages;

[0065] The dimensions of the point cloud features of the second stage are aligned with the dimensions of the image features; the cross-modal similarity between the image features of the third stage and the aligned point cloud features of the second stage is calculated; and the point cloud features of the second stage and the image features of the third stage are fused based on the cross-modal similarity to obtain a second fused feature map.

[0066] By fusing the high-level semantic features of RGB images with the concentrated abstract features of point clouds through CMAM, the semantic information can be complemented and enhanced, and the semantic richness of features can be improved.

[0067] The dimensions of the point cloud features of the third stage are aligned with the dimensions of the image features; the cross-modal similarity between the image features of the fourth stage and the aligned point cloud features of the third stage is calculated; and the point cloud features of the third stage and the image features of the fourth stage are fused based on the cross-modal similarity to obtain a third fused feature map.

[0068] Through CMAM, the semantically concentrated abstract features of RGB images are integrated with the higher semantic features of point clouds to further improve the semantic expression ability and spatial consistency of features.

[0069] The dimensions of the point cloud features of the fourth stage are aligned with the dimensions of the image features; the cross-modal similarity between the image features of the fifth stage and the aligned point cloud features of the fourth stage is calculated; and the point cloud features of the fourth stage and the image features of the fifth stage are fused based on the cross-modal similarity to obtain a fourth fused feature map.

[0070] By fusing the highest semantic features of RGB images and point clouds through CMAM, the core semantic information is complemented and enhanced, providing the most powerful feature support for the final multimodal application task.

[0071] The phased features extracted from RGB images and point cloud data are input into CMAM in the specified cascade order. CMAM's cross-modal similarity modeling, dynamic feature weighting and multi-granularity alignment mechanism are used to achieve cross-modal fusion of features at each level, giving full play to the advantages of RGB images and point cloud data at different levels, improving the expressive power and semantic richness of the fused features, and laying the foundation for subsequent tasks.

[0072] By fusing the phased features of each level of RGB image and point cloud data into CMAM in a specified cascade order, we can fully leverage the advantages of each at different levels. Low-level fusion ensures the consistency of basic details, mid-level fusion enhances the expressiveness of features, and high-level fusion complements and strengthens semantic information. This gradual fusion approach ensures that the final fused features contain both rich detailed information and strong semantic expression capabilities, better supporting subsequent multimodal application tasks such as object detection, classification, and segmentation.

[0073] S4: The four fused feature maps are passed through a global context cascade structure to obtain a first global context feature, a second global context feature, a third global context feature, and a fourth global context feature, respectively.

[0074] Specifically, this step includes:

[0075] The global context cascade structure is a multi-level feature fusion mechanism designed to integrate multi-scale information through feature interactions at different levels, thereby enhancing the model's understanding and perception of images. This structure consists of multiple GCM modules, each responsible for processing features at a specific level. Through upsampling and feature fusion operations, it transfers high-level global information to lower levels to enrich the semantic representation of low-level features. Specifically, the global context cascade structure typically includes a fifth global context module, a fourth global context module, a third global context module, and a second global context module, which are connected sequentially from high to low levels to form a top-down feature enhancement path.

[0076] In the fifth global context module, the fourth fused feature map is sequentially subjected to dilated convolution and global average pooling to extract global context information; the global context information is passed through the second convolutional layer to adjust its number of channels and dimension to obtain the first global context feature.

[0077] In the fourth global context module, the first global context feature is upsampled (such as bilinear interpolation or transposed convolution) to align its resolution with the third fused feature map; the upsampled first global context feature is fused with the third fused feature map through element-level addition or splicing operation, and the first fusion result is refined through the third convolution layer and activation function in sequence to obtain the second global context feature.

[0078] In the third global context module, the second global context feature is upsampled, and the first global context feature is upsampled again, so that the resolution of the first global context feature and the second global context feature are aligned with the second fused feature map; the second fused feature map is fused with the upsampled second global context feature and the upsampled first global context feature through a splicing operation, and the second fusion result is enhanced by the fourth convolution layer and the activation function in sequence to obtain the third global context feature.

[0079] In the second global context module, the third global context feature is upsampled, the second global context feature is upsampled again, and the first global context feature is upsampled for a third time, so that the resolutions of the first global context feature, the second global context feature, and the third global context feature are aligned with the first fusion feature map; the first fusion feature map is fused with the upsampled third global context feature, the upsampled second global context feature again, and the upsampled first global context feature for the third time through a splicing operation, and the third fusion result is refined and enhanced through the fifth convolutional layer and the activation function in sequence to obtain the fourth global context feature.

[0080] In this embodiment, the above activation functions are all ReLU activation functions.

[0081] The global context cascade structure uses a top-down feature fusion approach to gradually transfer high-level global context information to lower levels and integrate it with the low-level detailed features. This multi-scale feature interaction and fusion mechanism has the following functions and advantages:

[0082] 1. Combining global and local information: High-level features provide the overall semantics and context of an image, while low-level features contain rich details and pixel-level information. Through a cascaded structure, the model can simultaneously utilize both types of information, improving its understanding and perception of complex scenes.

[0083] 2. Feature Enhancement and Refinement: Each GCM module enhances and refines features to a certain extent, removing irrelevant or noisy information and highlighting key features. This gradual feature optimization process helps improve the robustness and accuracy of the model.

[0084] 3. Multi-scale feature representation: The cascade structure integrates features from different levels to form a feature representation that contains multi-scale information. This is crucial for capturing objects and structures of different sizes and shapes in images, and can improve the generalization and adaptability of the model.

[0085] 4. Efficient information transfer: Through upsampling and feature fusion operations, high-level global information can be effectively transferred to low-level layers, avoiding information loss during downsampling. At the same time, it reduces computational complexity and parameter count, and improves model efficiency.

[0086] In summary, the global context cascade structure is an effective multi-scale feature fusion mechanism. Through orderly module connections and feature interactions, it provides the model with a more comprehensive and accurate feature representation, thereby improving its performance in various computer vision tasks.

[0087] S5: The four global context features are passed through the attention mechanism respectively to obtain four levels of fusion optimization data.

[0088] Specifically, this step includes:

[0089] In the dynamic multi-scale dual-channel attention module,

[0090] For any global context feature,

[0091] It is subjected to global average pooling, adaptive average pooling, and local maximum pooling respectively, and the three pooling results are fused through splicing operations to obtain the fused pooling result;

[0092] The fusion pooling result is passed through the first layer of multi-layer perceptron to reduce the dimension and obtain the dimension reduction result;

[0093] The dimensionality reduction feature is passed through the ReLU activation function to obtain the first activation result;

[0094] The first activation result is subjected to feature extraction by the second-layer multi-layer perceptron to obtain a first extracted feature;

[0095] The first extracted feature is passed through the ReLU activation function to obtain the second activation result;

[0096] The second activation result is passed through the third layer of multi-layer perceptron to restore the number of channels and obtain the restored result;

[0097] The restored result is passed through the sigmoid activation function and the channel attention weight is output;

[0098] Multiply the channel attention weight by the first adjustment coefficient to obtain the updated channel attention weight;

[0099] Perform channel enhancement on the selected global context features based on the updated channel attention weights to obtain channel enhanced features;

[0100] Perform deformable convolution on the channel-enhanced features to obtain the convolved features;

[0101] The convolutional features are subjected to maximum pooling and average pooling, and the edge features are detected by the Canny operator. The features after maximum pooling and average pooling are fused with the detected edge features through splicing operations to obtain fused spatial features.

[0102] The fused spatial features are sequentially passed through the sixth convolutional layer and the activation function for spatial enhancement to obtain spatially enhanced features;

[0103] The spatially enhanced features are passed through the self-attention mechanism to obtain self-attention features;

[0104] The spatial enhancement features are fused with the self-attention features through a splicing operation to obtain features combined with the self-attention mechanism;

[0105] The features combined with the self-attention mechanism are subjected to convolution operation to output convolution features;

[0106] The convolution features are upsampled by the bilinear interpolation upsampling method to obtain the fused optimized data corresponding to the selected global context features;

[0107] Traverse all global context features, pass all global context features through the attention mechanism respectively, and obtain the fusion optimization data corresponding to each global context feature.

[0108] The dynamic multi-scale dual-channel attention module also includes:

[0109] 1. Adjust the order of attention calculations: Based on the characteristics of the input multimodal data and specific task requirements, in some cases, spatial attention calculations are performed first, followed by channel attention calculations. This allows the model to first focus on key spatial locations in the data, and then specifically enhance the features of these locations in the channel dimension, thereby more accurately capturing complex multimodal information and providing more valuable feature representations for subsequent improved data generation.

[0110] 2. Multi-scale Attention Mechanism: A multi-scale feature pyramid is constructed, and a dynamic multi-scale dual-channel attention module is applied to all global context features. The dynamic multi-scale dual-channel attention module on low-resolution features focuses on global semantic information, while the dynamic multi-scale dual-channel attention module on high-resolution features focuses on local detail features. The self-attention mechanism-based features obtained from each global context feature after passing through the dynamic multi-scale dual-channel attention module are then fused through a concatenation operation. This allows the model to comprehensively utilize multi-scale information, better adapt to multimodal data with different characteristics, and generate more comprehensive fused and optimized data based on the fused features.

[0111] 3. Reduce the amount of calculation and the number of parameters;

[0112] Low-rank approximation optimization: For the fully connected layer in channel attention, assuming that the original fully connected layer weight matrix is , is the dimension of the fully connected layer weight; through the low-rank approximation method, it is decomposed into two low-dimensional matrices and The product of . In this way, the original matrix multiplication operation amount is reduced from Reduced to , significantly reducing the number of parameters and computational complexity with almost no loss of performance.

[0113] Pruning operation: by calculating the connection weight Gradient To evaluate its importance, we can generally use the method based on the absolute value of the gradient, that is, to calculate the absolute value of the gradient of each connection weight . Set a threshold ,for During the pruning process, ensure that the performance loss of the model is within an acceptable range and that the pruned model can still maintain good performance.

[0114] Through the above method, the dynamic multi-scale dual-channel attention module can reduce the consumption of computing resources while ensuring the feature optimization effect, and improve the efficiency and scalability of the model.

[0115] The dynamic multi-scale dual-channel attention module achieves refined optimization of features by adjusting the attention calculation order, optimizing channel attention calculation, enhancing spatial attention calculation, combining self-attention mechanism, introducing multi-scale attention mechanism, dynamically adjusting attention weights, and reducing the amount of calculation and the number of parameters.

[0116] The advantages of fusion features are reflected here:

[0117] The fusion features generated by the cross-modal attention module and the dynamic multi-scale dual-channel attention module significantly enhance the complementarity and discriminability of the features. Specifically:

[0118] Semantic and geometric alignment: Multi-granularity alignment mechanisms (such as high-level semantic alignment, mid-level component alignment, and low-level pixel-point alignment) enable fused features to simultaneously preserve the texture details of RGB images and the three-dimensional spatial structure of point cloud data, reducing feature ambiguity, thereby reducing confusion during model training and accelerating convergence.

[0119] Multi-scale feature enhancement: The dynamic multi-scale dual-channel attention module uses a multi-scale pyramid structure and dynamic weight adjustment to generate features that can capture key information at different resolution levels, improving the model's adaptability to complex scenarios and further reducing the MSE and SSIM loss values.

[0120] Attention-guided optimization: The attention weights in the fused features (such as channel attention and spatial attention) directly guide the loss function to focus on key areas, such as object edges or sparse point cloud areas, thereby optimizing the model's ability to model local details.

[0121] In this embodiment, the first adjustment coefficient is obtained by dynamically learning the auxiliary network according to the input data; the input of the auxiliary network is any global context feature, and the output is the first adjustment coefficient.

[0122] In this embodiment, it also includes: calculating a loss function based on the fused optimized data and its corresponding original expected data, performing backpropagation based on the loss function, and adjusting network parameters until the loss function converges; the network parameters include the weights and biases of each convolutional layer.

[0123] Furthermore, the loss function construction process includes:

[0124] Based on the fusion optimization data and its corresponding original expected data, the mean square error loss is calculated by the mean square error loss function;

[0125] Based on the fusion optimization data and its corresponding original expected data, the structural similarity loss is calculated by the structural similarity loss function;

[0126] The loss function is constructed based on the mean square error loss and the structural similarity loss. The expression of the loss function is:

[0127] ;

[0128] in, represents the loss function; represents the weight coefficient; represents the mean square error loss; represents the structural similarity loss.

[0129] This embodiment also includes establishing a database and properly storing the resulting fusion optimization data. An appropriate storage method is selected based on the data type. Structured data can be stored in a relational database such as MySQL, with a well-designed table structure and necessary indexes to speed up queries. Unstructured or semi-structured data can be stored in a document-based database such as MongoDB. Furthermore, data should be regularly backed up to ensure security and integrity, creating a comprehensive database that facilitates future searches and comparisons.

[0130] The lightweight data fusion method based on dynamic multi-scale dual-channel attention provided in this embodiment has the following beneficial effects:

[0131] In the multimodal feature extraction stage, feature extraction is performed on RGB images and point cloud data in stages to mine features at different levels. Secondly, for cross-modal feature fusion, an innovative cross-modal attention module is introduced. Through cross-modal similarity modeling, dynamic feature weighting, and a multi-granularity alignment mechanism, the advantages of RGB images and point cloud data are complemented. Thirdly, for data fusion optimization, a dynamic multi-scale dual-channel attention module is proposed. This module achieves refined feature optimization by adjusting the attention calculation order, optimizing channel attention calculation, enhancing spatial attention calculation, combining self-attention mechanisms, introducing multi-scale attention mechanisms, dynamically adjusting attention weights, and reducing the amount of computation and parameters. Finally, in terms of cascade structure design, a cascade structure of global context modules is adopted to transfer high-level global information from top to bottom, enriching the semantic representation of low-level features and enhancing the model's understanding and perception of images. These innovations work together to effectively address the shortcomings of existing technologies, meet the urgent needs of intelligent security, autonomous driving, multimedia content analysis and other fields for efficient and accurate multimodal information processing, and provide strong support for the intelligent upgrade of the industry.

[0132] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A lightweight data fusion method based on dynamic multi-scale dual-channel attention, characterized by: include: S1: Acquire image data and point cloud data, and perform stage-by-stage feature extraction on the point cloud data and pre-processed image data, respectively, to obtain four-stage point cloud features and five-stage image features; Based on the features of all voxels divided from the point cloud data, the point cloud features of the first stage are obtained. Based on the point cloud features of the first stage, three point cloud feature aggregations with gradually decreasing neighborhoods are performed in succession to obtain the point cloud features of the remaining three stages. The staged feature extraction of preprocessed image data includes: The preprocessed image data is input into the ShuffleNetV2 network, and after passing through the first convolutional layer and the maximum pooling layer in the ShuffleNetV2 network, the image features of the first stage are output; The image features of the first stage are passed through the subsequent convolutional layers and channel shuffle modules in the ShuffleNetV2 network to output the image features of the second stage; Input the image features of the second stage into the EfficientNet-Lite network and output the image features of the third stage; Input the image features of the third stage into the MobileNetV2 network and output the image features of the fourth stage; Input the image features of the fourth stage into the MobileNetV2 network and output the image features of the fifth stage; S2: Align the dimensions of the point cloud features in the first stage with the dimensions of the image features; based on the cosine similarity calculation formula, calculate the cross-modal similarity between the aligned point cloud features in the first stage and the image features in the first and second stages respectively, enhance the aligned point cloud features in the first stage based on the two cross-modal similarities, splice the image features in the first stage, the image features in the second stage and the two enhanced point cloud feature maps, and normalize the spliced ​​results by layers to obtain the first fused feature map; S3: Match the image features of the remaining stages with the point cloud features of the remaining stages in sequence one by one, and perform feature fusion on the three sets of corresponding image features and point cloud features through step S2 to obtain the second fused feature map, the third fused feature map, and the fourth fused feature map; Align the dimensions of the point cloud features in the second, third, and fourth stages with the dimensions of the image features in the corresponding stages; Calculating the cross-modal similarity between the image features of the remaining stage and the point cloud features of the corresponding stage after alignment, and fusing the image features of the remaining stage with the point cloud features of the corresponding stage based on the cross-modal similarity to obtain the second, third, and fourth fused feature maps respectively; S4: The four fused feature maps are passed through the global context cascade structure to obtain the first global context feature, the second global context feature, the third global context feature, and the fourth global context feature respectively; S5: The four global context features are passed through the attention mechanism respectively to obtain four levels of fusion optimization data.

2. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 1 is characterized in that The preprocessing of image data includes performing adaptive histogram equalization and Gaussian filtering on the acquired image data.

3. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 1 is characterized in that The staged feature extraction of point cloud data includes: The three-dimensional space of the point cloud data is divided into multiple voxels of the same size; for each voxel, the average coordinates of all the points contained in it are calculated, and the average coordinates are used as the voxel feature of the corresponding voxel; Input all voxel features into the convolutional neural network and output the first stage point cloud features; For the point cloud features of the first stage, a set of core points are defined in the point cloud data, and the distance between each core point and the remaining points is calculated. Based on the distance and the point cloud features of the first stage, the point cloud features of the second stage are calculated. The calculation formula is as follows: ； in, Indicates the i The second stage point cloud features of the kernel points; Indicates the i The neighborhood point set of a core point; represents the Gaussian kernel function; Indicates the i The core point and its j The distance between neighboring points; Indicates the j The first-stage point cloud features of the neighboring points are obtained; all the core points in the point cloud data are traversed, and the corresponding first-stage point cloud features are replaced with the second-stage point cloud features of each core point to obtain the second-stage point cloud features; For the point cloud features of the second stage, randomly sample the points in the point cloud data. For each sampling point, calculate the second distance between each sampling point and the corresponding neighboring point. Based on the second distance and the point cloud features of the second stage, calculate the point cloud features of the third stage. The calculation formula is as follows: ； in, Indicates the g The point cloud features of the third stage of sampling points; Indicates the g The neighborhood point set of the sampling point; Indicates the g The sampling point and its h The second distance between adjacent points; Indicates the h The second-stage point cloud features of the neighboring points are obtained; all sampling points are traversed, and the corresponding second-stage point cloud features are replaced with the third-stage point cloud features of each sampling point to obtain the third-stage point cloud features; For the point cloud features of the third stage, all points in the point cloud data of the sampling points are randomly sampled again. For each second sampling point, the third distance between each second sampling point and the corresponding neighboring point is calculated. Based on the third distance and the point cloud features of the third stage, the point cloud features of the fourth stage are calculated. The calculation formula is as follows: ； in, Indicates the u Point cloud features of the fourth stage of sampling points; Indicates the u The neighborhood point set of the sampling point; Indicates the u The sampling point and its v The third distance between adjacent points; Indicates the v The point cloud features of the third stage of the neighboring points are traversed; all the second sampling points are traversed, and the point cloud features of the corresponding third stage are replaced with the point cloud features of the fourth stage of each second sampling point to obtain the point cloud features of the fourth stage.

4. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 1 is characterized in that The process of obtaining the first fusion feature map includes: Align the dimensions of the point cloud features from the first stage with the dimensions of the image features; The image features of the first and second stages and the point cloud features of the first stage after alignment are all convolved through a convolution layer containing a 1×1 convolution kernel; Based on the cosine similarity calculation formula, the cross-modal similarities between the point cloud features in the first stage after alignment and the image features in the first and second stages after convolution are calculated, and the first similarity matrix and the second similarity matrix are obtained respectively; the calculation formulas are: ； ； in, represents the first similarity matrix; represents the second similarity matrix; Represents the first 1×1 convolution kernel; Represents the second 1×1 convolution kernel; Represents a module; Represents the first stage of image features a Row, No. b Elements of the column; Represents the image features in the second stage a Row, No. b Elements of the column; Indicates the first stage of point cloud features after alignment k Row, No. l Elements of the column; Based on the first similarity matrix and the point cloud features of the first stage after alignment, the first fused point cloud feature map is obtained, and the calculation formula is: ； in, Represents the point cloud feature map after the first fusion; represents the softmax activation function; C represents the number of channels of the point cloud features in the first stage; Represents the point cloud features of the first stage after alignment; Based on the second similarity matrix and the point cloud features of the first stage after alignment, the second fused point cloud feature map is obtained. The calculation formula is: ； in, Represents the second fused point cloud feature map; The image features of the first stage, the image features of the second stage, the first fused point cloud feature map and the second fused point cloud feature map are spliced ​​together, and the spliced ​​feature map is normalized layer by layer to obtain the first fused feature map.

5. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 1 is characterized in that S4 include: The global context cascade structure includes a fifth global context module, a fourth global context module, a third global context module, and a second global context module; In the fifth global context module, the fourth fused feature map is sequentially subjected to dilated convolution and global average pooling to extract global context information; The global context information is passed through the second convolutional layer to adjust its channel number and dimension to obtain the first global context feature; In the fourth global context module, the first global context feature is upsampled to align its resolution with the third fused feature map; the upsampled first global context feature is fused with the third fused feature map through element-wise addition or concatenation, and the first fusion result is refined through the third convolutional layer and activation function to obtain the second global context feature; In the third global context module, the second global context feature is upsampled, and the first global context feature is upsampled again, so that the resolution of the first global context feature and the second global context feature are aligned with the second fused feature map; the second fused feature map is fused with the upsampled second global context feature and the upsampled first global context feature through a splicing operation, and the second fusion result is sequentially enhanced by the fourth convolution layer and the activation function to obtain the third global context feature; In the second global context module, the third global context feature is upsampled, the second global context feature is upsampled again, and the first global context feature is upsampled for a third time, so that the resolutions of the first global context feature, the second global context feature, and the third global context feature are aligned with the first fusion feature map; the first fusion feature map is fused with the upsampled third global context feature, the upsampled second global context feature again, and the upsampled first global context feature for the third time through a splicing operation, and the third fusion result is refined and enhanced through the fifth convolutional layer and the activation function in sequence to obtain the fourth global context feature.

6. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 1, characterized in that S5 include: For any global context feature, It is subjected to global average pooling, adaptive average pooling, and local maximum pooling respectively, and the three pooling results are fused through splicing operations to obtain the fused pooling result; The fusion pooling result is passed through the first layer of multi-layer perceptron to reduce the dimension and obtain the dimension reduction result; The dimensionality reduction feature is passed through the ReLU activation function to obtain the first activation result; The first activation result is subjected to feature extraction by the second-layer multi-layer perceptron to obtain a first extracted feature; The first extracted feature is passed through the ReLU activation function to obtain the second activation result; The second activation result is passed through the third layer of multi-layer perceptron to restore the number of channels and obtain the restored result; The restored result is passed through the sigmoid activation function and the channel attention weight is output; Multiply the channel attention weight by the first adjustment coefficient to obtain the updated channel attention weight; Perform channel enhancement on the selected global context features based on the updated channel attention weights to obtain channel enhanced features; Perform deformable convolution on the channel-enhanced features to obtain the convolved features; The convolutional features are subjected to maximum pooling and average pooling, and the edge features are detected by the Canny operator. The features after maximum pooling and average pooling are fused with the detected edge features through splicing operations to obtain fused spatial features. The fused spatial features are sequentially passed through the sixth convolutional layer and the activation function for spatial enhancement to obtain spatially enhanced features; The spatially enhanced features are passed through the self-attention mechanism to obtain self-attention features; The spatial enhancement features are fused with the self-attention features through a splicing operation to obtain features combined with the self-attention mechanism; The features combined with the self-attention mechanism are subjected to convolution operation to output convolution features; The convolution features are upsampled by the bilinear interpolation upsampling method to obtain the fused optimized data corresponding to the selected global context features; Traverse all global context features, pass all global context features through the attention mechanism respectively, and obtain the fusion optimization data corresponding to each global context feature.

7. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 6 is characterized in that The first adjustment coefficient is obtained by dynamic learning of the auxiliary network according to the input data; the input of the auxiliary network is any global context feature, and the output is the first adjustment coefficient.

8. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 6 is characterized in that The loss function is calculated based on the fused optimized data and its corresponding original expected data, and backpropagation is performed based on the loss function to adjust the network parameters until the loss function converges; the network parameters include the weights and biases of each convolutional layer.

9. The lightweight data fusion method based on dynamic multi-scale dual-channel attention according to claim 8 is characterized in that The loss function construction process includes: Based on the fusion optimization data and its corresponding original expected data, the mean square error loss is calculated by the mean square error loss function; Based on the fusion optimization data and its corresponding original expected data, the structural similarity loss is calculated by the structural similarity loss function; The loss function is constructed based on the mean square error loss and the structural similarity loss. The expression of the loss function is: ； in, represents the loss function; represents the weight coefficient; represents the mean square error loss; represents the structural similarity loss.

Images