Cross-modal feature fusion method based on attention mechanism

Abstract

The invention discloses a cross-modal feature fusion method based on an attention mechanism, and the method comprises the following steps: 1, inputting original image data into an improved SegNeXt network; step 2, inputting the convolution attention feature map into a dual-gating module; 3, inputting the feature sequence and the dynamic agent mark into an agent attention module to obtain a first modal high-level feature sequence; 4, inputting the first modal high-level feature sequence to a fourth stage; 5, inputting the second modal original data into a second modal coding network to obtain a second modal feature sequence; step 6, inputting the updated first modal high-level feature sequence and the second modal feature sequence into a multi-scale proxy diffusion module; and 7, inputting the cross-modal fusion feature sequence into a task decoding network to obtain a target task output result. The method is suitable for complex task scenes such as image and text, image and voice multi-modal information collaborative understanding and semantic segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic understanding and artificial intelligence, and particularly relates to a cross-modal feature fusion method based on an attention mechanism. BACKGROUND

[0002] In the semantic understanding task of computer vision and artificial intelligence, semantic segmentation, as a basic and key technology, has been widely used in automatic driving, medical image analysis, remote sensing recognition, human-computer interaction and other fields. Traditional semantic segmentation methods mostly rely on the pixel-by-pixel classification ability of convolutional neural networks (CNN) on images, and representative methods such as FCN, U-Net and subsequent multi-level networks based on encoder-decoder structure have achieved good results in a single visual modality. However, with the increase of task complexity, relying only on a single modality for semantic segmentation often faces problems such as incomplete information, large semantic differences between modalities and strong scene uncertainty, resulting in a significant decrease in segmentation accuracy in the case of blurred boundaries, adjacent heterogeneous objects or complex backgrounds.

[0003] In order to overcome the above problems, in recent years, researchers have gradually introduced cross-modal learning into the semantic segmentation task, trying to fuse information from multiple modalities (such as images and text, RGB and depth, infrared and visible light, etc.) to improve the completeness and robustness of semantic representation. Typical cross-modal methods mostly use feature splicing, alignment or attention mechanism to fuse different modalities to improve segmentation performance. However, the current fusion strategies mostly have the following shortcomings: first, there is a lack of effective inter-modal structure matching mechanism, which easily leads to semantic drift or ambiguity of fused features; second, the attention mechanism usually only relies on static weight calculation, which cannot fully exploit the dynamic interaction between features, especially when dealing with high-dimensional, multi-scale complex data, the performance is limited; third, in the context of large-scale data, the traditional attention operation has high computational cost, and there is a lack of efficient and scalable feature modeling method.

[0004] Although existing methods try to introduce proxy attention, gating mechanism and other strategies to alleviate the above problems, most of them are shallow designs, lacking systematic optimization of proxy structure, gating strategy and multi-scale interaction, resulting in limited accuracy, weak generalization ability and unstable structure of cross-modal semantic segmentation in real-world scenarios.

[0005] Therefore, how to provide a cross-modal feature fusion method based on an attention mechanism is a problem that those skilled in the art need to solve. SUMMARY

[0006] One objective of this invention is to propose a cross-modal feature fusion method based on an attention mechanism. This method combines deep neural network structures, attention mechanisms, multi-scale feature modeling, and diffusion operations to improve the fusion efficiency and expressive power of different modal features in multimodal semantic understanding tasks. Based on an improved SegNeXt network, this invention designs a multi-stage feature extraction process consisting of a dual-gating module and a surrogate attention module, enabling the extraction of multi-level, multi-granular semantic information from the original image. Simultaneously, a multi-scale surrogate diffusion mechanism is constructed to establish dynamic surrogate representations at multiple scales, enhancing intermodal information interaction through diffusion mapping and feature propagation. A cross-modal fusion module achieves deep fusion of aligned features, improving feature consistency and semantic integrity. This invention fully integrates the advantages of attention mechanisms, adaptive gating strategies, K-means clustering, diffusion mapping, and multi-scale feature modeling, possessing advantages such as sufficient semantic expression, high feature alignment accuracy, and strong cross-modal generalization ability, making it suitable for semantic segmentation and understanding tasks in multimodal scenarios.

[0007] A cross-modal feature fusion method based on an attention mechanism according to an embodiment of the present invention includes the following steps:

[0008] Step 1: Obtain the original image data of the first modality, input the original image data into the improved SegNeXt network, and perform the first and second stage preprocessing in sequence to obtain the feature map of the first modality in the intermediate layer.

[0009] Step 2: Input the first modality feature map of the intermediate layer into the third stage, perform convolutional attention processing to obtain the convolutional attention feature map, input the convolutional attention feature map into the dual gating module, perform dual gating processing, including channel gating processing and spatial gating processing, to obtain the third stage feature map;

[0010] Step 3: Flatten the feature map of the third stage to obtain the feature sequence, perform clustering operation on the feature sequence to obtain N dynamic agent labels, and input the feature sequence and dynamic agent labels into the agent attention module. In the agent attention module, perform attention calculation from feature to agent and attention calculation from agent to feature to obtain the first modality high-level feature sequence.

[0011] Step 4: Input the first modality high-level feature sequence into the fourth stage. In the fourth stage, convolutional attention processing, dual gating processing, feature flattening, clustering operation to generate dynamic surrogate labels, and surrogate attention processing are executed sequentially according to the processing flow of Step 2 and Step 3 to obtain the updated first modality high-level feature sequence.

[0012] Step 5: Obtain the raw data of the second modality, input the raw data of the second modality into the second modality coding network, perform feature extraction, and obtain the second modality feature sequence;

[0013] Step 6: Input the updated first modality high-level feature sequence and the second modality feature sequence into the multi-scale proxy diffusion module to construct the proxy feature sequence, and perform diffusion mapping processing and scale fusion processing at each scale to obtain the cross-modality aligned feature sequence;

[0014] Step 7: Input the cross-modal alignment feature sequence into the cross-modal fusion module, perform attention weight calculation on the cross-modal alignment feature sequence relative to the cross-modal proxy label, and generate the cross-modal fusion feature sequence based on the attention weight; input the cross-modal fusion feature sequence into the task decoding network to obtain the target task output result.

[0015] Optionally, the improved SegNeXt network includes a first stage, a second stage, a third stage, a fourth stage, a second modality coding network, a multi-scale proxy diffusion module, a cross-modal feature fusion module, and a task decoding network; the third and fourth stages are internally equipped with a dual gating module and a proxy attention module.

[0016] Optionally, the preprocessing in the first and second stages specifically includes:

[0017] The original image data of the first modality is acquired and input into the first stage of the improved SegNeXt network as input data.

[0018] In the first stage, the original image data of the first modality is subjected to the first convolution processing, the first convolution attention processing, and the first downsampling processing in sequence to obtain the first stage output feature map;

[0019] The output feature map from the first stage is used as input data to the second stage of the improved SegNeXt network;

[0020] In the second stage, the output feature map of the first stage is subjected to the second convolution processing, the second convolution attention processing, and the second downsampling processing in sequence to obtain the first modality feature map of the intermediate layer.

[0021] Optionally, step two specifically includes:

[0022] The first modality feature map of the intermediate layer is used as input data to the third stage of the improved SegNeXt network;

[0023] In the third stage, the first modality feature map of the intermediate layer is input into the convolutional attention module to perform convolutional attention processing and obtain the convolutional attention feature map;

[0024] The convolutional attention feature map is input into the dual gating module, where channel gating and spatial gating are performed sequentially.

[0025] Channel gating processing includes extracting channel statistics from the convolutional attention feature map along the channel dimension;

[0026] Channel weights are generated based on channel statistics, and then channel weights are multiplied channel by channel with the convolutional attention feature map to obtain the channel-gated feature map;

[0027] Spatial gating processing includes generating spatial weights in the spatial dimension of the channel gating feature map, and multiplying the spatial weights with the channel gating feature map point by point in the spatial dimension to obtain the third-stage feature map.

[0028] Optionally, step three specifically includes:

[0029] The third-stage feature map is flattened in space to form a one-dimensional feature sequence, thus obtaining the first feature sequence.

[0030] Perform K-means clustering on the first feature sequence to divide it into N clusters, and calculate the cluster center of each cluster.

[0031] Each cluster center is denoted as a dynamic proxy tag, resulting in N dynamic proxy tags;

[0032] The first feature sequence and N dynamic agent tags are input into the agent attention module as input data.

[0033] The surrogate attention module includes feature-to-surrogate attention units and surrogate-to-feature attention units.

[0034] In the feature-to-agent attention unit, feature-to-agent attention computation is performed on the first feature sequence and N dynamic agent tags to obtain intermediate agent features based on dynamic agent tags;

[0035] In the proxy-to-feature attention unit, proxy-to-feature attention computation is performed on the intermediate proxy features and the first feature sequence to obtain the first modality high-level feature sequence.

[0036] Optionally, step four specifically includes:

[0037] The high-level feature sequence of the first mode is used as input data and fed into the fourth stage of the improved SegNeXt network;

[0038] In the fourth stage, the high-level feature sequence of the first modality is input into the convolutional attention module to obtain the convolutional attention feature map of the fourth stage.

[0039] The fourth-stage convolutional attention feature map is input into the dual-gating module to obtain the fourth-stage feature map;

[0040] The fourth-stage feature map is flattened into a one-dimensional feature sequence in the spatial dimension to obtain the second feature sequence; K-means clustering operation is performed on the second feature sequence to divide the second feature sequence into N clusters, and the cluster center of each cluster is calculated. The cluster center is used as a dynamic proxy label to obtain N dynamic proxy labels.

[0041] The second feature sequence and N dynamic agent labels are input into the agent attention module to obtain the updated first modality high-level feature sequence.

[0042] Optionally, step five specifically includes:

[0043] Obtain the raw data of the second mode and input the raw data of the second mode as input data into the second mode coding network;

[0044] In the second modality coding network, the original data of the second modality is subjected to second modality preprocessing operation to obtain second modality preprocessed data;

[0045] The preprocessed data of the second modality is input into the second modality feature extraction subnetwork. Convolution processing, attention processing, and downsampling or pooling processing are performed sequentially in the second modality feature extraction subnetwork to obtain the intermediate feature sequence of the second modality.

[0046] The intermediate feature sequence of the second modality is subjected to feature transformation and dimension alignment processing. The transformed intermediate feature sequence of the second modality is rearranged into a second modality feature sequence that matches the high-level feature sequence of the first modality in feature dimension.

[0047] Optionally, step six specifically includes:

[0048] The updated first modality high-level feature sequence and the second modality feature sequence are input into the multi-scale proxy diffusion module as input data;

[0049] In the multi-scale proxy diffusion module, the updated first modality high-level feature sequence and the second modality feature sequence are subjected to multi-scale decomposition processing to obtain at least two scales of first modality sub-feature sequence and at least two scales of second modality sub-feature sequence;

[0050] At each scale, feature aggregation operations are performed on the first modality sub-feature sequence and the second modality sub-feature sequence at the same scale to obtain the first modality aggregated feature and the second modality aggregated feature;

[0051] Perform feature mapping processing on the first modality aggregated features and the second modality aggregated features, combine the mapped features to construct a surrogate representation at the current scale, and generate a surrogate feature sequence at the current scale;

[0052] At each scale, the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence at the same scale are subjected to diffusion mapping processing;

[0053] Scale fusion processing is performed on the surrogate feature sequences and sub-feature sequences after diffusion mapping at each scale. The fused features are then recombined along the scale dimension to obtain cross-modal aligned feature sequences.

[0054] Optionally, the diffusion mapping process performed on the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence of the same scale is specifically as follows:

[0055] Map the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence of the same scale to the same dimensional space;

[0056] Interactive diffusion propagation processing is performed on the mapped first modality sub-feature sequences, second modality sub-feature sequences, and surrogate feature sequences at the same scale. Feature diffusion weights are calculated based on the correlation between the first modality sub-feature sequences, second modality sub-feature sequences, and surrogate feature sequences at the same scale.

[0057] Furthermore, feature diffusion weights are used to bidirectionally diffuse and propagate feature information from the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence at the same scale.

[0058] Perform iterative update operations on the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence at the same scale after interactive diffusion propagation processing to obtain the surrogate feature sequence and the sub-feature sequence updated at the current scale.

[0059] Optionally, step seven specifically includes:

[0060] The cross-modal aligned feature sequence is input into the cross-modal fusion module. In the cross-modal fusion module, feature aggregation and feature mapping operations are performed on the cross-modal aligned feature sequence to obtain the cross-modal surrogate tag sequence.

[0061] In the cross-modal fusion module, query feature mapping, key feature mapping, and value feature mapping are performed on the cross-modal aligned feature sequence and the cross-modal surrogate tag sequence, respectively, to obtain query features, key features, and value features for attention computation.

[0062] Attention weights are calculated based on query features and key features. In the cross-modal fusion module, weighted aggregation processing is performed on the value features according to the attention weights to generate a cross-modal fusion feature sequence.

[0063] The cross-modal fusion feature sequence is input into the task decoding network. The task decoding network performs feature reconstruction and output mapping on the cross-modal fusion feature sequence to obtain the target task output result.

[0064] The beneficial effects of this invention are:

[0065] The attention-based cross-modal feature fusion method proposed in this invention demonstrates significant technical advantages in multimodal data understanding and fusion through the collaborative design of an improved SegNeXt network, a proxy attention mechanism, and a multi-scale proxy diffusion module.

[0066] This invention introduces a dual-gating module and a proxy attention module into the SegNeXt network structure, which explicitly model the convolutional attention feature map in terms of channel dimension and spatial dimension, respectively, effectively enhancing the saliency expression of local regions. At the same time, through the dynamic proxy modeling mechanism, it improves the generalization ability of high-level semantic features, making the feature extraction process more discriminative and adaptive.

[0067] By introducing K-means clustering to generate dynamic proxy labels in the third and fourth stages, and performing bidirectional interaction operations from feature to proxy and from proxy to feature in the proxy attention module, the correlation between local features and global semantics is further strengthened, which helps to build robust feature representations and provides high-quality semantic support for subsequent cross-modal fusion.

[0068] This invention innovatively proposes a multi-scale surrogate diffusion mechanism. During the feature alignment stage between the first and second modalities, it performs multi-scale decomposition, surrogate construction, diffusion mapping, and scale fusion on features of different modalities, fully mining the complementary information of different modalities at each scale. This mechanism strengthens the connections between modalities through interactive diffusion propagation, while suppressing feature redundancy and noise, ensuring consistency and discriminability during feature alignment.

[0069] By enhancing the attention of the aligned feature input to the cross-modal fusion module, the semantic consistency and recognition ability of the final fused representation are improved, providing a more semantically representative input for the task decoding network and effectively improving the performance of downstream tasks (such as semantic segmentation, image-text matching, etc.).

[0070] This invention not only enhances the expressive power of single-modal feature extraction, but also constructs an efficient cross-scale interaction mechanism in the process of multimodal information fusion, achieving accurate, efficient and robust cross-modal feature alignment and fusion. Attached Figure Description

[0071] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0072] Figure 1 This is an overall flowchart of a cross-modal feature fusion method based on an attention mechanism proposed in this invention;

[0073] Figure 2 This is a schematic diagram of the improved SegNeXt network structure for a cross-modal feature fusion method based on an attention mechanism proposed in this invention.

[0074] Figure 3 This is a schematic diagram illustrating the generation of dynamic proxy tags using K-means clustering in a cross-modal feature fusion method based on an attention mechanism proposed in this invention. Detailed Implementation

[0075] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0076] refer to Figures 1-3 A cross-modal feature fusion method based on an attention mechanism includes the following steps:

[0077] Step 1: Obtain the original image data of the first modality, input the original image data into the improved SegNeXt network, and perform the first and second stage preprocessing in sequence to obtain the feature map of the first modality in the intermediate layer.

[0078] Step 2: Input the first modality feature map of the intermediate layer into the third stage, perform convolutional attention processing to obtain the convolutional attention feature map, input the convolutional attention feature map into the dual gating module, perform dual gating processing, including channel gating processing and spatial gating processing, to obtain the third stage feature map;

[0079] Step 3: Flatten the feature map of the third stage to obtain the feature sequence, perform clustering operation on the feature sequence to obtain N dynamic agent labels, and input the feature sequence and dynamic agent labels into the agent attention module. In the agent attention module, perform attention calculation from feature to agent and attention calculation from agent to feature to obtain the first modality high-level feature sequence.

[0080] Step 4: Input the first modality high-level feature sequence into the fourth stage. In the fourth stage, convolutional attention processing, dual gating processing, feature flattening, clustering operation to generate dynamic surrogate labels, and surrogate attention processing are executed sequentially according to the processing flow of Step 2 and Step 3 to obtain the updated first modality high-level feature sequence.

[0081] Step 5: Obtain the raw data of the second modality, input the raw data of the second modality into the second modality coding network, perform feature extraction, and obtain the second modality feature sequence;

[0082] Step 6: Input the updated first modality high-level feature sequence and the second modality feature sequence into the multi-scale proxy diffusion module to construct the proxy feature sequence, and perform diffusion mapping processing and scale fusion processing at each scale to obtain the cross-modality aligned feature sequence;

[0083] Step 7: Input the cross-modal alignment feature sequence into the cross-modal fusion module, perform attention weight calculation on the cross-modal alignment feature sequence relative to the cross-modal proxy label, and generate the cross-modal fusion feature sequence based on the attention weight; input the cross-modal fusion feature sequence into the task decoding network to obtain the target task output result.

[0084] In this embodiment, the improved SegNeXt network includes a first stage, a second stage, a third stage, a fourth stage, a second modality coding network, a multi-scale proxy diffusion module, a cross-modal feature fusion module, and a task decoding network; the third and fourth stages are internally equipped with a dual gating module and a proxy attention module.

[0085] In this embodiment, the preprocessing in the first and second stages specifically includes:

[0086] The original image data of the first modality is acquired and input into the first stage of the improved SegNeXt network as input data.

[0087] In the first stage, the original image data of the first modality is subjected to the first convolution processing, the first convolution attention processing, and the first downsampling processing in sequence to obtain the first stage output feature map;

[0088] The output feature map from the first stage is used as input data to the second stage of the improved SegNeXt network;

[0089] In the second stage, the output feature map of the first stage is subjected to the second convolution processing, the second convolution attention processing, and the second downsampling processing in sequence to obtain the first modality feature map of the intermediate layer.

[0090] In this embodiment, step two specifically includes:

[0091] The first modality feature map of the intermediate layer is used as input data to the third stage of the improved SegNeXt network;

[0092] In the third stage, the first modality feature map of the intermediate layer is input into the convolutional attention module to perform convolutional attention processing and obtain the convolutional attention feature map;

[0093] The convolutional attention feature map is input into the dual gating module, where channel gating and spatial gating are performed sequentially.

[0094] Channel gating processing includes extracting channel statistics from the convolutional attention feature map along the channel dimension. The channel statistics include the mean, maximum, and summation result.

[0095] Channel weights are generated based on channel statistics, and then channel weights are multiplied channel by channel with the convolutional attention feature map to obtain the channel-gated feature map;

[0096] Spatial gating processing includes generating spatial weights in the spatial dimension of the channel gating feature map, and multiplying the spatial weights with the channel gating feature map point by point in the spatial dimension to obtain the third-stage feature map.

[0097] This invention provides a cross-modal feature fusion method based on an attention mechanism. In step two, a convolutional attention module and a dual-gating module are introduced to perform progressively refined processing on the intermediate layer feature map of the first modality. Specifically, the intermediate layer first modality feature map is first input into the convolutional attention module to extract a more discriminative convolutional attention feature map. In the dual-gating module, channel gating and spatial gating are performed sequentially. Channel gating is performed by extracting statistics (including average, maximum, and summation results) of the convolutional attention feature map in the channel dimension to generate channel weights, which are then multiplied with the original feature map channel by channel to obtain a channel-gated feature map. Then, spatial weights are generated through the spatial dimension and multiplied point by point with the channel-gated feature map to obtain a third-stage feature map with enhanced spatial structure. This invention, by combining an attention mechanism and dual gating, significantly improves the information representation and discrimination capabilities of the feature map in both spatial and channel aspects, laying a higher-quality feature foundation for subsequent proxy modeling and cross-modal fusion.

[0098] In this embodiment, step three specifically includes:

[0099] The third-stage feature map is flattened in space to form a one-dimensional feature sequence, thus obtaining the first feature sequence.

[0100] Perform K-means clustering on the first feature sequence to divide the first feature sequence into N clusters, and calculate the cluster center of each cluster. The cluster center can be obtained by averaging the feature vectors belonging to the same cluster in each dimension.

[0101] Each cluster center is denoted as a dynamic proxy tag, resulting in N dynamic proxy tags;

[0102] The first feature sequence and N dynamic agent tags are input into the agent attention module as input data.

[0103] The surrogate attention module includes feature-to-surrogate attention units and surrogate-to-feature attention units.

[0104] In the feature-to-agent attention unit, feature-to-agent attention computation is performed on the first feature sequence and N dynamic agent tags to obtain intermediate agent features based on dynamic agent tags;

[0105] In the proxy-to-feature attention unit, proxy-to-feature attention computation is performed on the intermediate proxy features and the first feature sequence to obtain the first modality high-level feature sequence.

[0106] This invention provides a cross-modal feature fusion method based on an attention mechanism. In step three, the third-stage feature map is represented in a structured and efficient manner through dynamic proxy modeling and proxy attention mechanism. Specifically, the third-stage feature map is first flattened into a one-dimensional feature sequence in the spatial dimension to generate a first feature sequence. Then, K-means clustering is performed on this feature sequence to divide it into N clusters, and the center of each cluster is calculated as a dynamic proxy label to form a representative proxy feature set. Next, the first feature sequence and the dynamic proxy label are input into the proxy attention module. Within the module, attention calculations from features to proxy and from proxy to features are performed sequentially to achieve bidirectional interactive updates of information, outputting a first-modal high-level feature sequence. This invention can effectively compress redundant features, highlight key regions, and enhance the expressive power and context modeling ability of features, thereby improving the representation accuracy and efficiency of subsequent cross-modal fusion.

[0107] In this embodiment, step four specifically includes:

[0108] The high-level feature sequence of the first mode is used as input data and fed into the fourth stage of the improved SegNeXt network;

[0109] In the fourth stage, the high-level feature sequence of the first modality is input into the convolutional attention module to obtain the convolutional attention feature map of the fourth stage.

[0110] The fourth-stage convolutional attention feature map is input into the dual-gating module to obtain the fourth-stage feature map;

[0111] The fourth-stage feature map is flattened into a one-dimensional feature sequence in the spatial dimension to obtain the second feature sequence; K-means clustering operation is performed on the second feature sequence to divide the second feature sequence into N clusters, and the cluster center of each cluster is calculated. The cluster center is used as a dynamic proxy label to obtain N dynamic proxy labels.

[0112] The second feature sequence and N dynamic proxy tags are input into the proxy attention module. In the proxy attention module, feature-to-proxy attention calculation is performed on the second feature sequence and the N dynamic proxy tags in the feature-to-proxy attention unit to obtain the second intermediate proxy feature. In the proxy-to-feature attention unit, proxy-to-feature attention calculation is performed on the second intermediate proxy feature and the second feature sequence to obtain the updated first modality high-level feature sequence.

[0113] In the fourth stage, this invention further introduces convolutional attention and dual gating mechanisms to perform deep structural modeling of the high-level feature sequence of the first modality, enhancing the local representation ability and channel and spatial responsiveness of the features. The processed fourth-stage feature map is flattened into a second feature sequence, and K-means clustering is used to generate a second round of dynamic proxy labels, thereby extracting representative key feature expressions. By inputting the second feature sequence and dynamic proxy labels into the proxy attention module, bidirectional attention interactions are performed between features and proxy, further integrating feature semantics and contextual relationships. This invention, through this phased, multi-layered attention modeling approach, effectively improves the semantic abstraction and structural preservation capabilities of the first modality features, laying a solid foundation for subsequent cross-modal alignment and fusion.

[0114] In this embodiment, step five specifically includes:

[0115] Obtain the raw data of the second mode and input the raw data of the second mode as input data into the second mode coding network;

[0116] In the second modality coding network, the original data of the second modality is subjected to second modality preprocessing operations to obtain second modality preprocessed data; the preprocessing operations include format normalization, feature normalization and feature encoding.

[0117] The preprocessed data of the second modality is input into the second modality feature extraction subnetwork. Convolution processing, attention processing, and downsampling or pooling processing are performed sequentially in the second modality feature extraction subnetwork to obtain the intermediate feature sequence of the second modality.

[0118] The intermediate feature sequence of the second modality is subjected to feature transformation and dimension alignment processing. The transformed intermediate feature sequence of the second modality is rearranged into a second modality feature sequence that matches the high-level feature sequence of the first modality in feature dimension.

[0119] In this embodiment, step six specifically includes:

[0120] The updated first modality high-level feature sequence and the second modality feature sequence are input into the multi-scale proxy diffusion module as input data;

[0121] In the multi-scale proxy diffusion module, the updated first modality high-level feature sequence and the second modality feature sequence are subjected to multi-scale decomposition processing to obtain at least two scales of first modality sub-feature sequence and at least two scales of second modality sub-feature sequence;

[0122] At each scale, feature aggregation operations are performed on the first modality sub-feature sequence and the second modality sub-feature sequence at the same scale to obtain the first modality aggregated feature and the second modality aggregated feature;

[0123] Perform feature mapping processing on the first modality aggregated features and the second modality aggregated features, combine the mapped features to construct a surrogate representation at the current scale, and generate a surrogate feature sequence at the current scale;

[0124] At each scale, the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence at the same scale are subjected to diffusion mapping processing;

[0125] Scale fusion processing is performed on the surrogate feature sequences and sub-feature sequences after diffusion mapping at each scale. The fused features are then recombined along the scale dimension to obtain cross-modal aligned feature sequences.

[0126] In step six, this invention introduces a multi-scale proxy diffusion module to achieve deep alignment and structured fusion of features from the first and second modalities. Specifically, after inputting the updated high-level feature sequences of the first and second modalities into the multi-scale proxy diffusion module, multi-scale decomposition is first performed on the two types of features to obtain sub-feature sequences at at least two scales. At each scale, feature aggregation and feature mapping are performed on the two modal sub-features to construct a proxy feature sequence for the current scale, thereby generating a proxy representation that can represent the semantic relationship at that scale. Based on this, the two modal sub-feature sequences and the proxy feature sequences are jointly subjected to diffusion mapping processing, enabling features to complete bidirectional diffusion and association strengthening across modalities within the same scale. Further, scale fusion processing is performed on the diffused proxy features and sub-features at each scale, obtaining structurally consistent and semantically aligned cross-modal aligned feature sequences through cross-scale recombination. This invention significantly improves the precision and consistency of cross-modal feature interactions through the combined effects of multi-scale structural modeling, proxy construction, and diffusion propagation, providing a more representative and fusionable aligned feature foundation for subsequent cross-modal fusion stages.

[0127] In this embodiment, the diffusion mapping process performed on the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence of the same scale specifically involves:

[0128] Map the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence of the same scale to the same dimensional space;

[0129] Interactive diffusion propagation processing is performed on the mapped first modality sub-feature sequences, second modality sub-feature sequences, and surrogate feature sequences at the same scale. Feature diffusion weights are calculated based on the correlation between the first modality sub-feature sequences, second modality sub-feature sequences, and surrogate feature sequences at the same scale.

[0130] Furthermore, feature diffusion weights are used to bidirectionally diffuse and propagate feature information from the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence at the same scale.

[0131] Perform iterative update operations on the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence at the same scale after interactive diffusion propagation processing to obtain the surrogate feature sequence and the sub-feature sequence updated at the current scale.

[0132] The interactive diffusion and propagation process is as follows:

[0133] Feature mapping processing is performed on the first modality sub-feature sequence, the second modality sub-feature sequence, and the surrogate feature sequence at the same scale to ensure that the three types of feature sequences maintain consistency in feature dimensions. The feature mapping processing is implemented through linear mapping.

[0134] After feature mapping is completed, correlation calculation is performed based on the relationships between the three types of feature sequences after mapping. The correlation calculation is performed according to the similarity between sequences, obtaining the correlation between the first modality sub-feature sequence and the surrogate feature sequence, the correlation between the second modality sub-feature sequence and the surrogate feature sequence, and the correlation between the two modality sub-feature sequences. Feature diffusion weights are generated based on the correlation results.

[0135] After obtaining the feature diffusion weights, a bidirectional diffusion update operation is performed on the three types of feature sequences. The diffusion update includes diffusing the surrogate feature sequence according to its weights to the first modality sub-feature sequence and the second modality sub-feature sequence, ensuring that the information of the surrogate feature propagates within the same scale of modal features; simultaneously, the first and second modality sub-feature sequences are diffusing to the surrogate feature sequence according to their corresponding weights, achieving reverse propagation of modal features to surrogate features. After the bidirectional diffusion update, a fusion process is performed on the three types of feature sequences. The fusion process uses weighted fusion to obtain the intermediate feature sequence for this round of diffusion.

[0136] Interactive diffusion propagation processing is performed iteratively, performing feature mapping, correlation calculation, feature diffusion weight generation, bidirectional diffusion update, and diffusion result fusion operations cyclically within a set number of iterations, so that the surrogate feature sequence, the first modality sub-feature sequence, and the second modality sub-feature sequence obtain updated feature representations after gradual diffusion on the same scale.

[0137] In step six, this invention introduces a multi-scale proxy diffusion module to achieve deeper structural alignment and feature enhancement between cross-modal features. Specifically, the updated first-modal high-level feature sequence and the second-modal feature sequence are used as input. Multi-scale decomposition yields sub-feature representations at multiple scales. At each scale, feature aggregation and feature mapping are performed to construct proxy representations. Finally, diffusion mapping is performed on the sub-features and proxy features, thereby achieving cross-modal feature interaction and complementarity within the same scale. Further, after diffusion is completed at all scales, scale fusion processing is performed to structurally reorganize and integrate information from multiple scales, resulting in a cross-modal aligned feature sequence. This invention, by integrating strategies such as scale decomposition, proxy construction, diffusion mapping, and fusion reorganization, not only improves the fine-grained expressive power of semantic alignment between different modalities but also enhances the consistency of multi-scale semantic expression. This provides richer, more closely related, and structurally connected input features for cross-modal feature fusion and decoding tasks, ultimately improving the overall cross-modal inference and prediction performance.

[0138] In this embodiment, step seven specifically includes:

[0139] The cross-modal aligned feature sequence is input into the cross-modal fusion module. In the cross-modal fusion module, feature aggregation and feature mapping operations are performed on the cross-modal aligned feature sequence to obtain the cross-modal surrogate tag sequence.

[0140] In the cross-modal fusion module, query feature mapping, key feature mapping, and value feature mapping are performed on the cross-modal aligned feature sequence and the cross-modal surrogate tag sequence, respectively, to obtain query features, key features, and value features for attention computation.

[0141] Attention weights are calculated based on query features and key features. In the cross-modal fusion module, weighted aggregation processing is performed on the value features according to the attention weights to generate a cross-modal fusion feature sequence.

[0142] The cross-modal fusion feature sequence is input into the task decoding network. The task decoding network performs feature reconstruction and output mapping on the cross-modal fusion feature sequence to obtain the target task output result.

[0143] Example 1:

[0144] To verify the feasibility and effectiveness of the attention-based cross-modal feature fusion method proposed in this invention in practical tasks, the invention was applied to a multimodal defect identification scenario in an intelligent industrial inspection platform. During daily operation, this industrial platform needs to jointly analyze surface images (first mode) collected on the production line and vibration sequence data output by structural sensors (second mode) to determine whether products have latent or surface defects. However, traditional single-modal detection methods often rely on visual images for identification. For samples with fine textures, complex appearances, or significant surface noise interference, the identification accuracy is low. Especially when there is no direct correspondence between structural vibration signals and image information, traditional methods struggle to achieve effective fusion, resulting in high false positive and false negative rates.

[0145] In this industrial scenario, this invention constructs an improved SegNeXt network to perform layer-by-layer convolutional attention enhancement and dual-gating processing on images acquired from the production line, fully expressing the local structure and channel importance of image features. Simultaneously, a surrogate attention module is introduced, which performs clustering on the image feature sequences, generates dynamic surrogate labels, and establishes a bidirectional attention mapping from features to surrogates and from surrogates to features, thereby strengthening the high-level semantic features of key product areas. The structural vibration signal of the second modality is processed by an encoding network, transforming the vibration data into a sequence consistent with the visual modality in feature dimensions through format standardization, feature normalization, convolutional coding, and feature alignment. Subsequently, using the multi-scale surrogate diffusion module proposed in this invention, through multi-scale decomposition, scale-level surrogate construction, diffusion mapping, and scale fusion, the deep semantic relationships between the two modalities are gradually aligned and propagated, strengthening the structural correspondence between modalities and providing tightly aligned feature representations for the final fusion stage.

[0146] To conduct quantitative verification, this embodiment selected 3000 industrial product sample data, including visual images and vibration signals acquired simultaneously. The data acquisition process lasted two months, with no fewer than 50 batches of samples collected daily. The product types covered 12 different process models, and the defect types included 9 complex categories such as cracks, peeling, dents, and abnormal vibration. The method of this invention was compared with three typical cross-modal detection methods: a traditional visual CNN model, an image-vibration fusion model based on unidirectional alignment (Method A), and a bimodal model based on attention-weighted fusion (Method B), to comprehensively evaluate the performance of this invention in complex scenarios. Specific experimental data are shown in Table 1.

[0147] Table 1. Performance Comparison of the Invention Method and Typical Comparative Methods in Industrial Defect Detection

[0148] Method name Detection accuracy (%) Defect positioning accuracy (%) Missed detection rate (%) False detection rate (%) Average processing time (ms) Subjective score (10 full marks) Traditional visual CNN model 82.5 78.4 12.3 11.4 109 6.5 One-way alignment image vibration fusion (A) 85.1 81.7 10.2 9.6 119 7.2 Attention weighted bimodal model (B) 87.3 84.5 8.9 7.8 122 7.8 The method of the present application (whole process) 93.8 90.6 4.8 3.9 128 9.1

[0149] In actual testing, the method of this invention first performs convolutional attention and gating enhancement on the image modal to strengthen the surface features of the product; then, it suppresses redundant information through dynamic proxy modeling and strengthens the structural mapping between modalities at different scales by relying on a multi-scale diffusion mechanism, thus solving the fusion failure problem caused by the inconsistency of modal structures in traditional methods.

[0150] The results in Table 1 show that traditional visual models experience a significant drop in recognition rate under conditions of noise, occlusion, or complex surface textures. Methods A and B, due to their relatively weak integration of structural and visual modalities, still exhibit some errors. In contrast, the method of this invention can continuously enhance important features within a multi-scale, multi-modal complementary system, resulting in stable and superior performance on complex backgrounds and samples with weak features.

[0151] After statistical evaluation of all test samples, this invention achieved optimal results in multiple indicators, including final detection accuracy, defect localization accuracy, false negative rate, false positive rate, and consistency of fused features. For example, in terms of defect type recognition accuracy, the traditional visual CNN model had an average accuracy of 82.5%, Method A 85.1%, and Method B 87.3%, while this invention achieved 93.8%. Regarding the false positive rate on complex working condition samples, the traditional method exceeded 11%, while this invention reduced the false positive rate to below 4%. In terms of processing latency, the fusion operation of this invention took an average of 128ms, only about 6ms more than the bimodal attention model, still meeting the real-time requirements of industrial platforms. Furthermore, in the subjective evaluation of 15 quality inspectors, the defect localization results generated by this invention received an average score of 9.1 (out of 10), significantly higher than the comparison methods.

[0152] This embodiment fully demonstrates the application value of the present invention in cross-modal industrial inspection tasks. Through improved SegNeXt encoding, dynamic proxy attention, multi-scale diffusion alignment and cross-modal fusion strategy, it successfully solves the problems of structural inconsistency, insufficient feature representation and weak adaptability to complex scenes in traditional single-modal and simple bimodal fusion methods, making the multimodal detection results more accurate, robust and deployable.

[0153] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An attention mechanism-based cross-modal feature fusion method, characterized in that, Comprising the following steps: Step one, obtaining the original image data of the first modality, inputting the original image data into the improved SegNeXt network, sequentially performing the preprocessing of the first stage and the second stage, and obtaining the intermediate layer first modality feature map; Step two, inputting the intermediate layer first modality feature map into the third stage, performing convolution attention processing to obtain the convolution attention feature map, inputting the convolution attention feature map into the double gate module, and performing double gate processing, including channel gate processing and spatial gate processing, to obtain the third stage feature map; Step three, performing feature flattening on the third stage feature map to obtain a feature sequence, performing clustering operation on the feature sequence to obtain N dynamic proxy labels, and inputting the feature sequence and the dynamic proxy labels into the proxy attention module to perform feature-to-proxy attention calculation and proxy-to-feature attention calculation in the proxy attention module, thereby obtaining the first modality high-level feature sequence; Step four, inputting the first modality high-level feature sequence into the fourth stage, and sequentially performing convolution attention processing, double gate processing, feature flattening, clustering operation to generate dynamic proxy labels, and proxy attention processing in the fourth stage according to the processing procedures of steps two and three, thereby obtaining an updated first modality high-level feature sequence; Step five, obtaining the original data of the second modality, inputting the second modality original data into the second modality encoding network for feature extraction, and obtaining the second modality feature sequence; Step six: inputting the updated first modality high-level feature sequence and the second modality feature sequence into the multi-scale proxy diffusion module to construct proxy feature sequences, and performing diffusion mapping processing and scale fusion processing on each scale to obtain cross-modality alignment feature sequences; Step seven, inputting the cross-modality alignment feature sequence into the cross-modality fusion module, performing attention weight calculation on the cross-modality alignment feature sequence relative to the cross-modality proxy label, and generating a cross-modality fusion feature sequence according to the attention weight; Inputting the cross-modality fusion feature sequence into the task decoding network to obtain the target task output result. 2.The cross-modal feature fusion method based on attention mechanism according to claim 1, characterized in that, The improved SegNeXt network comprises a first stage, a second stage, a third stage, a fourth stage, a second modality encoding network, a multi-scale proxy diffusion module, a cross-modality feature fusion module, and a task decoding network; the third stage and the fourth stage are internally provided with a double gate module and a proxy attention module. 3.The method of claim 1, wherein, The preprocessing of the first stage and the second stage is specifically: Obtaining the original image data of the first modality, inputting the original image data of the first modality as input data into the first stage of the improved SegNeXt network; In the first stage, the original image data of the first modality is sequentially subjected to first convolution processing, first convolution attention processing, and first downsampling processing to obtain a first stage output feature map; Inputting the first stage output feature map as input data into the second stage of the improved SegNeXt network; In the second stage, the first stage output feature map is sequentially subjected to second convolution processing, second convolution attention processing, and second downsampling processing to obtain an intermediate layer first modality feature map.

4. The cross-modal feature fusion method based on attention mechanism according to claim 1, characterized in that, The step two is specifically: inputting the intermediate layer first modality feature map into a third stage of the improved SegNeXt network as input data; in the third stage, inputting the intermediate layer first modality feature map into a convolution attention module to perform convolution attention processing and obtain a convolution attention feature map; inputting the convolution attention feature map into a double gating module to sequentially perform channel gating processing and spatial gating processing in the double gating module; the channel gating processing comprises extracting channel statistics of the convolution attention feature map in the channel dimension; generating channel weights according to the channel statistics, and multiplying the channel weights with the convolution attention feature map in the channel dimension to obtain a channel gated feature map; the spatial gating processing comprises generating spatial weights of the channel gated feature map in the spatial dimension, and multiplying the spatial weights with the channel gated feature map in the spatial dimension point by point to obtain a third stage feature map.

5. The cross-modal feature fusion method based on attention mechanism according to claim 1, characterized in that, The step three is specifically: performing feature flattening processing on the third stage feature map to flatten the third stage feature map in the spatial dimension into a one-dimensional feature sequence to obtain a first feature sequence; performing K-means clustering operation on the first feature sequence to divide the first feature sequence into N clusters, and calculating cluster centers of each cluster respectively; each cluster center is recorded as a dynamic proxy label to obtain N dynamic proxy labels; inputting the first feature sequence and the N dynamic proxy labels into a proxy attention module as input data; the proxy attention module comprises a feature-to-proxy attention unit and a proxy-to-feature attention unit; in the feature-to-proxy attention unit, performing feature-to-proxy attention calculation on the first feature sequence and the N dynamic proxy labels to obtain intermediate proxy features based on the dynamic proxy labels; in the proxy-to-feature attention unit, performing proxy-to-feature attention calculation on the intermediate proxy features and the first feature sequence to obtain a first modality high-level feature sequence.

6. The cross-modal feature fusion method based on attention mechanism according to claim 1, characterized in that, The step four is specifically: inputting the first modality high-level feature sequence into a fourth stage of the improved SegNeXt network as input data; in the fourth stage, inputting the first modality high-level feature sequence into a convolution attention module to obtain a fourth stage convolution attention feature map; inputting the fourth stage convolution attention feature map into a double gating module to obtain a fourth stage feature map; flattening the fourth stage feature map in the spatial dimension into a one-dimensional feature sequence to obtain a second feature sequence; performing K-means clustering operation on the second feature sequence to divide the second feature sequence into N clusters, and calculating cluster centers of each cluster respectively, and obtaining N dynamic proxy labels by taking each cluster center as a dynamic proxy label; inputting the second feature sequence and the N dynamic proxy labels into a proxy attention module as input data to obtain an updated first modality high-level feature sequence.

7. The cross-modal feature fusion method based on attention mechanism according to claim 1, characterized in that, The step five is specifically: obtaining original data of a second modality, and inputting the original data of the second modality into a second modality encoding network as input data; performing second modality preprocessing operation on the original data of the second modality in the second modality encoding network to obtain second modality preprocessing data; The second modality pre-processing data is input into a second modality feature extraction sub-network, convolution processing, attention processing and down-sampling processing are sequentially performed in the second modality feature extraction sub-network, and a second modality intermediate feature sequence is obtained; The second modality intermediate feature sequence is subjected to feature transformation and dimension alignment processing, and the transformed second modality intermediate feature sequence is rearranged into a second modality feature sequence matched with the first modality high-level feature sequence in the feature dimension. 8.The method of claim 1, wherein, The step six is specifically: The updated first modality high-level feature sequence and the second modality feature sequence are input into a multi-scale agent diffusion module as input data; In the multi-scale agent diffusion module, multi-scale decomposition processing is performed on the updated first modality high-level feature sequence and the second modality feature sequence, and first modality sub-feature sequences of at least two scales and second modality sub-feature sequences of at least two scales are obtained; On each scale, the first modality sub-feature sequence of the same scale and the second modality sub-feature sequence of the same scale are respectively subjected to feature aggregation operations, and first modality aggregated features and second modality aggregated features are obtained; The first modality aggregated features and the second modality aggregated features are subjected to feature mapping processing, and the mapped features are combined to construct an agent representation of the current scale, generating an agent feature sequence of the current scale; On each scale, the first modality sub-feature sequence of the same scale, the second modality sub-feature sequence of the same scale and the agent feature sequence of the same scale are subjected to diffusion mapping processing. The agent feature sequence after diffusion mapping processing of each scale and the sub-feature sequence after diffusion mapping processing of each scale are subjected to scale fusion processing, and the fused features are reorganized in the scale dimension to obtain a cross-modality alignment feature sequence.

9. The cross-modal feature fusion method based on an attention mechanism according to claim 8, characterized in that, The diffusion mapping processing of the first modality sub-feature sequence of the same scale, the second modality sub-feature sequence of the same scale and the agent feature sequence of the same scale is specifically: The first modality sub-feature sequence of the same scale, the second modality sub-feature sequence of the same scale and the agent feature sequence of the same scale are mapped to the same dimension space; The first modality sub-feature sequence of the same scale, the second modality sub-feature sequence of the same scale and the agent feature sequence of the same scale after mapping are subjected to interactive diffusion propagation processing, and feature diffusion weights are calculated according to the correlation between the first modality sub-feature sequence of the same scale, the second modality sub-feature sequence of the same scale and the agent feature sequence of the same scale; And the feature information of the first modality sub-feature sequence of the same scale, the second modality sub-feature sequence of the same scale and the agent feature sequence of the same scale is bidirectionally diffused and propagated by using the feature diffusion weights; The first modality sub-feature sequence of the same scale, the second modality sub-feature sequence of the same scale and the agent feature sequence of the same scale after interactive diffusion propagation processing are subjected to iterative update operations, and an updated agent feature sequence of the current scale and an updated sub-feature sequence of the current scale are obtained.

10. The cross-modal feature fusion method based on attention mechanism according to claim 1, characterized in that, The step seven is specifically: The cross-modality alignment feature sequence is input into a cross-modality fusion module as input data, and feature aggregation operations and feature mapping processing are performed on the cross-modality alignment feature sequence in the cross-modality fusion module to obtain a cross-modality agent label sequence; The cross-modal alignment feature sequence and the cross-modal proxy label sequence are respectively subjected to query feature mapping, key feature mapping and value feature mapping in the cross-modal fusion module to obtain query features, key features and value features for attention calculation; Attention weights are calculated based on the query features and the key features, and weighted aggregation processing is performed on the value features according to the attention weights in the cross-modal fusion module to generate a cross-modal fusion feature sequence; The cross-modal fusion feature sequence is input as input data into a task decoding network, and the cross-modal fusion feature sequence is subjected to feature reconstruction processing and output mapping processing in the task decoding network to obtain a target task output result.

Images