A Deep Learning-Based Multi-Directional Dynamic Feature Fusion Method

Abstract

This invention proposes a deep learning-based multi-directional dynamic feature fusion method, comprising the following steps: Step 1: Prepare a remote sensing semantic segmentation dataset; Step 2: Construct a receptive field module, using convolutional kernels of different sizes to process features, allowing different kernels to focus on different feature subspaces, increasing feature diversity and better handling information and details at different scales; Step 3: Construct a cross-window context interaction module, consisting of window attention and cross-window attention. Window attention focuses on local features, while cross-window attention focuses on global features, compensating for the shortcomings of the Vision Transformer architecture in constructing cross-scale attention; Step 4: Construct a multi-directional dynamic feature fusion method, combining the receptive field module and the cross-window context interaction module. This effectively combines the advantages of convolutional neural networks and Transformers, effectively solving the problems of confusing similar features and segmenting small targets in remote sensing urban images.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and specifically to a multi-directional dynamic feature fusion method based on deep learning. Background Technology

[0002] With the continuous advancement of computer technology and satellite remote sensing technology, semantic segmentation of remote sensing images has become an important research direction in remote sensing image processing. By performing semantic segmentation on remote sensing images, each pixel in the image can be assigned to a specific land cover category, such as buildings, roads, vegetation, and water bodies, at the pixel level. This technology is widely used in disaster monitoring, land use planning, urban construction, and environmental monitoring.

[0003] Existing semantic segmentation methods are typically based on deep learning techniques, particularly the combination of convolutional neural networks (CNNs) and Transformer models. CNNs capture detailed features through local receptive fields, while Transformer models can model global feature correlations. These methods have made significant progress in the segmentation of large objects. However, existing hybrid models still face certain challenges in the recognition of small objects and the problem of cross-scale feature fusion.

[0004] While existing models combining convolutional neural networks and Transformers can effectively extract both local and global features, their different feature extraction mechanisms can lead to information loss or feature imbalance during feature fusion. Furthermore, existing methods typically employ serial or parallel structures when handling multi-scale features, often neglecting the mutual guidance between cross-scale features, resulting in suboptimal segmentation performance for small objects. The similarity between specific categories (such as buildings and roads, trees and vegetation) also increases the difficulty of segmentation, especially in complex remote sensing image scenes.

[0005] To improve the accuracy of semantic segmentation of remote sensing images, especially the ability to identify small objects in complex scenes, there is an urgent need for a technical solution that can effectively and dynamically supplement and fuse local and global features. Furthermore, existing methods do not adequately consider the contributions of features at different scales during feature fusion, which poses new requirements for improving the segmentation performance across scales.

[0006] Therefore, a multi-directional dynamic feature fusion method based on deep learning is proposed, which can effectively combine the advantages of convolutional neural networks and Transformers, and is the key to solving the current bottleneck of semantic segmentation technology for remote sensing images. Summary of the Invention

[0007] This invention provides a multi-directional dynamic feature fusion method based on deep learning to solve the problem of confusing similar ground features and segmenting small targets in remote sensing urban images.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A deep learning-based multi-directional dynamic feature fusion method includes the following steps:

[0010] S1. Prepare the remote sensing semantic segmentation dataset;

[0011] S2. Construct a receptive field module and use convolutional kernels of different sizes to process features, so that different convolutional kernels focus on different feature subspaces, increasing the diversity of features and better processing information and details at different scales.

[0012] S3. Construct a cross-window context interaction module, which consists of two parts: window attention and cross-window attention. Window attention focuses on local features, while cross-window attention focuses on global features, making up for the shortcomings of the Vision Transformer architecture in constructing cross-scale attention.

[0013] S4. Construct a multi-directional dynamic feature fusion method, combining the receptive field module and the cross-window context interaction module.

[0014] In S2 above, the following sub-steps are used when constructing the receptive field module, i.e., the RFB module:

[0015] Three parallel feature extraction branches are created, each using a different convolutional structure to process the input features;

[0016] The first branch, branch0, extracts local features. It first compresses the input feature map using a 1x1 convolution, then extracts local features using a 3x3 convolution with the kernel dilation rate set to 1, resulting in feature map X0.

[0017] The second branch, branch1, extracts features at the first scale. First, it compresses the input feature map through a 1x1 convolution, then performs a normal 3x3 convolution operation to preserve the extraction of local features, and finally uses a 3x3 dilated convolution with a dilation rate of 3 to further process the feature map, resulting in feature map X1.

[0018] The third branch, branch2, extracts features at the second scale, which is larger than the first scale. First, it performs channel compression on the input feature map using a 1x1 convolution, then performs a normal 5x5 convolution operation, and finally uses a 3x3 dilated convolution with a dilation rate of 5 to further process the feature map, further expanding the receptive field and capturing global information, resulting in feature map X2. The features extracted by the three branches are then concatenated along the channel dimension to obtain feature map X3.

[0019] Then, a 1x1 convolution is used to fuse the concatenated features, which is used to reduce the dimensionality of the output feature map X3 of the three branches to obtain feature map X4;

[0020] The input X is passed through a shortcut path, creating a tensor with the same dimensions as the main path output, resulting in the convolutional feature short.

[0021] The fused feature X4 and the feature short obtained through the shortcut path are added together to achieve residual connection, resulting in feature map X5;

[0022] Apply the ReLU activation function to return the fused feature map X6.

[0023] In S3 above, the following sub-steps are used when building the cross-window context interaction module, i.e., the IOWTB module:

[0024] The input feature map F0 is passed through a BN layer to obtain the standardized feature map F1;

[0025] Input F1 into the IOWTBAttation module to perform window partitioning and obtain feature map F2;

[0026] The output F2 of the attention mechanism is randomly dropped by drop_path to enhance the regularization effect of the model, thereby avoiding overfitting and obtaining feature map F3;

[0027] The attention output feature F3 of F0 is added to form a new feature map F4;

[0028] The feature map after residual connection is input into the BN layer for further normalization to obtain the normalized feature map F5.

[0029] Feature map F6 is obtained by nonlinearly expanding and transforming feature map F5 using MLP;

[0030] The regularization effect is further enhanced by using DropPath, resulting in feature map F7.

[0031] Add feature map F4 and feature map F7 together to form the final output, resulting in the final feature map F8.

[0032] In S3 above, the IOWTBAttation module is constructed using the following sub-steps:

[0033] The IOWTBAttation module consists of two parts: Window Attention (WA) and Cross-Window Attention (CWA). WA focuses on local information within a window, while CWA focuses on global information across windows, thus making up for the shortcomings of the Vision Transformer architecture in building cross-scale attention.

[0034] WA is a window-based multi-head attention branch that captures local contextual information. CWA operation can capture long-distance dependencies and obtain global attention features. The feature map F0 is rearranged and divided into blocks by branch1 and branch2. The ID sequence is divided by window segmentation operation, so that the features in each local window become an independent block, resulting in feature maps F1 and F2.

[0035] Feature map F1 calculates the query Q, key K, and value V features through the qkv convolutional layer to obtain Q1, K1, and V1, as shown in formula (1):

[0036] Q,K,V=X , , (1)

[0037] The qkv convolutional layer is a 1x1 convolutional layer that takes an input of dim channels and outputs 3 * dim channels.

[0038] Feature map F2 calculates query Q, key K and value V features through the qkv1 linear layer. The qkv1 linear layer is also used to generate qkv. This is the processing of local features. Linear layers are more suitable for processing small local features, while convolution is more suitable for processing spatial information. Another qkv is generated through formula (1) to obtain Q2, K2, V2.

[0039] Perform a matrix dot product operation on q1 and k1, calculate the similarity between the query and the key in each window, and obtain the attention score. Multiply the attention score by the scaling factor scale to prevent the attention value from being too large, and obtain the final score Dots0, as shown in formula (2):

[0040] Attention_Scores (2)

[0041] Where Q and K are the feature matrices for querying Q1 and key K1, and querying Q2 and key K2, respectively. This represents the transpose of the key matrix. It is the dimension of query and key;

[0042] Perform a matrix dot product operation on Q2 and K2, calculate the similarity between the query and the key in each window, obtain the attention score, multiply the attention score by the scaling factor scale to prevent the attention value from being too large, and use formula (2) to obtain the final score Dots2.

[0043] The attention score Dots0 is added to the relative position bias to correct the attention score within each window, resulting in Dots1;

[0044] Apply softmax to the attention score Dots1 and normalize it to obtain the attention weight attn0;

[0045] The attention weight attn0 is applied to the value V to obtain the weighted attention result attn1, as shown in formula (3);

[0046] +B)V;(3)

[0047] The attention score Dots2 is added to the relative position bias to correct the attention score within each window, resulting in Dots3;

[0048] Apply softmax to the attention score Dots3 and normalize it to obtain the attention weight attn2;

[0049] Apply the attention weight attn2 to the value V, and use formula (3) to obtain the weighted attention result attn3;

[0050] The attention results attn1 and attn3 are rearranged to the shape of the original input, the redundant padding is removed, and the original feature map size is restored. The global attention and local attention results are added together, and features at different scales are fused to obtain feature map F3.

[0051] The input feature map F3 is subjected to depthwise convolution, which processes each channel of the input feature map independently to capture spatial information. A 1x1 convolution operation is performed on all channels at each position, that is, a linear combination is performed in the channel dimension to mix information from different channels. Finally, the output is normalized to maintain training stability and obtain feature map F4.

[0052] In S4 above, a multi-directional dynamic feature fusion method is constructed using the following sub-steps:

[0053] The image dataset is parsed using the Swing Transformer Block 1 module to obtain the feature map res1;

[0054] The feature map res1 is input into the Swing Transformer Block 2 module for parsing to obtain the feature map res2;

[0055] The feature map res2 is input into the Swing Transformer Block 3 module for parsing to obtain the feature map res3;

[0056] The feature map res3 is input into the Swing Transformer Block 4 module for parsing to obtain the feature map res4;

[0057] First, ReLU activation is performed on res3_w1, then res3_w1 is normalized to generate a weight; then the weight is applied to res3 and res4 which has been upsampled and convolutionally processed, and finally res5 is obtained through further processing by the RFB module;

[0058] res2_w1 is activated and normalized using ReLU to generate weights; then the weights are applied to res2 and the upsampled res5, and processed by the RFB module to obtain res6;

[0059] Perform the same ReLU and normalization operations on res1_w1 to generate weights; then fuse res1 and the upsampled res6, and further extract features through the IOWTB module to obtain res7;

[0060] The weights are generated using res2_w2, and res2, res6 and res7 after downsampling are weighted and fused. The features are further extracted using the IOWTB module to obtain res8.

[0061] Weights are generated using res3_w2, and res3, res5 and the downsampled res8 are weighted and fused. Then, features are further extracted using the IOWTB module to obtain res9.

[0062] Weights are generated using res4_w2, and the downsampled res4 and res9 are weighted and fused together. The final res10 is obtained after processing by the IOWTB module.

[0063] Finally, the four feature maps res7, res8, res9, and res10 are fused to obtain res11;

[0064] The feature map res11 is processed by the segmentation head to obtain the final output.

[0065] This invention provides a multi-directional dynamic feature fusion method based on deep learning, which has the following beneficial effects:

[0066] 1. A receptive field module is introduced, using convolutional kernels of different sizes to process features. This allows different convolutional kernels to focus on different feature subspaces, increasing feature diversity and better handling of information and details at different scales. RFB is customized to enhance the model's feature extraction and generalization capabilities in local regions, which helps to achieve a more detailed understanding of complex spatial patterns.

[0067] 2. A cross-window context interaction module is introduced, consisting of two parts: window attention and cross-window attention. Window attention focuses on local features, while cross-window attention focuses on global features, which makes up for the shortcomings of the Vision Transformer architecture in building cross-scale attention. The cross-window context interaction module enhances the model's ability to interact with information across windows and model remote dependencies. This comprehensive feature processing framework is crucial for solving the complex dynamic problems of remote sensing image segmentation.

[0068] 3. A deep learning-based multi-directional dynamic feature fusion method is proposed, which dynamically merges the local attention provided by the receptive field module with the global attention facilitated by the cross-window context interaction module. This innovative fusion method leverages the complementary advantages of different receptive fields, enabling the network to pay more attention to discriminative features in similar categories. Attached Figure Description

[0069] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0070] Figure 1 This is a flowchart of the present invention;

[0071] Figure 2 This is a schematic diagram of the receptive field RFB module;

[0072] Figure 3 This is a structural diagram of the IOWTB cross-window context interaction module;

[0073] Figure 4 yes Figure 2 A structural diagram of the IOWTBAttation module in the middle;

[0074] Figure 5 This is a schematic diagram of the feature fusion method proposed in this invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the following will describe the specific technical solutions of this invention systematically and completely in conjunction with the accompanying drawings provided by this invention. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0076] Example 1:

[0077] like Figure 1 As shown, a deep learning-based multi-directional dynamic feature fusion method includes the following steps:

[0078] Step 1: Prepare the remote sensing semantic segmentation dataset;

[0079] Step 2: Construct the receptive field module and use convolutional kernels of different sizes to process features. This allows different convolutional kernels to focus on different feature subspaces, increasing feature diversity and better handling of information and details at different scales.

[0080] Step 3: Construct a cross-window context interaction module, which consists of two parts: window attention and cross-window attention. Window attention focuses on local features, while cross-window attention focuses on global features, thus making up for the shortcomings of the Vision Transformer architecture in constructing cross-scale attention.

[0081] Step 4: Construct a multi-directional dynamic feature fusion method, combining the receptive field module and the cross-window context interaction module.

[0082] like Figure 2 As shown, in step 2, when constructing the receptive field module, i.e., the RFB module, the following sub-steps are used:

[0083] Three parallel feature extraction branches are created, each using a different convolutional structure to process the input features;

[0084] The first branch, branch0, extracts local features. It first compresses the input feature map using a 1x1 convolution, then extracts local features using a 3x3 convolution with the kernel dilation rate set to 1, resulting in feature map X0.

[0085] The second branch, branch1, extracts features at a larger scale. First, it compresses the input feature map through a 1x1 convolution, then performs a normal 3x3 convolution operation to preserve the extraction of local features, and finally uses a 3x3 dilated convolution with a dilation rate of 3 to further process the feature map, resulting in feature map X1.

[0086] The third branch, branch2, extracts features at a larger scale. First, it compresses the input feature map through a 1x1 convolution, then performs a normal 5x5 convolution operation, and finally uses a 3x3 dilated convolution with a dilation rate of 5 to further process the feature map, further expand the receptive field, capture global information, and obtain feature map X2.

[0087] The features extracted from the three branches are concatenated along the channel dimension to obtain feature map X3;

[0088] Then, a 1x1 convolution is used to fuse the concatenated features, which is used to reduce the dimensionality of the output feature map X3 of the three branches to obtain feature map X4;

[0089] The input X is passed through a shortcut path, creating a tensor with the same dimensions as the main path output, resulting in the convolutional feature short.

[0090] The fused feature X4 and the feature short obtained through the shortcut path are added together to achieve residual connection, resulting in feature map X5;

[0091] Apply the ReLU activation function to return the fused feature map X6.

[0092] like Figure 2 As shown, in step 3, when building the cross-window context interaction module, i.e., the IOWTB module, the following sub-steps are used:

[0093] The input feature map F0 is passed through a BN layer to obtain the standardized feature map F1;

[0094] Input F1 into the IOWTBAttation module to perform window partitioning and obtain feature map F2;

[0095] The output F2 of the attention mechanism is randomly dropped by drop_path to enhance the regularization effect of the model, thereby avoiding overfitting and obtaining feature map F3;

[0096] The attention output feature F3 of F0 is added to form a new feature map F4;

[0097] The feature map after residual connection is input into the BN layer for further normalization to obtain the normalized feature map F5.

[0098] Feature map F6 is obtained by nonlinearly expanding and transforming feature map F5 using MLP;

[0099] The regularization effect is further enhanced by using DropPath, resulting in feature map F7.

[0100] Add feature map F4 and feature map F7 together to form the final output, and obtain the final feature map F8;

[0101] like Figure 3 As shown, in step 3, the construction of the IOWTBAttation module involves the following sub-steps:

[0102] The OWTBAttation module consists of two parts: Window Attention (WA) and Cross-Window Attention (CWA). WA focuses on local information within a window, while CWA focuses on global information across windows, thus making up for the shortcomings of the Vision Transformer architecture in building cross-scale attention.

[0103] WA is a window-based multi-head attention branch that captures local contextual information. CWA operation can capture long-distance dependencies and obtain global attention features. The feature map F0 is rearranged and divided into blocks by branch1 and branch2. The ID sequence is divided by window segmentation operation, so that the features in each local window become an independent block, resulting in feature maps F1 and F2.

[0104] Feature map F1 calculates the query Q, key K, and value V features through the qkv convolutional layer to obtain Q1, K1, and V1, as shown in formula (1):

[0105] Q,K,V=X , , (1)

[0106] The qkv convolutional layer is a 1x1 convolutional layer that takes an input of dim channels and outputs 3 * dim channels.

[0107] Feature map F2 calculates query Q, key K and value V features through the qkv1 linear layer. The qkv1 linear layer is also used to generate qkv. This is the processing of local features. Linear layers are more suitable for processing small local features, while convolution is more suitable for processing spatial information. Another qkv is generated through formula (1) to obtain Q2, K2, V2.

[0108] Perform a matrix dot product operation on q1 and k1, calculate the similarity between the query and the key in each window, and obtain the attention score. Multiply the attention score by the scaling factor scale to prevent the attention value from being too large, and obtain the final score Dots0, as shown in formula (2):

[0109] Attention_Scores (2)

[0110] Where Q and K are the feature matrices for querying Q1 and key K1, and querying Q2 and key K2, respectively. This represents the transpose of the key matrix. It is the dimension of query and key;

[0111] Perform a matrix dot product operation on Q2 and K2, calculate the similarity between the query and the key in each window, obtain the attention score, multiply the attention score by the scaling factor scale to prevent the attention value from being too large, and use formula (2) to obtain the final score Dots2.

[0112] The attention score Dots0 is added to the relative position bias to correct the attention score within each window, resulting in Dots1;

[0113] Apply softmax to the attention score Dots1 and normalize it to obtain the attention weight attn0;

[0114] The attention weight attn0 is applied to the value V to obtain the weighted attention result attn1, as shown in formula (3);

[0115] +B)V;(3)

[0116] The attention score Dots2 is added to the relative position bias to correct the attention score within each window, resulting in Dots3;

[0117] Apply softmax to the attention score Dots3 and normalize it to obtain the attention weight attn2;

[0118] Apply the attention weight attn2 to the value V, and use formula (3) to obtain the weighted attention result attn3;

[0119] The attention results attn1 and attn3 are rearranged to the shape of the original input, the redundant padding is removed, and the original feature map size is restored. The global attention and local attention results are added together, and features at different scales are fused to obtain feature map F3.

[0120] The input feature map F3 undergoes depthwise convolution, processing each channel of the input feature map independently to capture spatial information. A 1x1 convolution operation is performed on all channels at each location, i.e., a linear combination is performed along the channel dimension, mixing information from different channels. Finally, the output is normalized to maintain training stability, resulting in feature map F4.

[0121] In step 4, a multi-directional dynamic feature fusion method is constructed, using the following sub-steps:

[0122] The image dataset is parsed using the Swing Transformer Block 1 module to obtain the feature map res1;

[0123] The feature map res1 is input into the Swing Transformer Block 2 module for parsing to obtain the feature map res2;

[0124] The feature map res2 is input into the Swing Transformer Block 3 module for parsing to obtain the feature map res3;

[0125] The feature map res3 is input into the Swing Transformer Block 4 module for parsing to obtain the feature map res4;

[0126] First, ReLU activation is performed on res3_w1, then res3_w1 is normalized to generate a weight. Next, the weight is applied to res3 and res4 (which has undergone upsampling and convolution), and finally further processed by the RFB module to obtain res5.

[0127] res2_w1 is activated by ReLU and normalized to generate weights. These weights are then applied to res2 and the upsampled res5, and processed by the RFB module to obtain res6.

[0128] The same ReLU and normalization operations are performed on res1_w1 to generate weights. Then, res1 and the upsampled res6 are fused, and features are further extracted using the IOWTB module to obtain res7.

[0129] The weights are generated using res2_w2, and res2, res6 and res7 after downsampling are weighted and fused. The features are further extracted using the IOWTB module to obtain res8.

[0130] Weights are generated using res3_w2, and res3, res5 and the downsampled res8 are weighted and fused. Then, features are further extracted using the IOWTB module to obtain res9.

[0131] Weights are generated using res4_w2, and the downsampled res4 and res9t are weighted and fused together. The final res10 is obtained by processing the IOWTB module.

[0132] Finally, the four feature maps res7, res8, res9, and res10 are fused to obtain res11;

[0133] The feature map res11 is processed by the segmentation head to obtain the final output.

[0134] This invention not only solves the common problems of inter-class similarity and small object segmentation in the field of semantic segmentation of remote sensing images, but also designs a multi-directional dynamic feature fusion method based on deep learning. Through this method, we can effectively improve the model's ability to model short-distance and long-distance interactions, thereby extracting highly discriminative features necessary to distinguish closely related classes and identify small objects.

[0135] Example:

[0136] 1) Parameter settings

[0137] This invention implemented all experiments using PyTorch on a single NVIDIA RTX3090 GPU, with AdamW as the optimizer and a learning rate of 0.0006. We used SoftCrossEntropyLoss and DiceLoss as the joint loss function and the inverse propagation exponent to measure the difference between the 2D segmentation map and the obtained ground truth. For each method, we used F1, mIOU, and OA as evaluation metrics.

[0138] Precision: (4)

[0139] Recall: (5)

[0140] F1 value: F1 = (6)

[0141] Mean Intersection over Union (mIOU): mIOU = (7)

[0142] Overall accuracy (OA): (8)

[0143] in, , , and These represent the proportions of positive samples correctly predicted as positive, positive samples incorrectly predicted as negative, negative samples correctly predicted as negative, and negative samples incorrectly predicted as positive for a specific category. Overall accuracy (OA) is the overall accuracy calculated for all categories.

[0144] 2) Experimental Results

[0145] To verify the effectiveness of this solution, further explanations and elaborations will be provided below based on experimental data.

[0146] Extensive ablation experiments were conducted on the ISPRS Vaihingen and ISPRS Potsdam datasets to validate the effectiveness and robustness of our method. The same test-time augmentation strategy and auxiliary loss were used in all ablation studies to ensure fair comparison. We conducted three sets of experiments: 1) Base represents a network using Swin-s as the encoder and directly upsampling features. 2) Base+RFB+RFB represents a network using RFB in both paths of the decoder. 3) Base+RFB+IOWTB represents a network where the bottom-up path of the decoder is replaced with an IOWTB-based network.

[0147] Table 1

[0148]

[0149] The ablation study results are shown in Table 1. Compared to the base model, Base+RFB+RFB improved the F1 score by 1.88%, OA (overall accuracy) by 0.27%, and mIoU (mean intersection-over-union ratio) by 2.52% on the Vaihingen dataset. On the Potsdam dataset, the average F1 score improved by 0.77%, OA by 0.1%, and mIoU by 1.35%. These improvements indicate that the segmentation capability is enhanced by using a multi-directional feature interaction method in the decoder. Compared to Base+RFB+RFB, Base+RFB+IOWTB improved the F1 score by 1.71%, OA by 0.7%, and mIoU by 2.73% on the Vaihingen dataset. On the Potsdam dataset, the average F1 score, OA, and mIoU improved by 1.3%, 1.02%, and 2.18%, respectively. This demonstrates that the interaction fusion method between CNN and Transformer significantly improves the performance, highlighting the model's advantages in complementary modeling of short and long distances.

Claims

1. A multi-directional dynamic feature fusion method based on deep learning, characterized in that, Includes the following steps: S1. Prepare the remote sensing semantic segmentation dataset; S2. Construct the receptive field module, i.e., the RFB module, which uses convolutional kernels of different sizes to process features, so that different convolutional kernels focus on different feature subspaces; S3. Construct a cross-window context interaction module, namely the IOWTB module, which consists of two parts: window attention and cross-window attention. Window attention focuses on local features, while cross-window attention focuses on global features. S4. Construct a multi-directional dynamic feature fusion method, combining the receptive field module and the cross-window context interaction module; The following sub-steps are used: The image dataset is parsed using the Swing Transformer Block 1 module to obtain the feature map res1; The feature map res1 is input into the Swing Transformer Block 2 module for parsing to obtain the feature map res2; The feature map res2 is input into the Swing Transformer Block 3 module for parsing to obtain the feature map res3; The feature map res3 is input into the Swing Transformer Block 4 module for parsing to obtain the feature map res4; First, ReLU activation is performed on res3_w1, then res3_w1 is normalized to generate a weight; then the weight is applied to res3 and res4 which has been upsampled and convolutionally processed, and finally res5 is obtained through further processing by the RFB module; res2_w1 is activated and normalized using ReLU to generate weights; then the weights are applied to res2 and the upsampled res5, and processed by the RFB module to obtain res6; Perform the same ReLU and normalization operations on res1_w1 to generate weights; then fuse res1 and the upsampled res6, and further extract features through the IOWTB module to obtain res7; The weights are generated using res2_w2, and res2, res6 and res7 after downsampling are weighted and fused. The features are further extracted using the IOWTB module to obtain res8. Weights are generated using res3_w2, and res3, res5 and the downsampled res8 are weighted and fused. Then, features are further extracted using the IOWTB module to obtain res9. Weights are generated using res4_w2, and the downsampled res4 and res9 are weighted and fused together. The final res10 is obtained after processing by the IOWTB module. Finally, the four feature maps res7, res8, res9, and res10 are fused to obtain res11; The feature map res11 is processed by the segmentation head to obtain the final output.

2. The multi-directional dynamic feature fusion method based on deep learning as described in claim 1, characterized in that, In S2, the following sub-steps are used when constructing the receptive field module: Three parallel feature extraction branches are created, each using a different convolutional structure to process the input features; The first branch, branch0, extracts local features. It first compresses the input feature map using a 1x1 convolution, then extracts local features using a 3x3 convolution with the kernel dilation rate set to 1, resulting in feature map X0. The second branch, branch1, extracts features at the first scale. First, it compresses the input feature map through a 1x1 convolution, then performs a normal 3x3 convolution operation to preserve the extraction of local features, and finally uses a 3x3 dilated convolution with a dilation rate of 3 to further process the feature map, resulting in feature map X1. The third branch, branch2, extracts features at the second scale, which is larger than the first scale. First, the input feature map is compressed by a 1x1 convolution, then by a normal 5x5 convolution, and finally by a 3x3 dilated convolution with a dilation rate of 5 to further process the feature map, further expand the receptive field, capture global information, and obtain feature map X2. The features extracted from the three branches are concatenated along the channel dimension to obtain feature map X3; Then, a 1x1 convolution is used to fuse the concatenated features, which is used to reduce the dimensionality of the output feature map X3 of the three branches to obtain feature map X4; The input X is passed through a shortcut path, creating a tensor with the same dimensions as the main path output, resulting in the convolutional feature short. The fused feature X4 and the feature short obtained through the shortcut path are added together to achieve residual connection, resulting in feature map X5; Apply the ReLU activation function to return the fused feature map X6.

3. The multi-directional dynamic feature fusion method based on deep learning according to claim 2, characterized in that, In S3, the following sub-steps are used when constructing the cross-window context interaction module: The input feature map F0 is passed through a BN layer to obtain the standardized feature map F1; Input F1 into the IOWTBAttation module to perform window partitioning and obtain feature map F2; The output F2 of the attention mechanism is randomly dropped by drop_path to enhance the regularization effect of the model, thereby avoiding overfitting and obtaining feature map F3; The attention output feature F3 of F0 is added to form a new feature map F4; The feature map after residual connection is input into the BN layer for further normalization to obtain the normalized feature map F5. Feature map F6 is obtained by nonlinearly expanding and transforming feature map F5 using MLP; The regularization effect is further enhanced by using DropPath, resulting in feature map F7. Add feature map F4 and feature map F7 together to form the final output, resulting in the final feature map F8.

4. The multi-directional dynamic feature fusion method based on deep learning according to claim 3, characterized in that, In S3, the IOWTBAttation module is constructed using the following sub-steps: The IOWTBAttation module consists of two parts: Window Attention (WA) and Cross-Window Attention (CWA). WA focuses on local information within a window, while CWA focuses on global information across windows. WA is a window-based multi-head attention branch that captures local contextual information. CWA operation can capture long-distance dependencies and obtain global attention features. The feature map F0 is rearranged and divided into blocks by branch1 and branch2. The ID sequence is divided by window segmentation operation, so that the features in each local window become an independent block, resulting in feature maps F1 and F2. Feature map F1 calculates the query Q, key K, and value V features through the qkv convolutional layer to obtain Q1, K1, and V1, as shown in formula (1): Q,K,V=X , , ;(1) The qkv convolutional layer is a 1x1 convolutional layer that takes an input of dim channels and outputs 3 * dim channels. Feature map F2 calculates query Q, key K and value V features through the qkv1 linear layer. The qkv1 linear layer is also used to generate qkv. This is the processing of local features. Linear layers are more suitable for processing small local features, while convolution is more suitable for processing spatial information. Another qkv is generated through formula (1) to obtain Q2, K2, V2. Perform a matrix dot product operation on q1 and k1, calculate the similarity between the query and the key in each window, and obtain the attention score. Multiply the attention score by the scaling factor scale to prevent the attention value from being too large, and obtain the final score Dots0, as shown in formula (2): Attention_Scores ;(2) Where Q and K are the feature matrices for querying Q1 and key K1, and querying Q2 and key K2, respectively. This represents the transpose of the key matrix. It is the dimension of query and key; Perform a matrix dot product operation on Q2 and K2, calculate the similarity between the query and the key in each window, obtain the attention score, multiply the attention score by the scaling factor scale to prevent the attention value from being too large, and use formula (2) to obtain the final score Dots2. The attention score Dots0 is added to the relative position bias to correct the attention score within each window, resulting in Dots1; Apply softmax to the attention score Dots1 and normalize it to obtain the attention weight attn0; The attention weight attn0 is applied to the value V to obtain the weighted attention result attn1, as shown in formula (3); +B)V;(3) The attention score Dots2 is added to the relative position bias to correct the attention score within each window, resulting in Dots3; Apply softmax to the attention score Dots3 and normalize it to obtain the attention weight attn2; Apply the attention weight attn2 to the value V, and use formula (3) to obtain the weighted attention result attn3; The attention results attn1 and attn3 are rearranged to the shape of the original input, the redundant padding is removed, and the original feature map size is restored. The global attention and local attention results are added together, and features at different scales are fused to obtain feature map F3. The input feature map F3 is subjected to depthwise convolution, which processes each channel of the input feature map independently to capture spatial information. A 1x1 convolution operation is performed on all channels at each position, that is, a linear combination is performed in the channel dimension to mix information from different channels. Finally, the output is normalized to maintain training stability and obtain feature map F4.

Images