A dynamic multi-scale convolution and cross-level attention feature fusion method and device

Abstract

The application discloses a dynamic multi-scale convolution and cross-level attention feature fusion method and device, and relates to the technical field of image semantic segmentation. The basic strategy of the method is to receive the feature map output by the decoder of the same level and the residual feature output by the encoder, and to perform double calibration through channel attention and spatial attention. The channel enhanced feature and the spatial enhanced feature are added to obtain preliminary fusion features. Then, the cross-layer features and the preliminary fusion features are spliced in the channel dimension, and a new feature map after dynamic fusion is output by means of dynamic multi-scale convolution. Finally, local and global feature extraction is performed on the new feature map, and a gating mechanism is used to finely screen and fuse the local and global features and the dynamic fusion features. In this way, the cross-level feature fusion can be realized by means of dynamic calculation and cross-level context as a link, the context modeling capability of the segmentation model is enhanced, and the image segmentation task capability is improved.

Description

Technical Field

[0001] This invention belongs to the field of image semantic segmentation technology, specifically relating to a method and apparatus for dynamic multi-scale convolution and cross-level attention feature fusion. Background Technology

[0002] Currently, feature fusion is a key step in improving the performance of segmentation models in semantic segmentation tasks of remote sensing images. In U-Nets (a widely used deep learning network structure for image segmentation, specifically composed of an encoder forming a contraction path and a decoder forming an expansion path, creating a U-shaped structure; where the contraction path extracts features through convolutional and pooling layers, and the expansion path restores image resolution and performs classification through transposed convolutional layers) used for image semantic segmentation, simple addition and concatenation operations are often employed to integrate multi-scale features. However, these methods cannot effectively adjust the feature contributions of different layers and lack sufficient support for cross-layer feature interactions. In addition, existing attention mechanisms typically compute channel attention and spatial attention independently, easily neglecting their joint optimization. Furthermore, the local receptive field of CNNs (Convolutional Neural Networks, a core model in deep learning that uses convolutional, pooling, and fully connected layers to achieve image recognition and visual task processing) limits the capture of global information. Additionally, the computational cost of the Transformer model architecture (a revolutionary neural network architecture proposed in 2017, whose core innovation is the self-attention mechanism, which completely replaces the traditional CNN structure) is relatively high.

[0003] Therefore, in view of the above situation, how to provide a dynamic multi-scale convolution and cross-level attention feature fusion scheme for image semantic segmentation, so as to realize cross-level feature fusion, enhance the context modeling ability of the segmentation model, and thus improve the ability of image segmentation tasks, is a topic that urgently needs to be studied by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for dynamic multi-scale convolution and cross-level attention feature fusion, in order to solve the problems of existing image semantic segmentation schemes, such as the inability to effectively adjust the feature contributions of different layers, insufficient interaction of cross-level features, easy neglect of joint optimization of channel attention and spatial attention, and / or limitations in capturing global information.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Firstly, a method for fusing dynamic multi-scale convolution and cross-level attention features is provided, including:

[0007] Receive main branch features from the decoder and residual features and cross-layer features from the encoder, wherein the decoder and the encoder are at the same level in the U-shaped network;

[0008] Based on the main branch features and the residual features, a dual attention mechanism including channel attention and spatial attention is applied to calibrate and obtain the channel attention calculation results and the spatial attention calculation results.

[0009] The channel attention calculation results and the spatial attention calculation results are merged to obtain a preliminary fused feature map;

[0010] The cross-layer features and the residual features are size aligned using bilinear interpolation to obtain aligned cross-layer features.

[0011] The preliminary fused feature map and the aligned cross-layer features are concatenated, and a new dynamically fused feature map is output using dynamic multi-scale convolution based on the concatenation result and the preliminary fused feature map.

[0012] The new feature map is subjected to local and global feature extraction processing to obtain local and global features;

[0013] The local and global features and the new feature map are subjected to weight-based control fusion processing using a gating mechanism to obtain the final fused features and output them.

[0014] Based on the above-mentioned invention, a novel scheme for dynamic multi-scale convolution and cross-level attention feature fusion for image semantic segmentation is provided. The basic strategy is to receive the feature map output by the decoder and the residual features output by the encoder at the same level, perform dual calibration through channel attention and spatial attention, and add the channel enhancement features and spatial enhancement features to obtain the preliminary fusion features. Then, cross-level features are introduced and concatenated with the preliminary fusion features in the channel dimension. Dynamic multi-scale convolution is used to output a new dynamically fused feature map. Finally, local and global features are extracted from the new feature map, and a gating mechanism is used to finely filter the local and global features and the dynamically fused features before fusion output. In this way, dynamic computation is used as a means and cross-level context is used as a link to realize the organic dynamic fusion and recombination of features—cross-level feature fusion, which enhances the context modeling ability of the segmentation model, thereby improving the image segmentation task capability and facilitating practical application and promotion.

[0015] In one possible design, the channel attention is calculated using the following formula:

[0016]

[0017] In the formula, This represents the channel attention calculation result. This indicates the main branch characteristic. This represents the residual characteristic. This represents the global average pooling function. This represents the max pooling function. Indicated based on having two layers The processing function of a multilayer perceptron with a convolutional structure. This represents the Sigmoid function. This indicates pixel-by-pixel multiplication.

[0018] In one possible design, the formula for calculating the spatial attention is as follows:

[0019]

[0020] In the formula, This represents the spatial attention calculation result. This indicates the main branch characteristic. This represents the residual characteristic. Indicates pixel-by-pixel multiplication. This represents the processing function that performs average pooling along the channel dimension. This represents the function that performs max pooling along the channel dimension. This represents the processing function that performs splicing along the channel dimension. This represents a processing function that performs multi-kernel convolution and has a kernel size of... , This represents the Sigmoid function.

[0021] In one possible design, the preliminary fused feature map and the aligned cross-layer features are concatenated, and based on the concatenation result and the preliminary fused feature map, a dynamically fused new feature map is output using dynamic multi-scale convolution, including:

[0022] Based on the splicing result and the preliminary fused feature map, the dynamic attention weights are calculated according to the following formula. :

[0023]

[0024] In the formula, This represents the preliminary fusion feature map. This represents the aligned cross-layer features. This represents the processing function that performs splicing along the channel dimension. This represents the processing function based on a multilayer perceptron with SE attention mechanism. Represents the normalization function;

[0025] The preliminary fused feature map is subjected to multi-kernel convolution according to the following formula, and the outputs of each branch convolution are stacked to obtain the feature stacking result. :

[0026]

[0027] In the formula, This represents a processing function that performs multi-kernel convolution and has a kernel size of... , This indicates a processing function that stacks data along the channel dimension.

[0028] Based on the dynamic attention weights and the feature stacking result The new feature map after dynamic fusion is output by weighted summation according to the following formula. :

[0029]

[0030] In the formula, Represents a positive integer.

[0031] In one possible design, the new feature map is subjected to local and global feature extraction processing to obtain local and global features, including:

[0032] The new feature map is processed using the following formula to extract local and global features to obtain enhanced local and global features. :

[0033]

[0034] In the formula, This represents the new feature map. This represents a function that performs depthwise separable convolutions with a kernel size of 3. This represents the global average pooling function. Indicates to proceed Convolution processing functions, This represents the Sigmoid function. This indicates pixel-by-pixel multiplication.

[0035] In one possible design, a gating mechanism is used to perform weight-based controlled fusion processing on the local and global features and the new feature map to obtain the final fused features, including:

[0036] The new feature map is processed according to the following formula. Convolution and Sigmoid operations are used to obtain pixel-level gated weight maps. :

[0037]

[0038] In the formula, This represents the new feature map. Indicates to proceed Convolution processing functions, Represents the Sigmoid function;

[0039] According to the pixel-level gating weight map The local and global features and the new feature map are subjected to weight-based controlled fusion processing according to the following formula to obtain the final fused features. :

[0040]

[0041] In the formula, This represents the local and global features. This indicates pixel-by-pixel multiplication.

[0042] Secondly, a dynamic multi-scale convolution and cross-level attention feature fusion device is provided, including a multi-feature receiving unit, a dual attention calibration unit, a preliminary feature fusion unit, a feature alignment processing unit, a dynamic multi-scale fusion unit, a feature extraction processing unit, and a control fusion processing unit.

[0043] The multi-feature receiving unit is used to receive main branch features from the decoder and residual features and cross-layer features from the encoder, wherein the decoder and the encoder are at the same level in the U-shaped network;

[0044] The dual attention calibration unit is communicatively connected to the multi-feature receiving unit and is used to calibrate and obtain the channel attention calculation result and the spatial attention calculation result by applying a dual attention mechanism that includes channel attention and spatial attention based on the main branch feature and the residual feature.

[0045] The preliminary feature fusion unit is communicatively connected to the dual attention calibration unit and is used to merge the channel attention calculation results and the spatial attention calculation results to obtain a preliminary fused feature map.

[0046] The feature alignment processing unit is communicatively connected to the multi-feature receiving unit and the preliminary feature fusion unit, and is used to apply bilinear interpolation to perform size alignment processing on the cross-layer feature and the residual feature to obtain the aligned cross-layer feature.

[0047] The dynamic multi-scale fusion unit is communicatively connected to the feature alignment processing unit, and is used to perform splicing processing on the preliminary fused feature map and the aligned cross-layer features, and output a new dynamically fused feature map based on the splicing result and the preliminary fused feature map by means of dynamic multi-scale convolution.

[0048] The feature extraction processing unit is communicatively connected to the dynamic multi-scale fusion unit and is used to perform local and global feature extraction processing on the new feature map to obtain local and global features.

[0049] The control fusion processing unit is communicatively connected to the feature extraction processing unit. It is used to perform weight-based control fusion processing on the local and global features and the new feature map using a gating mechanism to obtain the final fused features and output them.

[0050] Thirdly, the present invention provides a computer device comprising a storage module, a processing module, and a transceiver module connected in sequence for communication, wherein the storage module is used to store a computer program, the transceiver module is used to send and receive messages, and the processing module is used to read the computer program and execute the dynamic multi-scale convolution and cross-level attention feature fusion method as described in the first aspect or any possible design in the first aspect.

[0051] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the dynamic multi-scale convolution and cross-level attention feature fusion method as described in the first aspect or any possible design in the first aspect.

[0052] Fifthly, the present invention provides a computer program product, including a computer program or instructions, wherein when the computer program or instructions are executed by a computer, the method for dynamic multi-scale convolution and cross-level attention feature fusion as described in the first aspect or any possible design in the first aspect is implemented.

[0053] The beneficial effects of the above scheme are:

[0054] (1) This invention creatively provides a new scheme for dynamic multi-scale convolution and cross-level attention feature fusion for image semantic segmentation. The basic strategy is to receive the feature map output by the decoder and the residual feature output by the encoder at the same level, and perform dual calibration through channel attention and spatial attention. Then, the channel enhancement feature and the spatial enhancement feature are added to obtain the preliminary fusion feature. Then, cross-level features and the preliminary fusion feature are introduced and concatenated in the channel dimension. The new feature map after dynamic fusion is output by means of dynamic multi-scale convolution. Finally, local and global features are extracted from the new feature map. The local and global features and the dynamic fusion feature are finely screened and fused after using the gating mechanism. Thus, the organic dynamic fusion and recombination of features can be realized by using dynamic calculation as a means and cross-level context as a link. This enhances the context modeling ability of the segmentation model and improves the ability of image segmentation tasks.

[0055] (2) It can have good context modeling ability, that is, by refining features from the channel and spatial dimensions respectively through the dual attention mechanism, it can significantly improve the discrimination of small targets and complex boundaries; and by bridging local and global features, it can simultaneously preserve details (local convolution) and long-range dependencies (global pooling), which is better than a single dilated convolution or self-attention mechanism.

[0056] (3) It can achieve multi-scale feature adaptive fusion, that is, by innovatively introducing dynamic convolution and combining spatial attention map to enhance the focus on key regions, and by using cross-layer feature splicing to achieve direct interaction between high and low layer features, it can enhance the complementarity of details and semantics.

[0057] (4) It can realize dynamic fusion output of features, that is, it can generate learnable weights by means of gating mechanism, and can adaptively adjust the contribution ratio of different features, which is convenient for practical application and promotion. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the dynamic multi-scale convolution and cross-level attention feature fusion method provided in the embodiments of this application.

[0060] Figure 2 These are comparative example diagrams of three sets of image segmentation effects provided in the embodiments of this application.

[0061] Figure 3This is a schematic diagram of the structure of the dynamic multi-scale convolution and cross-level attention feature fusion device provided in the embodiments of this application.

[0062] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0064] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0065] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0066] Example

[0067] like Figure 1As shown, the dynamic multi-scale convolution and cross-level attention feature fusion method provided in the first aspect of this embodiment can be executed, but is not limited to, by computer devices with certain computing resources, such as servers, personal computers (PCs, referring to a type of multi-purpose computer of a size, price, and performance suitable for personal use; desktops, laptops, mini-laptops, tablets, and ultrabooks are all considered personal computers), smartphones, personal digital assistants (PDAs), or wearable devices. Figure 1 As shown, the dynamic multi-scale convolution and cross-level attention feature fusion method includes, but is not limited to, the following steps S1 to S7.

[0068] S1. Receive main branch features from the decoder and residual features and cross-layer features from the encoder, wherein the decoder and the encoder are at the same level in the U-shaped network.

[0069] In step S1, the U-shaped network refers to an existing network structure used for image semantic segmentation tasks, specifically, but not limited to, the UnetFormer network structure (a U-net-like hybrid architecture used for remote sensing semantic segmentation tasks, consisting of a CNN as the encoder and a Transformer architecture as the decoder). Therefore, the main branch features, the residual features, and the cross-layer features can be conventionally received and obtained. The main branch features can be denoted as... The residual characteristic can be denoted as: The cross-layer feature can be denoted as , This indicates the batch size (i.e., the number of samples processed in one forward propagation; for example, if 32 images are input at once, then...). ), This represents the number of channels in the feature image of the layer containing the decoder and encoder. This represents the vertical height of the feature image of the layer containing the decoder and encoder. This represents the horizontal width of the feature image of the layer containing the decoder and encoder. This represents the number of channels in the feature image across layers. This represents the vertical height of the feature image across layers. This represents the horizontal width of the feature image across layers.

[0070] S2. Based on the main branch features and the residual features, apply a dual attention mechanism that includes channel attention and spatial attention to calibrate and obtain the channel attention calculation results and spatial attention calculation results.

[0071] In step S2, the dual attention mechanism is used to perform dual attention calibration on the main branch features and the residual features: channel attention calibration and spatial attention calibration; thus, the channel attention calculation result is the channel enhancement feature, and the spatial attention calculation result is the spatial enhancement feature. Specifically, the formula for calculating the channel attention is as follows:

[0072]

[0073] In the formula, This represents the channel attention calculation result. This indicates the main branch characteristic. This represents the residual characteristic. This represents the global average pooling function. This represents the max pooling function. Indicated based on having two layers The processing function of a multilayer perceptron with a convolutional structure. This represents the Sigmoid function. This represents pixel-wise multiplication; the formula for calculating spatial attention is as follows:

[0074]

[0075] In the formula, This represents the spatial attention calculation result. This indicates the main branch characteristic. This represents the residual characteristic. Indicates pixel-by-pixel multiplication. This represents the processing function that performs average pooling along the channel dimension. This represents the function that performs max pooling along the channel dimension. This represents the processing function that performs splicing along the channel dimension. This represents a processing function that performs multi-kernel convolution and has a kernel size of... , This represents the Sigmoid function. Furthermore, the aforementioned global average pooling, max pooling, multilayer perceptron, average pooling along the channel dimension, max pooling along the channel dimension, concatenation along the channel dimension, multi-kernel convolution, and the Sigmoid function are all existing techniques and will not be elaborated upon here.

[0076] S3. The channel attention calculation results and the spatial attention calculation results are merged to obtain a preliminary fused feature map.

[0077] In step S3, the merging process can be performed according to the following formula:

[0078]

[0079] In the formula, This represents the preliminary fusion feature map.

[0080] S4. Apply bilinear interpolation to align the dimensions of the cross-layer features and the residual features to obtain aligned cross-layer features.

[0081] In step S4, the bilinear interpolation method is an interpolation algorithm used in numerical analysis, widely applied in signal processing, digital image and video processing, etc. Therefore, the size alignment process can be performed according to the following formula:

[0082]

[0083] In the formula, This represents the aligned cross-layer features. This represents the processing function for performing bilinear interpolation.

[0084] S5. The preliminary fused feature map and the aligned cross-layer features are concatenated, and a new dynamically fused feature map is output by means of dynamic multi-scale convolution based on the concatenation result and the preliminary fused feature map.

[0085] In step S5, the entire step is to achieve dynamic multi-scale fusion. Specifically, the preliminary fused feature map and the aligned cross-layer features are spliced ​​together, and based on the splicing result and the preliminary fused feature map, a new dynamically fused feature map is output using dynamic multi-scale convolution, including but not limited to the following steps S51 to S53.

[0086] S51. Based on the splicing result and the preliminary fusion feature map, the dynamic attention weight is calculated according to the following formula. :

[0087]

[0088] In the formula, This represents the preliminary fusion feature map. This represents the aligned cross-layer features. This represents the processing function that performs splicing along the channel dimension. This represents the processing function based on a multilayer perceptron with SE attention mechanism. This represents the normalization function.

[0089] In step S51, the dynamic attention weights .also, Specifically, this can be achieved through a small SE (Squeeze-and-Excitation)-MLP (Multilayer Perceptron) network.

[0090] S52. Apply multi-kernel convolution to the preliminary fused feature map according to the following formula, and stack the outputs of each branch convolution to obtain the feature stacking result. :

[0091]

[0092] In the formula, This represents a processing function that performs multi-kernel convolution and has a kernel size of... , This represents the processing function for stacking along the channel dimension.

[0093] In step S52, the feature stacking result .also, The method of adding a new dimension, specifically stacking along the channel dimension, is also an existing technique and will not be elaborated here.

[0094] S53. Based on the dynamic attention weights and the feature stacking result The new feature map after dynamic fusion is output by weighted summation according to the following formula. :

[0095]

[0096] In the formula, Represents a positive integer.

[0097] In step S53, due to the feature stacking result and the dynamic attention weights Therefore, in the above formula, " " represents all elements in the corresponding dimension, i.e. , Furthermore, the new feature map .

[0098] S6. Perform local and global feature extraction processing on the new feature map to obtain local and global features.

[0099] In step S6, the entire step aims to bridge local and global features. Specifically, local and global feature extraction processing is performed on the new feature map to obtain local and global features, including but not limited to: performing local and global feature extraction processing on the new feature map according to the following formula to obtain feature-enhanced local and global features. :

[0100]

[0101] In the formula, This represents the new feature map. This represents a function that performs depthwise separable convolutions with a kernel size of 3. This represents the global average pooling function. Indicates to proceed Convolution processing functions, This represents the Sigmoid function. This represents pixel-wise multiplication. Therefore, the new feature map can be extracted using depthwise separable convolution. The local features are obtained, and global channel weights are obtained using global attention. Then, the local features are multiplied by the global weights and then combined with the new feature map. The enhanced features are obtained by summing them. Furthermore, the aforementioned depthwise separable convolution and... Convolution is also a current technique, so it will not be elaborated on here.

[0102] S7. Using a gating mechanism, perform weight-based control fusion processing on the local and global features and the new feature map to obtain the final fused features and output them.

[0103] In step S7, the gating mechanism is used to control the fusion of the two branches to obtain the final output. Specifically, the gating mechanism is used to perform weight-based control fusion processing on the local and global features and the new feature map to obtain the final fused features, including but not limited to the following steps S71 to S72.

[0104] S71. Perform the following formula on the new feature map. Convolution and Sigmoid operations are used to obtain pixel-level gated weight maps. :

[0105]

[0106] In the formula, This represents the new feature map. Indicates to proceed Convolution processing functions, This represents the Sigmoid function.

[0107] S72. Based on the pixel-level gated weight map The local and global features and the new feature map are subjected to weight-based controlled fusion processing according to the following formula to obtain the final fused features. :

[0108]

[0109] In the formula, This represents the local and global features. This indicates pixel-by-pixel multiplication.

[0110] In step S72, the final fusion feature .

[0111] To verify the practical application effect of the aforementioned steps S1 to S7, this embodiment also integrates them into the UnetFormer network structure as a module (i.e., the functional module that implements the aforementioned steps S1 to S7, denoted as the CSDF module), and selects the internationally publicly available Vaihingen high-resolution remote sensing dataset for image semantic segmentation testing: the dataset is divided into 1400 images of size 512×512, and divided into a training set (1260 images) and a test set (140 images) at a ratio of 9:1. The core evaluation indicators are F1 score, mean intersection-over-union ratio (mIoU), and overall accuracy (OA), and the experimental results are shown in Table 1 below:

[0112] Table 1. Statistical Table of Experimental Results

[0113]

[0114] Based on the above experimental results, it can be seen that the technical solution provided in this embodiment can effectively achieve cross-level feature fusion through core technologies such as dual attention, multi-scale dynamic convolution, and gating mechanisms, thereby enhancing the context modeling capability of the segmentation model and, through methods such as... Figure 2 The comparison examples of the three sets of image segmentation results shown demonstrate that the technical solution provided in this embodiment can further improve the image segmentation task.

[0115] Therefore, based on the dynamic multi-scale convolution and cross-level attention feature fusion method described in steps S1 to S7 above, a new scheme for dynamic multi-scale convolution and cross-level attention feature fusion for image semantic segmentation is provided. The basic strategy is to receive the feature map output by the decoder and the residual features output by the encoder at the same level, perform dual calibration through channel attention and spatial attention, and add the channel enhancement features and spatial enhancement features to obtain the preliminary fusion features. Then, cross-level features and the preliminary fusion features are introduced and concatenated in the channel dimension. Dynamic multi-scale convolution is used to output a new dynamically fused feature map. Finally, local and global features are extracted from the new feature map, and a gating mechanism is used to finely filter the local and global features and the dynamically fused features before fusion output. In this way, dynamic computation can be used as a means and cross-level context as a link to realize the organic dynamic fusion and recombination of features—cross-level feature fusion, which enhances the context modeling ability of the segmentation model, thereby improving the ability of image segmentation tasks and facilitating practical application and promotion.

[0116] like Figure 3 As shown, the second aspect of this embodiment provides a virtual device for implementing the dynamic multi-scale convolution and cross-level attention feature fusion method described in the first aspect, including a multi-feature receiving unit, a dual attention calibration unit, a preliminary feature fusion unit, a feature alignment processing unit, a dynamic multi-scale fusion unit, a feature extraction processing unit, and a control fusion processing unit;

[0117] The multi-feature receiving unit is used to receive main branch features from the decoder and residual features and cross-layer features from the encoder, wherein the decoder and the encoder are at the same level in the U-shaped network;

[0118] The dual attention calibration unit is communicatively connected to the multi-feature receiving unit and is used to calibrate and obtain the channel attention calculation result and the spatial attention calculation result by applying a dual attention mechanism that includes channel attention and spatial attention based on the main branch feature and the residual feature.

[0119] The preliminary feature fusion unit is communicatively connected to the dual attention calibration unit and is used to merge the channel attention calculation results and the spatial attention calculation results to obtain a preliminary fused feature map.

[0120] The feature alignment processing unit is communicatively connected to the multi-feature receiving unit and the preliminary feature fusion unit, and is used to apply bilinear interpolation to perform size alignment processing on the cross-layer feature and the residual feature to obtain the aligned cross-layer feature.

[0121] The dynamic multi-scale fusion unit is communicatively connected to the feature alignment processing unit, and is used to perform splicing processing on the preliminary fused feature map and the aligned cross-layer features, and output a new dynamically fused feature map based on the splicing result and the preliminary fused feature map by means of dynamic multi-scale convolution.

[0122] The feature extraction processing unit is communicatively connected to the dynamic multi-scale fusion unit and is used to perform local and global feature extraction processing on the new feature map to obtain local and global features.

[0123] The control fusion processing unit is communicatively connected to the feature extraction processing unit. It is used to perform weight-based control fusion processing on the local and global features and the new feature map using a gating mechanism to obtain the final fused features and output them.

[0124] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be found in the dynamic multi-scale convolution and cross-level attention feature fusion method described in the first aspect, and will not be repeated here.

[0125] like Figure 4 As shown, the third aspect of this embodiment provides a computer device for executing the dynamic multi-scale convolution and cross-level attention feature fusion method as described in the first aspect. The device includes a storage module, a processing module, and a transceiver module connected in sequence. The storage module stores a computer program, the transceiver module sends and receives messages, and the processing module reads the computer program and executes the dynamic multi-scale convolution and cross-level attention feature fusion method as described in the first aspect. Specifically, the storage module may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processing module may, but is not limited to, use a microprocessor of the STM32F105 series. Furthermore, the computer device may also include, but is not limited to, a power supply module, a display screen, and other necessary components.

[0126] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the dynamic multi-scale convolution and cross-level attention feature fusion method described in the first aspect, and will not be repeated here.

[0127] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions comprising the dynamic multi-scale convolution and cross-level attention feature fusion method as described in the first aspect. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the dynamic multi-scale convolution and cross-level attention feature fusion method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0128] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be found in the dynamic multi-scale convolution and cross-level attention feature fusion method described in the first aspect, and will not be repeated here.

[0129] This fifth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the dynamic multi-scale convolution and cross-level attention feature fusion method as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0130] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fusing dynamic multi-scale convolution and cross-level attention features, characterized in that, include: The system receives main branch features from the decoder and residual and cross-layer features from the encoder, wherein the decoder and the encoder are at the same level in a U-shaped network, which is used for image semantic segmentation tasks and adopts the UnetFormer network structure. Based on the main branch features and the residual features, a dual attention mechanism including channel attention and spatial attention is applied to calibrate and obtain the channel attention calculation results and the spatial attention calculation results. The channel attention calculation results and the spatial attention calculation results are merged to obtain a preliminary fused feature map; The cross-layer features and the residual features are size aligned using bilinear interpolation to obtain aligned cross-layer features. The preliminary fused feature map and the aligned cross-layer features are concatenated, and a new dynamically fused feature map is output using dynamic multi-scale convolution based on the concatenation result and the preliminary fused feature map. The new feature map is subjected to local and global feature extraction processing to obtain local and global features. Specifically, this includes performing local and global feature extraction processing on the new feature map according to the following formula to obtain feature-enhanced local and global features. : In the formula, This represents the new feature map. This represents a function that performs depthwise separable convolutions with a kernel size of 3. This represents the global average pooling function. Indicates to proceed Convolution processing functions, This represents the Sigmoid function. Indicates pixel-by-pixel multiplication; The local and global features and the new feature map are fused using a gating mechanism based on weights to obtain the final fused features, which are then output to the UnetFormer network structure for the image semantic segmentation task.

2. The dynamic multi-scale convolution and cross-level attention feature fusion method according to claim 1, characterized in that, The formula for calculating the channel attention is as follows: In the formula, This represents the channel attention calculation result. This indicates the main branch characteristic. This represents the residual characteristic. This represents the global average pooling function. This represents the max pooling function. Indicated based on having two layers The processing function of a multilayer perceptron with a convolutional structure. This represents the Sigmoid function. This indicates pixel-by-pixel multiplication.

3. The dynamic multi-scale convolution and cross-level attention feature fusion method according to claim 1, characterized in that, The formula for calculating spatial attention is as follows: In the formula, This represents the spatial attention calculation result. This indicates the main branch characteristic. This represents the residual characteristic. Indicates pixel-by-pixel multiplication. This represents the processing function that performs average pooling along the channel dimension. This represents the function that performs max pooling along the channel dimension. This represents the processing function that performs splicing along the channel dimension. This represents a processing function that performs multi-kernel convolution and has a kernel size of... , This represents the Sigmoid function.

4. The dynamic multi-scale convolution and cross-level attention feature fusion method according to claim 1, characterized in that, The preliminary fused feature map and the aligned cross-layer features are concatenated, and based on the concatenation result and the preliminary fused feature map, a dynamically fused new feature map is output using dynamic multi-scale convolution, including: Based on the splicing result and the preliminary fused feature map, the dynamic attention weights are calculated according to the following formula. : In the formula, This represents the preliminary fusion feature map. This represents the aligned cross-layer features. This represents the processing function that performs splicing along the channel dimension. This represents the processing function based on a multilayer perceptron with SE attention mechanism. Represents the normalization function; The preliminary fused feature map is subjected to multi-kernel convolution according to the following formula, and the outputs of each branch convolution are stacked to obtain the feature stacking result. : In the formula, This represents a processing function that performs multi-kernel convolution and has a kernel size of... , This represents the processing function for stacking along the channel dimension; Based on the dynamic attention weights and the feature stacking result The new feature map after dynamic fusion is output by weighted summation according to the following formula. : In the formula, Represents a positive integer.

5. The dynamic multi-scale convolution and cross-level attention feature fusion method according to claim 1, characterized in that, The local and global features and the new feature map are subjected to weight-based controlled fusion processing using a gating mechanism to obtain the final fused features, including: The new feature map is processed according to the following formula. Convolution and Sigmoid operations are used to obtain pixel-level gated weight maps. : In the formula, This represents the new feature map. Indicates to proceed Convolution processing functions, Represents the Sigmoid function; According to the pixel-level gating weight map The local and global features and the new feature map are subjected to weight-based controlled fusion processing according to the following formula to obtain the final fused features. : In the formula, This represents the local and global features. This indicates pixel-by-pixel multiplication.

6. A dynamic multi-scale convolution and cross-level attention feature fusion device, characterized in that, It includes a multi-feature receiving unit, a dual attention calibration unit, a preliminary feature fusion unit, a feature alignment processing unit, a dynamic multi-scale fusion unit, a feature extraction processing unit, and a control fusion processing unit; The multi-feature receiving unit is used to receive main branch features from the decoder and residual features and cross-layer features from the encoder, wherein the decoder and the encoder are at the same level in the U-shaped network, and the U-shaped network is used for image semantic segmentation tasks and adopts the UnetFormer network structure. The dual attention calibration unit is communicatively connected to the multi-feature receiving unit and is used to calibrate the channel attention calculation result and the spatial attention calculation result by applying a dual attention mechanism that includes channel attention and spatial attention based on the main branch feature and the residual feature. The preliminary feature fusion unit is communicatively connected to the dual attention calibration unit and is used to merge the channel attention calculation results and the spatial attention calculation results to obtain a preliminary fused feature map. The feature alignment processing unit is communicatively connected to the multi-feature receiving unit and the preliminary feature fusion unit, and is used to perform size alignment processing on the cross-layer feature and the residual feature by applying bilinear interpolation to obtain the aligned cross-layer feature. The dynamic multi-scale fusion unit is communicatively connected to the feature alignment processing unit, and is used to perform splicing processing on the preliminary fused feature map and the aligned cross-layer features, and output a new dynamically fused feature map based on the splicing result and the preliminary fused feature map by means of dynamic multi-scale convolution. The feature extraction processing unit, communicatively connected to the dynamic multi-scale fusion unit, is used to perform local and global feature extraction processing on the new feature map to obtain local and global features. Specifically, this includes performing local and global feature extraction processing on the new feature map according to the following formula to obtain feature-enhanced local and global features. : In the formula, This represents the new feature map. This represents a function that performs depthwise separable convolutions with a kernel size of 3. This represents the global average pooling function. Indicates to proceed Convolution processing functions, This represents the Sigmoid function. Indicates pixel-by-pixel multiplication; The control fusion processing unit is communicatively connected to the feature extraction processing unit. It is used to perform weight-based control fusion processing on the local and global features and the new feature map using a gating mechanism to obtain the final fused features and output them to the UnetFormer network structure for the image semantic segmentation task.

7. A computer device, characterized in that, It includes a storage module, a processing module, and a transceiver module that are sequentially connected in communication. The storage module is used to store computer programs, the transceiver module is used to send and receive messages, and the processing module is used to read the computer programs and execute the dynamic multi-scale convolution and cross-level attention feature fusion method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores instructions that, when executed on a computer, perform the dynamic multi-scale convolution and cross-level attention feature fusion method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the dynamic multi-scale convolution and cross-level attention feature fusion method as described in any one of claims 1 to 5.

Images