An image segmentation method based on multi-dimensional interactive attention mechanism

Abstract

The present invention discloses an image segmentation method based on a multi-dimensional interactive attention mechanism, relates to the technical field of image segmentation, proposes a variable spatial attention method, applies deformable convolution to the spatial attention method, designs a multi-dimensional spatial attention method branch, performs three-dimensional transformation on the input features, obtains the dependency between three different dimensions, obtains multi-dimensional spatial attention weights, averages the three output feature maps to obtain a multi-dimensional fused spatial attention method output, then proposes a fused channel attention method branch, fuses the two branches in the channel attention method in advance, makes the channel attention method more lightweight, and is used to obtain the channel attention weights, further realizes the multi-dimensional interactive attention method, further fuses the multi-dimensional channel and spatial attention outputs, and takes the weighted average to obtain the final method output.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and specifically to an image segmentation method based on a multi-dimensional interactive attention mechanism. Background Art

[0002] The goal of image segmentation is to find objects of interest in a single image or video frame and label the pixel locations of the objects within the image. Typically, the segmentation result is represented by an image with the same length and width as the original image, with pixels of different colors representing different object categories.

[0003] There are currently some problems in the field of image segmentation:

[0004] 1. Difficulty handling fuzzy boundaries: Semantic segmentation models typically classify images at the pixel level, making it difficult to handle situations with fuzzy boundaries. For example, for a person and background in an image, the model may assign pixels at the fuzzy boundary to one of the two categories: person and background.

[0005] 2. High training data cost: Deep learning models require a large amount of training data, and semantic segmentation tasks require every pixel to be labeled, so the cost of labeled data is high. In addition, due to the complexity of semantic segmentation tasks, more labeled data is required to achieve good performance.

[0006] 3. Difficulty in handling category imbalance: In practical applications, the number of pixels in different categories may vary greatly. In this case, the model tends to predict the category with a larger number and ignore the category with a smaller number.

[0007] 4. Generally, semantic segmentation tasks involve many objects in an image, and the objects of interest are not necessarily salient objects in the image. For example, autonomous driving tasks require simultaneous detection of vehicles and pedestrians in an image, while the medical field may require simultaneous segmentation of lesions of widely varying sizes. Models often struggle to accurately handle these vastly different objects simultaneously. Furthermore, due to limitations in shooting angle and distance, objects of the same category often appear at different sizes in different images. These are the multi-scale issues in detection and segmentation tasks. Addressing this multi-scale issue is a key challenge in segmentation tasks.

[0008] Numerous attention methods have been proposed in the field of computer vision. These methods are primarily categorized as channel attention, spatial attention, and hybrid attention. Spatial attention methods tend to overlook channel information in feature maps, while channel attention methods tend to ignore dependencies in the spatial dimension. Therefore, hybrid attention methods attempt to comprehensively identify dependencies between channels and space. However, currently common hybrid attention mechanisms still fall short of achieving the desired effect. For example, the widely used attention mechanism (CBAM) simply concatenates the two methods, resulting in independent computations and an inability to effectively capture dependencies across different dimensions. Summary of the Invention

[0009] The purpose of the present invention is to provide an image segmentation method based on a multi-dimensional interactive attention mechanism to solve the problems raised in the above background technology.

[0010] In order to solve the above technical problems, the present invention provides the following technical solutions: an image segmentation method based on a multi-dimensional interactive attention mechanism, the method comprising:

[0011] Step S100: compressing the input image to obtain a descriptor of the input image, and performing operations on the descriptor to obtain a feature map of the input image;

[0012] Step S200: Perform dimensionality transformation on the input image, repeat the method in step S100 to obtain feature maps with different dimensional features, perform dimensionality transformation on the feature maps with different dimensional features so that the dimensional structure of the features is consistent with the input image, calculate the mean of the feature maps, and output a first feature map;

[0013] Step: S300: using a global average pooling method to obtain a first descriptor for the input image, and using a global maximum pooling method to obtain a second descriptor for the input image, and performing dimensionality compression on the two descriptors to obtain a first descriptor and a second descriptor after dimensionality reduction;

[0014] Step S400: The first descriptor after dimensionality reduction and the second descriptor after dimensionality reduction are activated by an activation function and then added together to obtain a third descriptor. A fully connected layer is used to map the compressed dimension of the third descriptor back to the original dimension. Finally, an activation function is used to activate it to obtain the channel attention mechanism weight.

[0015] Step: S500: multiply the first feature map by the channel attention weight to obtain the second feature map, and multiply the input image by the channel attention weight to obtain the third feature map;

[0016] Step S600: performing weighted averaging on the second feature map, the third feature map, and the input image to obtain a final output image.

[0017] Furthermore, step S100 includes:

[0018] Step S101: Input image X in , the dimension structure of the input image is: C×H×W, where C represents the number of input image channels, H represents the input image height, and W represents the input image width;

[0019] Step S102: compress the channels of the input image to obtain a single-channel descriptor, and then splice them in the channel dimension to obtain a dual-channel descriptor X double ;

[0020] Step S103: Fusing the dual-channel descriptors to obtain the signature X triple , for X triple Perform the operation to obtain the feature map X of the input image out1 , feature map X out1 The dimensional structure is: C×H×W.

[0021] Furthermore, step S102 includes:

[0022] Step S11: compress the channel dimension of the input image;

[0023] Step S12: Obtain two single-channel descriptors with a structure of 1×H×W through the average pooling function and the maximum pooling function respectively;

[0024] Step S13: Concatenate the two single-channel descriptors in the channel dimension and output a dual-channel descriptor X with a shape of 2×H×W double ;

[0025] Among them, X double =h1(AvgPool(X in ), MaxPool (X in )),

[0026] Among them, h1(A, B) represents the channel dimension splicing of A and B, AvgPool represents the average pooling function, and MaxPool represents the maximum pooling function.

[0027] Furthermore, step S103 includes:

[0028] Step S14: Use three convolution kernels with 2 input channels and 1 output channel to further concatenate the dual-channel descriptors and output the feature X triple ,

[0029] Among them, X triple =h2(f 3×3 (X double ), f 5×5 (Xdouble ), f 7×7 (X double )),

[0030] Among them, h2(A, B, C) represents the channel dimension splicing of A, B and C, f 3×3 , f 5×5 and f 7×7 Represents deformable convolution with kernel sizes of 3×3, 5×5, and 7×7 respectively;

[0031] Step S15: Use a 7×7 convolutional layer and a sigmoid function to obtain the final attention weight structure of 1×H×W. Multiply the input image and the attention weight in the spatial dimension to obtain the output feature map X with a structure of C×H×W. out1 ,

[0032] Among them, X out1 =X in *σ(g 7×7 (X triple )),

[0033] Among them, σ represents the sigmoid activation function, * represents the element multiplication operation, g 7×7 Represents a convolutional layer of size 7×7.

[0034] Step S100 designs an implementation scheme for a variable spatial attention method branch. The variable spatial attention method branch uses a more flexible sampling deformable convolution to fuse the features of the shape of 2×H×W output by the global maximum pooling and the global average pooling, further increasing the receptive field and improving the accuracy of the final extraction of the spatial attention weight, that is, strengthening the ability to capture the dependency of spatial dimensions.

[0035] Furthermore, step S200 includes:

[0036] Step S201: performing dimension conversion on the input image to obtain a feature map of the dimension-converted image;

[0037] Step S202: Dimensionally transform the feature map of the dimensionally transformed image, repeat the method in step S100, and calculate the features of the first dimension of each dimension-transformed image to obtain the feature map X out1 Feature maps with consistent dimensional structure;

[0038] The first dimension is the dimension parameter at the first position in the image dimension structure. For example, when the image dimension structure is C×H×W, the first dimension is C; when the image dimension structure is H×C×W, the first dimension is H; when the image dimension structure is W×C×H, the first dimension is W;

[0039] Step S203: Calculate the average value of all feature maps and output the first feature map Y1.

[0040] Where Y1=(X out1 +P1(X out2 )+P2(X out3 )) / 3,

[0041] Among them, X out2 Represents a feature map with a dimension structure of H×C×W, X out3 It represents a feature map with a dimensional structure of W×C×H, P1 represents a dimensional structure conversion operation that converts the H×C×W dimensional structure into C×H×W, and P2 represents a dimensional structure conversion operation that converts the W×C×H dimensional structure into C×H×W.

[0042] Step S200 designs an implementation scheme for a branch of a multidimensional spatial attention method, which only improves the ability to mine spatial dimension dependencies, but does not take into account the relationship between channel dimensions and spatial dimensions. Therefore, based on the variable spatial attention method, a branch of a multidimensional spatial attention method is designed to further mine the dependencies between multiple dimensions.

[0043] Furthermore, step S300 includes:

[0044] Step S301: applying a global average pooling method to an input image to obtain a first descriptor, and applying a global maximum pooling method to the input image to obtain a second descriptor, wherein the dimensional structure of the first descriptor is C×1×1, and the dimensional structure of the second descriptor is C×1×1, where C represents the number of channels of the input image;

[0045] Step S302: reducing the channel dimensions of the first descriptor and the second descriptor to obtain a first descriptor and a second descriptor after dimension reduction, respectively.

[0046] Step S300 designs an implementation scheme for fusing the branches of the channel attention method, reducing the number of fully connected layers by fusing the two branches in advance, that is, reducing the size of the channel attention method.

[0047] Furthermore, step S400 includes:

[0048] Step S401: adding the results of activating the first descriptor after dimensionality reduction and the second descriptor after dimensionality reduction through the ReLU activation function to obtain a third descriptor;

[0049] Step S402: using a fully connected layer to map the compressed channel dimension of the third descriptor back to the original dimension, where the number of channels in the original dimension is C;

[0050] Step S403: Activate the third descriptor mapped back to the original dimension through the sigmoid function to obtain the final channel attention mechanism weight with a shape of C×1×1.

[0051] Three fully connected layers are used in the fusion channel attention method branch. The first two fully connected layers are responsible for dimensionality reduction and dependency capture of features compressed by global average pooling and global maximum pooling. The third fully connected layer upgrades the channel dimension to the original dimension and is used to calculate the final channel attention weight. Compared with general methods, the fusion channel attention method branch in this paper is more lightweight.

[0052] Furthermore, step S500 includes:

[0053] Step S501: The channel attention weight corresponding to the first feature map is multiplied by the first feature map in the channel dimension to obtain the second feature map Y2, where Y2=W1⊙Y1, W1 represents the attention weight value corresponding to Y1, and ⊙ represents the channel dimension multiplication operation;

[0054] Step S502: Multiply the channel attention weight corresponding to the input image by the channel dimension of the input image to obtain the third feature map Y3, where Y3=W2⊙X in , W1 represents X in The corresponding attention weight value is obtained.

[0055] Furthermore, the weighted averaging method in step S600 includes:

[0056] X out =(αX in +βY2+γY3) / α+β+γ,

[0057] Among them, α, β and γ correspond to X in The coefficients of Y2 and Y3 represent X in , the contribution of Y2 and Y3 to the final output image, X out Represents the final output image.

[0058] Steps S500 and S600 design an implementation scheme of a multi-dimensional interactive attention method;

[0059] Based on the multi-dimensional spatial attention method branch and the fusion channel attention method branch, a multi-dimensional interactive attention method was constructed. On a macro level, the multi-dimensional interactive attention method takes into account the spatial information relationship and channel information relationship of the input features at the same time, and on a detailed level, it can interact with features of different dimensions, greatly improving the ability to mine the correlation between multi-dimensional information.

[0060] Compared with existing technologies, the present invention achieves the following beneficial effects: Starting from the attention mechanism, the present invention applies the attention mechanism to computer vision. First, the present invention proposes a Variable Spatial Attention Method (DSAM), which applies deformable convolution to the spatial attention method, improving the flexibility of feature point sampling and thus enhancing feature extraction capabilities. Next, based on DSAM, a Multidimensional Spatial Attention Method (MSAM) branch is designed. By performing a three-dimensional transformation on the input features, the dependencies between height and width, channel and height, and channel and width are obtained, resulting in multidimensional spatial attention weights. The three output feature maps are averaged to obtain the multidimensional fused spatial attention method output. Secondly, a Fused Channel Attention Method (FCAM) ​​branch is proposed, which pre-fuse the two different global pooling branches in the channel attention method, making the channel attention method more lightweight and used to obtain channel attention weights. Finally, based on MSAM and FCAM, a Multidimensional Interactive Attention Method (MIAM) is proposed. This further fuses the multidimensional channel and spatial attention outputs and takes a weighted average to obtain the final method output. The multi-dimensional interactive attention method created by the present invention can more fully capture the information interaction between different dimensions to make up for the shortcomings of the existing channel attention and spatial attention methods, and effectively improve the segmentation accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0062] Figure 1 This is a flowchart of an image segmentation method based on a multi-dimensional interactive attention mechanism in the present invention;

[0063] Figure 2 This is a schematic diagram of an implementation scheme of a variable spatial attention method for an image segmentation method based on a multi-dimensional interactive attention mechanism in the present invention;

[0064] Figure 3 This is a schematic diagram of an implementation scheme of a multi-dimensional spatial attention method branch of an image segmentation method based on a multi-dimensional interactive attention mechanism in the patent of this invention;

[0065] Figure 4 This is a schematic diagram of the design scheme of a branch of a fusion channel attention method for an image segmentation method based on a multi-dimensional interactive attention mechanism in the present invention;

[0066] Figure 5 This is a schematic diagram of an implementation scheme of a multi-dimensional interactive attention method for an image segmentation method based on a multi-dimensional interactive attention mechanism in the present invention;

[0067] Figure 6 This is a schematic diagram of the DeepLabv3 network structure of the patented image segmentation method based on a multi-dimensional interactive attention mechanism that integrates the multi-dimensional interactive attention method;

[0068] Figure 7 This is a schematic diagram of the U-Net network structure of an image segmentation method based on a multi-dimensional interactive attention mechanism in the present invention;

[0069] Figure 8 This is a schematic diagram of the Unet inference results of an image segmentation method based on a multi-dimensional interactive attention mechanism in the patent of this invention, which integrates a multi-dimensional interactive attention method. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] See also Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 、 Figure 7 and Figure 8 , the present invention provides a technical solution:

[0072] Step S100: compressing the input image to obtain a descriptor of the input image, and performing operations on the descriptor to obtain a feature map of the input image;

[0073] Wherein, step S100 includes:

[0074] Step S101: Input image X in The dimensional structure of the input image is: C×H×W, where C represents the number of input image channels, H represents the input image height, and W represents the input image width;

[0075] Step S102: compress the channels of the input image to obtain a single-channel descriptor, and then splice them in the channel dimension to obtain a dual-channel descriptor X double ;

[0076] Wherein, step S102 includes:

[0077] Step S11: compress the channel dimension of the input image;

[0078] Step S12: Obtain two single-channel descriptors with a structure of 1×H×W through the average pooling function and the maximum pooling function respectively;

[0079] Step S13: Concatenate the two single-channel descriptors in the channel dimension and output a dual-channel descriptor X with a shape of 2×H×W double ;

[0080] Among them, X double =h1(AvgPool(X in ), MaxPool (X in )),

[0081] Among them, h1(A, B) represents the channel dimension splicing of A and B, AvgPool represents the average pooling function, and MaxPool represents the maximum pooling function;

[0082] Step S103: Fusing the dual-channel descriptors to obtain the signature X triple , for X triple Perform the operation to obtain the feature map X of the input image out1 , the feature map X out1 The dimensional structure is: C×H×W;

[0083] Step S103 includes:

[0084] Step S14: Use three convolution kernels with 2 input channels and 1 output channel to further concatenate the dual-channel descriptors and output the feature X triple , where X triple =h2(f 3×3 (X double ), f 5×5 (X double ), f 7×7 (X double )),

[0085] Among them, h2(A, B, C) represents the channel dimension splicing of A, B and C, f 3×3 , f 5×5 and f 7×7 Represents deformable convolution with kernel sizes of 3×3, 5×5, and 7×7 respectively;

[0086] Step S15: Use a 7×7 convolutional layer and a sigmoid function to obtain the final attention weight structure of 1×H×W. Multiply the input image and the attention weight in the spatial dimension to obtain the output feature map X with a structure of C×H×W. out1 ,

[0087] Among them, X out1 =X in*σ(g 7×7 (X triple )),

[0088] Among them, σ represents the sigmoid activation function, * represents the element multiplication operation, g 7×7 Represents a convolutional layer of size 7×7.

[0089] Figure 2 This paper presents an implementation of a variable spatial attention method that extracts channel-dimensional features from the input image. The variable spatial attention method uses sampled deformable convolution to fuse the 2×H×W features output by global max pooling and global average pooling.

[0090] Step S200: Perform dimensionality transformation on the input image, repeat the method in step S100 to obtain feature maps with different dimensional features, perform dimensionality transformation on the feature maps with different dimensional features so that the dimensional structure of the features is consistent with the input image, calculate the mean of the feature maps, and output a first feature map;

[0091] Wherein, step S200 includes:

[0092] Step S201: performing dimension conversion on the input image to obtain a feature map of the dimension-converted image;

[0093] Step S202: Dimensionally transform the feature map of the dimensionally transformed image, repeat the method in step S100, and calculate the features of the first dimension of each dimension-transformed image to obtain the feature map X out1 Feature maps with consistent dimensional structure;

[0094] Step S203: Calculate the average value of all feature maps and output the first feature map Y1.

[0095] Where Y1=(X out1 +P1(X out2 )+P2(X out3 )) / 3,

[0096] Among them, X out2 Represents a feature map with a dimension structure of H×C×W, X out3 It represents a feature map with a dimensional structure of W×C×H, P1 represents a dimensional structure conversion operation that converts the H×C×W dimensional structure into C×H×W, and P2 represents a dimensional structure conversion operation that converts the W×C×H dimensional structure into C×H×W.

[0097] Figure 3It shows an implementation scheme of a branch of the multidimensional spatial attention method. The multidimensional spatial attention method branch can be generally divided into three branches in design. By transforming the dimension of the input feature map, the dependency relationships between height and width, channel and width, and channel and height are extracted respectively, and finally the three output features are fused to obtain the final output feature.

[0098] Step S300: applying a global average pooling method to the input image to obtain a first descriptor, applying a global maximum pooling method to the input image to obtain a second descriptor, and performing dimensionality compression on the two descriptors to obtain a first descriptor and a second descriptor after dimensionality reduction;

[0099] The steps of dimensionally compressing the two descriptors in step S300 include:

[0100] Step S301: A first descriptor is obtained by applying a global average pooling method to the input image, and a second descriptor is obtained by applying a global maximum pooling method to the input image. The dimensional structure of the first descriptor is C×1×1, and the dimensional structure of the second descriptor is C×1×1, where C represents the number of channels of the input image.

[0101] Step S302: reducing the channel dimensions of the first descriptor and the second descriptor to obtain a first descriptor and a second descriptor after dimension reduction, respectively.

[0102] Among them, the dimensional structure of the first descriptor after dimensionality reduction is C / r×1×1, the dimensional structure of the second descriptor after dimensionality reduction is C / r×1×1, and the dimensional structure of the third descriptor is C / r×1×1, where r represents the dimensionality reduction ratio.

[0103] Figure 4 It represents a design scheme for a branch of the fused channel attention method. In terms of the process, for an input feature map with a shape of C×H×W, the fused channel attention method branch first compresses the spatial dimension, and uses two global pooling operations in the spatial dimension to obtain two descriptors with a shape of C×1×1. The two descriptors are respectively reduced in the channel dimension by two fully connected layers with a dimensionality reduction ratio of r, that is, two descriptors with a shape of C / r×1×1 are obtained. Then, they are respectively activated by the Relu activation function and added together to fuse them into a descriptor with a shape of C / r×1×1. Then, a fully connected layer is used to map the dimension back to the original C, and then activated by the sigmoid function to obtain the final channel attention mechanism weight with a shape of C×1×1.

[0104] Step S400: The first descriptor after dimensionality reduction and the second descriptor after dimensionality reduction are activated by an activation function and then added together to obtain a third descriptor. A fully connected layer is used to map the compressed dimension of the third descriptor back to the original dimension. Finally, an activation function is used to activate it to obtain the channel attention mechanism weight.

[0105] The step of obtaining the channel attention mechanism weight in step S400 includes:

[0106] Step S401: adding the results of activating the first descriptor after dimensionality reduction and the second descriptor after dimensionality reduction through the ReLU activation function to obtain a third descriptor;

[0107] Step S402: Use a fully connected layer to map the compressed channel dimension of the third descriptor back to the original dimension, where the number of channels in the original dimension is C;

[0108] Step S403: Activate the third descriptor mapped back to the original dimension through the sigmoid function to obtain the final channel attention mechanism weight with a shape of C×1×1.

[0109] Step S500: multiplying the first feature map by the channel attention weight to obtain a second feature map, and multiplying the input image by the channel attention weight to obtain a third feature map;

[0110] Wherein, step S500 includes:

[0111] Step S501: The channel attention weight corresponding to the first feature map is multiplied by the first feature map in the channel dimension to obtain the second feature map Y2, where Y2=W1⊙Y1, W1 represents the attention weight value corresponding to Y1, and ⊙ represents the channel dimension multiplication operation;

[0112] Step S502: Multiply the channel attention weight corresponding to the input image by the channel dimension of the input image to obtain the third feature map Y3, where Y3=W2⊙X in , W1 represents X in The corresponding attention weight value is obtained.

[0113] Step S600: performing weighted averaging on the second feature map, the third feature map, and the input image to obtain a final output image.

[0114] The method for performing weighted averaging in step S600 includes:

[0115] X out =(αX in +βY2+γY3) / α+β+γ,

[0116] Among them, α, β and γ correspond to X inThe coefficients of Y2 and Y3 represent X in , the contribution of Y2 and Y3 to the final output image, X out Represents the final output image.

[0117] Figure 5 A multi-dimensional interactive attention method is proposed, which is based on the multi-dimensional spatial attention method branch and the fusion channel attention method branch.

[0118] Example 1: The multi-dimensional interactive attention method is applied to classic semantic segmentation models such as DeepLabv3, U-Net, and FCN. Experiments are conducted on PASCAL VOC2012, the most widely used dataset in the field of computer vision, to test the effectiveness of the method proposed in this chapter, and the method is verified on the medical impact dataset.

[0119] The experimental equipment is two Tesla T4 graphics cards, the batch size of each graphics card is 8, the total batch size is 16, the optimizer uses SGD, the learning rate is set to 0.005, the momentum is set to 0.9, the weight_decay is set to 0.0004, the epoch is set to 100, and the warmup_epochs is set to 1. The learning rate is updated once per step, and the learning rate update algorithm is: lr = lr × lr factor ,

[0120] where lr factor =(1-(step-warmup×num step ) / ((epochs-warmup)×num step )) 0.9 ,

[0121] where lr factor is the learning rate scaling factor, step represents the current number of steps, epochs represents the total number of training rounds, and num step Represents the total number of training steps. Since the essence of the semantic segmentation task is multi-classification of pixels, the cross entropy loss is used as the loss function for this experiment. The experiment is implemented based on the Pytorch deep learning framework, using Pytorch 1.13.0 and Python 3.8.

[0122] The experimental results on VOC2012 are shown in Table 1. After adding the method in this paper, the segmentation effect of the model is improved. Figure 6 Represents a DeepLabv3 network structure that integrates a multi-dimensional interactive attention method. Figure 7 It represents a U-Net network structure that integrates multi-dimensional interactive attention methods;

[0123] Table 1 Comparison of model effects after adding the multi-dimensional interactive attention method. The models marked with "*" are the models after adding the multi-dimensional interactive attention method:

[0124] Model Backbone mIoU (%) Params DeepLabv3 ResNet50 76.3 39.6M DeepLabv3* ResNet50 78.0 40.9M DeepLabv3 ResNet101 78.4 58.6M DeepLabv3* ResNet101 80.8 59.9M U-Net Vgg16 44.7 26.5M U-Net* Vgg16 45.5 27.8M

[0125] Table 1

[0126] Table 2 shows the comparison of the segmentation effects of the models on medical imaging datasets. Figure 8 Represents the Unet inference results of the fused multi-dimensional interactive attention method.

[0127] Table 2 shows the comparison of segmentation effects on the medical image dataset. The models marked with "*" are the models after adding the multi-dimensional interactive attention method:

[0128] Model Backbone mIoU (%) Params Unet U-Net Encoder 78.3 4.32M Unet* U-Net Encoder 81.0 5.61M Unet Vgg16 80.8 26.5M Unet* Vgg16 82.3 27.8M

[0129] Table 2

[0130] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0131] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An image segmentation method based on a multi-dimensional interactive attention mechanism, characterized by: The steps include: Step S100: compressing the input image to obtain a descriptor of the input image, and performing operations on the descriptor to obtain a feature map of the input image; Step S200: Perform dimensionality transformation on the input image, repeat the method in step S100 to obtain feature maps with different dimensional features, perform dimensionality transformation on the feature maps with different dimensional features so that the dimensional structure of the features is consistent with the input image, calculate the mean of the feature maps, and output a first feature map; Step S300: applying a global average pooling method to the input image to obtain a first descriptor, applying a global maximum pooling method to the input image to obtain a second descriptor, and performing dimensionality compression on the two descriptors to obtain a first descriptor and a second descriptor after dimensionality reduction; Step S400: The first descriptor after dimensionality reduction and the second descriptor after dimensionality reduction are activated by an activation function and then added together to obtain a third descriptor. A fully connected layer is used to map the compressed dimension of the third descriptor back to the original dimension. Finally, an activation function is used to activate it to obtain the channel attention mechanism weight. Step S500: multiplying the first feature map by the channel attention weight to obtain a second feature map, and multiplying the input image by the channel attention weight to obtain a third feature map; Step S600: performing weighted averaging on the second feature map, the third feature map, and the input image to obtain a final output image.

2. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 1, characterized in that: Step S100 includes: Step S101: Input image X in The dimensional structure of the input image is: C×H×W, where C represents the number of input image channels, H represents the input image height, and W represents the input image width; Step S102: compress the channels of the input image to obtain a single-channel descriptor, and then splice them in the channel dimension to obtain a dual-channel descriptor X double ; Step S103: Fusing the dual-channel descriptors to obtain the signature X triple , for X triple Perform the operation to obtain the feature map X of the input image out1 , the feature map X out1 The dimensional structure is: C×H×W.

3. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 2, characterized in that: Step S102 includes: Step S11: compress the channel dimension of the input image; Step S12: Obtain two single-channel descriptors with a structure of 1×H×W through the average pooling function and the maximum pooling function respectively; Step S13: Concatenate the two single-channel descriptors in the channel dimension and output a dual-channel descriptor X with a shape of 2×H×W double ; Among them, X double =h1(AvgPool(X in ), MaxPool (X in )), Among them, h1(A, B) represents the channel dimension splicing of A and B, AvgPool represents the average pooling function, and MaxPool represents the maximum pooling function.

4. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 3, characterized in that: Step S103 includes: Step S14: Use three convolution kernels with 2 input channels and 1 output channel to further concatenate the dual-channel descriptors and output the feature X triple , Among them, X triple =h2(f 3×3 (X double ), f 5×5 (X double ), f 7×7 (X double )), Among them, h2(A, B, C) represents the channel dimension splicing of A, B and C, f 3×3 , f 5×5 and f 7×7 Represents deformable convolution with kernel sizes of 3×3, 5×5, and 7×7 respectively; Step S15: Use a 7×7 convolutional layer and a sigmoid function to obtain the final attention weight structure of 1×H×W. Multiply the input image and the attention weight in the spatial dimension to obtain the output feature map X with a structure of C×H×W. out1 , Among them, X out1 =X in *σ(g 7×7 (X) triple )) Among them, σ represents the sigmoid activation function, * represents the element multiplication operation, g 7×7 Represents a convolutional layer of size 7×7.

5. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 2, characterized in that: Step S200 includes: Step S201: performing dimension conversion on the input image to obtain a feature map of the dimension-converted image; Step S202: Dimensionally transform the feature map of the dimensionally transformed image, repeat the method in step S100, and calculate the features of the first dimension of each dimension-transformed image to obtain the feature map X out1 A feature map with consistent dimensional structure, wherein the first dimension is the dimensional parameter located at the first position in the image dimensional structure; Step S203: Calculate the average value of all feature maps and output the first feature map Y1. Where Y1=(X out1 +P1(X out2 )+P2(X out3 )) / 3, Among them, X out2 Represents a feature map with a dimension structure of H×C×W, X out3 It represents a feature map with a dimensional structure of W×C×H, P1 represents a dimensional structure conversion operation that converts the H×C×W dimensional structure into C×H×W, and P2 represents a dimensional structure conversion operation that converts the W×C×H dimensional structure into C×H×W.

6. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 5, characterized in that: The steps of dimensionally compressing the two descriptors in step S300 include: Step S301: applying a global average pooling method to an input image to obtain a first descriptor, and applying a global maximum pooling method to the input image to obtain a second descriptor, wherein the dimensional structure of the first descriptor is C×1×1, and the dimensional structure of the second descriptor is C×1×1, where C represents the number of channels of the input image; Step S302: reducing the channel dimensions of the first descriptor and the second descriptor to obtain a first descriptor and a second descriptor after dimension reduction, respectively.

7. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 6, characterized in that: The dimensional structure of the first descriptor after dimensionality reduction is C / r×1×1, the dimensional structure of the second descriptor after dimensionality reduction is C / r×1×1, and the dimensional structure of the third descriptor is C / r×1×1, where r represents the dimensionality reduction ratio.

8. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 7, characterized in that: The step of obtaining the channel attention mechanism weight in step S400 includes: Step S401: adding the results of activating the first descriptor after dimensionality reduction and the second descriptor after dimensionality reduction through the ReLU activation function to obtain a third descriptor; Step S402: using a fully connected layer to map the compressed channel dimension of the third descriptor back to the original dimension, where the number of channels in the original dimension is C; Step S403: Activate the third descriptor mapped back to the original dimension through the sigmoid function to obtain the final channel attention mechanism weight with a shape of C×1×1.

9. The image segmentation method based on the multi-dimensional interactive attention mechanism according to claim 8, characterized in that: Step S500 includes: Step S501: The channel attention weight corresponding to the first feature map is multiplied by the first feature map in the channel dimension to obtain the second feature map Y2, where Y2=W1⊙Y1, W1 represents the attention weight value corresponding to Y1, and ⊙ represents the channel dimension multiplication operation; Step S502: Multiply the channel attention weight corresponding to the input image by the channel dimension of the input image to obtain the third feature map Y3, where Y3=W2⊙X in , W1 represents X in The corresponding attention weight value is obtained.

10. The image segmentation method based on multi-dimensional interactive attention mechanism according to claim 9, characterized in that: The method for performing weighted averaging in step S600 includes: X out =(αX in +βY2+γY3) / α+β+γ, Among them, α, β and γ correspond to X in The coefficients of Y2 and Y3 represent X in , the contribution of Y2 and Y3 to the final output image, X out Represents the final output image.

Images