Double-path medical image segmentation method and device based on multistage comprehensive attention

Abstract

The invention provides a double-path medical image segmentation method and device based on multistage comprehensive attention, and relates to the technical field of medical images. The method comprises the following steps: acquiring to-be-segmented medical image data, and inputting the to-be-segmented medical image data into a constructed comprehensive attention converter to obtain a medical image segmentation result; the comprehensive attention converter comprises a dual-path encoder, a first mixed attention converter module and a decoder; the encoder comprises a plurality of layers; the decoder comprises a multi-scale feature fusion decoder and a multi-scale aggregation module; the multi-scale feature fusion decoder comprises a channel-based efficient feature fusion module and a comprehensive attention module; the channel-based efficient feature fusion module comprises a multi-channel connectivity extraction module and a channel-level feature fusion module; the comprehensive attention module comprises an enhanced converter module, a channel attention module, a space attention module and a convolution block; the multi-scale aggregation module includes a second hybrid attention converter module and a multi-layer prediction link. And the feature extraction performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a dual-path medical image segmentation method and device based on multi-level comprehensive attention. Background Art

[0002] Automatic medical image segmentation has become a hot topic, which not only improves the quality of medical services, but also provides strong support for patients' health. Accurate and reliable medical image segmentation can significantly help doctors make accurate diagnoses, which has far-reaching significance for the early diagnosis and rehabilitation treatment of patients.

[0003] CNN (Convolutional Neural Networks) has shown powerful capabilities in various image-related applications. However, CNN models have a common weakness: due to the limitations of convolution operations, they cannot effectively capture long-range relationships and global context in images. In order to obtain a good receptive field, CNN models need to stack many layers, which may lead to problems such as gradient explosion and overfitting, preventing them from achieving the desired accuracy goals.

[0004] Recent studies have successfully applied the transformer model to computer vision, solving the problem of small receptive field caused by convolutional limitations. Some studies have combined the U-shaped structure, which has been proven effective in semantic segmentation, with the transformer model. The transformer model's ability to capture long-range relationships solves this problem well, but it often ignores the importance of local relationships, which are also crucial to achieve accurate segmentation, and previous models often ignore the importance of features with rich multi-scale information from encoders and decoders.

[0005] In addition, our experiments on the impact of skip connections concluded that the performance of the model is largely related to the settings of the skip connections. Missing or too many connections to a specific layer may lead to performance differences, especially when dealing with different datasets. Therefore, a lot of experiments must be performed to find the most appropriate connection method, which is very time-consuming. Summary of the invention

[0006] In order to solve the technical problem of how to simultaneously capture long-range relationships and local context features in medical image segmentation feature extraction, and to automatically find the most appropriate skip link method, the embodiment of the present invention provides a dual-path medical image segmentation method and device based on multi-level comprehensive attention. The technical solution is as follows:

[0007] On the one hand, a dual-path medical image segmentation method based on multi-level comprehensive attention is provided, the method is implemented by a medical image segmentation device, and the method includes:

[0008] S1. Obtain medical image data to be segmented.

[0009] S2. Input the medical image data into the constructed comprehensive attention converter.

[0010] S3. Obtain the medical image segmentation result based on the medical image data and the comprehensive attention converter.

[0011] Among them, the comprehensive attention converter includes: a dual-path encoder, a first hybrid attention converter module and a decoder.

[0012] The dual-path encoder includes: a first-layer dual-path encoder, a second-layer dual-path encoder, a third-layer dual-path encoder and a fourth-layer dual-path encoder.

[0013] The decoder includes: a multi-scale feature fusion decoder and a multi-scale aggregation module.

[0014] The multi-scale feature fusion decoder includes: a channel-based efficient feature fusion module and a comprehensive attention module.

[0015] The channel-based efficient feature fusion module includes: a multi-path connectivity extraction module and a channel-level feature fusion module.

[0016] The comprehensive attention module includes: enhanced converter module, channel attention module, spatial attention module and convolution block.

[0017] The multi-scale aggregation module includes: a second hybrid attention converter module and a multi-layer prediction chain.

[0018] Optionally, obtaining a medical image segmentation result according to the medical image data and the comprehensive attention converter in S3 includes:

[0019] S31. After block embedding, the medical image data is input into the first-layer dual-path encoder to obtain the output features of the first-layer dual-path encoder, and the features are input into the second-layer dual-path encoder to obtain the output features of the second-layer dual-path encoder, and then input into the third-layer dual-path encoder to obtain the output features of the third-layer dual-path encoder, and finally input into the fourth-layer dual-path encoder to obtain the output features of the fourth-layer dual-path encoder.

[0020] S32. Input the output features of the first layer dual-path encoder, the output features of the second layer dual-path encoder, the output features of the third layer dual-path encoder, and the output features of the fourth layer dual-path encoder into the first hybrid attention converter module respectively to obtain the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module, and the fourth layer output of the first hybrid attention converter module.

[0021] S33. Input the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module, and the fourth layer output of the first hybrid attention converter module into the decoder to obtain the medical image segmentation result.

[0022] Optionally, the dual-path encoder includes at least: a convolution-based feature encoding path, an attention-based feature encoding path, a feature fusion module, and adaptive pooling.

[0023] The convolution-based feature encoding path uses adaptive pooling and depth-separable convolution in parallel to learn high-frequency signals, as shown below:

[0024] AdaptivePool(X)=att n ·X (1)

[0025] Y h1 =FC(AdaptivePool(X h1 )) (2)

[0026] Y h2 =DSConc(FC(X h2 )) (3)

[0027] In the formula, AdaptivePool represents adaptive pooling, X represents the input feature, and att n represents the attention score of the neighbors, X h1 represents the part of the high-frequency signal used for adaptive pooling, X h2 represents the part of the high-frequency signal used for depthwise separable convolution, FC represents the linear layer, DSConc represents the depthwise separable convolution including depthwise convolution and pointwise convolution, and Y h1 and Y h2 Represent the outputs of adaptive pooling and depth-wise separable convolution, respectively.

[0028] The attention-based feature encoding path uses dual self-attention to learn low-frequency signals, that is, dual attention based on local windows and sliding windows, as shown below:

[0029] X1=W-MSA(LN(AdaptivePool(X l )))+AdaptivePool(X l ) (4)

[0030] X2=MLP(LN(X1))+X1 (5)

[0031] X3=SW-MSA(LN(X2))+X2 (6)

[0032] Y l =MLP(LN(X3))+X3 (7)

[0033] Where, X l represents low-frequency signal, LN represents layer normalization, W-MSA represents local window self-attention, SW-MSA represents sliding window self-attention, MLP represents multi-layer perceptron, X i (i=1,2,3) represents the output of each stage. The whole process uses a residual structure to improve performance. l represents the output of the attention-based feature encoding pathway.

[0034] The feature fusion module fuses the features output by the convolution-based feature encoding path and the attention-based feature encoding path in the channel dimension, and uses depthwise separable convolution to exchange information between blocks channel by channel and point by point, as shown in the following formula:

[0035] Y C =Concat(Y l ,Y h1 ,Y h2 ) (8)

[0036] Y=FC(Y c +DSConv(Y c )) (9)

[0037] Where Y l ,Y h1 ,Y h2 They represent the output of the attention-based feature encoding path and the convolution-based feature encoding path, Concat represents the merging of channel dimensions, DSConv represents depth-wise separable convolution, FC represents the linear layer, and Y c represents the merged features, and Y is the output.

[0038] The first hybrid attention converter module includes at least: a channel-level interleaved attention module, a self-attention module and a feedforward neural network; the channel-level interleaved attention module is introduced before the self-attention module to first capture long-distance features at the channel level, and then on this basis, perceive the key local features at the sequence level through the self-attention mechanism to realize automatic learning of the connection method of jump links.

[0039] The channel-based multi-head attention calculation formula of the first hybrid attention converter module is shown in equation (10):

[0040]

[0041] In the formula, Q, K, and V represent query, key, and value respectively. K=X Σ×W K , V = X Σ ×W V ,in represents the data of the input of the first hybrid attention converter module after block embedding, B represents batch, H represents height, W represents width, C represents the number of channels, and X represents Σ Expressed as Feature merging at the channel level, represents the weight matrix, where R represents a set of real numbers, represents the number of channels of the four skip link inputs, C Σ represents the sum of the number of channels of the four input features, T represents the transpose of the matrix, and d represents the number of blocks.

[0042] Optionally, the channel-based efficient feature fusion module includes: a multi-path connectivity extraction module, a first channel-level feature fusion module, a second channel-level feature fusion module, and a third channel-level feature fusion module.

[0043] The comprehensive attention module includes: a first comprehensive attention module, a second comprehensive attention module, a third comprehensive attention module and a fourth comprehensive attention module.

[0044] In S33, the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module, and the fourth layer output of the first hybrid attention converter module are input into the decoder to obtain a medical image segmentation result, including:

[0045] S331. Input the fourth layer output of the first hybrid attention converter module into the multi-path connectivity extraction module to obtain the output of the multi-path connectivity extraction module, and input the output of the multi-path feature extraction module into the first comprehensive attention module to obtain the output of the first comprehensive attention module.

[0046] S332. Input the third layer output of the first hybrid attention converter module and the output of the first comprehensive attention module into the first channel-level feature fusion module to obtain the output of the first channel-level feature fusion module, and input the output of the first channel-level feature fusion module into the second comprehensive attention module to obtain the output of the second comprehensive attention module.

[0047] S333. Input the second layer output of the first hybrid attention converter module and the output of the second comprehensive attention module into the second channel-level feature fusion module to obtain the output of the second channel-level feature fusion module, and input the output of the second channel-level feature fusion module into the third comprehensive attention module to obtain the output of the third comprehensive attention module.

[0048] S334. Input the first layer output of the first hybrid attention converter module and the output of the third comprehensive attention module into the third channel-level feature fusion module to obtain the output of the third channel-level feature fusion module, and input the third channel-level feature fusion module into the fourth comprehensive attention module to obtain the output of the fourth comprehensive attention module.

[0049] S335. Input the output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module and the output of the fourth comprehensive attention module into a multi-scale aggregation module to obtain a medical image segmentation result.

[0050] Optionally, the multi-path connectivity extraction module captures the directional information of the channel dimension in the deep network, represents the output of the fourth layer of the hybrid attention converter module as X, and then obtains the connection mask through a dual-path method of global maximum pooling and global average pooling, and then splits X at the channel level, and extracts the directional information in a multi-path manner, as shown in the following formula:

[0051] X f =δ(Conv(GAP(Conv(X)))+Conv(GMP(Conv(X)))) (11)

[0052] Y=X fi ×(SA(X i )+CA(X i ))+X i (12)

[0053] In the formula, GAP and GMP represent global average pooling and global maximum pooling, X f represents the connection mask, δ represents the activation function, X fi represents the link mask based on channel segmentation, X i represents the input features based on channel segmentation, CA and SA represent the channel attention module and the spatial attention module.

[0054] The channel-level feature fusion module transfers and fuses directional information. Each module uses a dual-path approach of global average pooling and global maximum pooling to perform channel-level fusion, as shown in the following equations (13)-(16):

[0055]

[0056] GMP(X)=MAX(X(i,j))(i∈{1,2,3…H},j∈{1,2,3…W}) (14)

[0057] y1=δ(L1(GAP(O i ))+L2(GAP(D i )))+δ(L3(GMP(Oi ))+L4(GMP(D i ))) (15)

[0058]

[0059] In the formula, GAP represents global average pooling, GMP represents global maximum pooling, X represents input features, H represents height, W represents width, X(i,j) represents features at height i and width j, y1 represents the area of ​​interest in the feature, δ(·) represents the ReLU activation function, and L i (i∈1,2,3,4) represents the linear layer, O i represents the output layer i of the first hybrid attention transformer module, D i represents the features of the i-th layer of the decoder, Represents the feature weighted by δ(y).

[0060] Optionally, the comprehensive attention module includes an enhanced converter module, a channel attention module, a spatial attention module, and a convolution block, so as to retain local features while learning long-distance dependencies.

[0061] The enhanced converter module at least includes: a self-attention layer and a feedforward neural network layer; the enhanced part is located in the feedforward neural network, which uses a dynamic convolution layer and layer normalization to a three-layer residual layer normalization to effectively improve the representation ability of local features, as shown in the following equations (17) and (18):

[0062]

[0063] out1=FC(GELU(y)+x in1 ) (18)

[0064] In the formula, y2 represents the output of dynamic convolution, x in1 represents the output of self-attention, FC represents the fully connected layer, IdynamicConv represents the dynamic convolution, represents layer normalization, GELU represents the activation function, and out1 represents the output of the feedforward neural network.

[0065] The channel attention module is shown in formula (19):

[0066]

[0067] In the formula, x in2 represents the output of the feedforward neural network layer, GAP represents global average pooling, GMP represents global maximum pooling, conv represents 1×1 convolution, and δ represents the ReLu activation function.

[0068] The spatial attention module is shown in equation (20):

[0069]

[0070] In the formula, x in3 represents the output of the channel attention module, C a represents the average value of the feature in the channel dimension, C m It represents the maximum value of the feature in the channel dimension, conv represents 7×7 convolution, and δ represents the ReLU activation function.

[0071] The output of the comprehensive attention module is shown in equation (21):

[0072] out2=SA(CA(FFN(SelfAttention(x input )))) (twenty one)

[0073] Where out2 represents the output of the comprehensive attention module, SA represents the spatial attention module, CA represents the channel attention module, FFN represents the feedforward neural network, SelfAttention represents the self-attention, and x input Represents the input features of the comprehensive attention module.

[0074] Optionally, in S335, the output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module, and the output of the fourth comprehensive attention module are input into the multi-scale aggregation module to obtain a medical image segmentation result, including:

[0075] The output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module and the output of the fourth comprehensive attention module are respectively input into the second hybrid attention converter module to obtain the first layer output of the second hybrid attention converter module, the second layer output of the second hybrid attention converter module, the third layer output of the second hybrid attention converter module and the fourth layer output of the second hybrid attention converter module.

[0076] The first layer output of the second hybrid attention converter module, the second layer output of the second hybrid attention converter module, the third layer output of the second hybrid attention converter module, and the fourth layer output of the second hybrid attention converter module are respectively input into the multi-layer prediction chain to obtain the medical image segmentation result, as shown in the following equations (22)(23):

[0077] p1,p2,p3,p4=MATrans(x1,x2,x3,x4) (22)

[0078] output=w×p1+x×p2+y×p3+z×p4 (23)

[0079] In the formula, x i (i∈(1,2,3,4)) represents the output of the comprehensive attention module, p i (i∈(1,2,3,4)) represents the predicted values ​​of the four prediction links, w,x,y,z represent the weights of each predicted value, and output represents the final prediction result.

[0080] On the other hand, a dual-path medical image segmentation device based on multi-level comprehensive attention is provided, and the device is applied to a dual-path medical image segmentation method based on multi-level comprehensive attention, and the device includes:

[0081] The acquisition module is used to acquire the medical image data to be segmented.

[0082] Input module, used to input medical image data into the constructed comprehensive attention transformer.

[0083] The output module is used to obtain the medical image segmentation result according to the medical image data and the comprehensive attention converter.

[0084] Among them, the comprehensive attention converter includes: a dual-path encoder, a first hybrid attention converter module and a decoder.

[0085] The dual-path encoder includes: a first-layer dual-path encoder, a second-layer dual-path encoder, a third-layer dual-path encoder and a fourth-layer dual-path encoder.

[0086] The decoder includes: a multi-scale feature fusion decoder and a multi-scale aggregation module.

[0087] The multi-scale feature fusion decoder includes: a channel-based efficient feature fusion module and a comprehensive attention module.

[0088] The channel-based efficient feature fusion module includes: a multi-path connectivity extraction module and a channel-level feature fusion module.

[0089] The comprehensive attention module includes: enhanced converter module, channel attention module, spatial attention module and convolution block.

[0090] The multi-scale aggregation module includes: a second hybrid attention converter module and a multi-layer prediction chain.

[0091] On the other hand, a medical image segmentation device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned dual-path medical image segmentation methods based on multi-level comprehensive attention is implemented.

[0092] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned dual-path medical image segmentation methods based on multi-level comprehensive attention.

[0093] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0094] In the present invention, a new method is proposed to synthesize attention converters, which adopts a dual-path approach in the encoding layer, respectively utilizing the advantages of convolution and converters to extract high-frequency signals and low-frequency signals in features. A cascade approach is adopted in the decoder. Different from using attention gates in the decoder to obtain important areas of the image, the present invention adopts an enhanced converter module, which improves the converter's ability to obtain local features by adding dynamic convolution layers, as well as two classic and effective convolutional attention methods based on space and channels. These methods retain long-distance relationships while extracting local features, thereby improving the feature extraction performance of global context and local texture in the decoder. In order to efficiently capture and fuse directional information in deep networks, an efficient feature fusion module based on channels is adopted to extract, transfer and fuse this information. At the same time, the present invention introduces a hybrid attention converter module between the encoder and the corresponding decoder to set jump connections more efficiently. The hybrid attention converter module adds channel-level multi-head cross attention before multi-head self-attention, thereby effectively obtaining the relationship between channel dimensions with rich multi-scale features, and then captures the relationship between different sequences through self-attention. Before the final prediction, the present invention adopts hierarchical prediction links. For the features from the decoder, the hybrid attention transformer module is used again to extract features with rich multi-scale features, so that the final prediction result contains both semantic information from deep layers and low-level structural features from shallow layers. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0096] Figure 1 It is a flow chart of a dual-path medical image segmentation method based on multi-level comprehensive attention provided by an embodiment of the present invention;

[0097] Figure 2 is a framework diagram of a comprehensive attention converter provided by an embodiment of the present invention;

[0098] Figure 3is a schematic diagram of adaptive pooling provided by an embodiment of the present invention;

[0099] Figure 4 is a schematic diagram of a dual-path encoder provided by an embodiment of the present invention;

[0100] Figure 5 is a schematic diagram of block-by-block self-attention and channel-by-channel cross-attention provided by an embodiment of the present invention;

[0101] Figure 6 is a structural diagram of a hybrid attention converter module provided by an embodiment of the present invention;

[0102] Figure 7 is a structural diagram of a multi-path connectivity extraction module provided by an embodiment of the present invention;

[0103] Figure 8 is a structural diagram of a channel-level feature fusion module provided in an embodiment of the present invention;

[0104] Fig. 9 is a structural diagram of a comprehensive attention module provided by an embodiment of the present invention;

[0105] Fig.10 is a structural diagram of a multi-scale aggregation module provided by an embodiment of the present invention;

[0106] Fig.11 It is a block diagram of a dual-path medical image segmentation device based on multi-level comprehensive attention provided by an embodiment of the present invention;

[0107] Fig.12 It is a structural schematic diagram of a medical image segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0108] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0109] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0110] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0111] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0112] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0113] The embodiment of the present invention provides a dual-path medical image segmentation method based on multi-level comprehensive attention, which can be implemented by a medical image segmentation device, which can be a terminal or a server. Figure 1 The flowchart of the dual-path medical image segmentation method based on multi-level comprehensive attention is shown in FIG. The processing flow of the method may include the following steps:

[0114] S1. Obtain medical image data to be segmented.

[0115] S2. Input the medical image data into the constructed comprehensive attention converter.

[0116] S3. Obtain the medical image segmentation result based on the medical image data and the comprehensive attention converter.

[0117] Figure 2 The structure of the comprehensive attention converter is presented, including: a dual-path encoder, a first hybrid attention converter module, and a decoder.

[0118] The dual-path encoder includes: a first-layer dual-path encoder, a second-layer dual-path encoder, a third-layer dual-path encoder and a fourth-layer dual-path encoder.

[0119] The decoder includes: a multi-scale feature fusion decoder and a multi-scale aggregation module.

[0120] The multi-scale feature fusion decoder includes: a channel-based efficient feature fusion module and a comprehensive attention module.

[0121] The channel-based efficient feature fusion module includes: a multi-path connectivity extraction module and a channel-level feature fusion module.

[0122] The comprehensive attention module includes: enhanced converter module, channel attention module, spatial attention module and convolution block.

[0123] The multi-scale aggregation module includes: a second hybrid attention converter module and a multi-layer prediction chain.

[0124] The present invention proposes an innovative comprehensive attention converter, which adopts a U-shaped structure and consists of a layered encoder and decoder. The comprehensive attention converter includes three modules in total, a hybrid attention converter module, a multi-scale feature fusion decoder, and a multi-scale aggregation module. The hybrid attention converter module extracts multi-scale features from the output of the dual-path encoder by utilizing channel-level staggered attention and self-attention. This method helps to automatically find the most suitable encoder and decoder interconnection method, thereby reducing the effort required to find the best setting in a large number of experiments. The multi-scale feature fusion decoder contains two submodules, a channel-based efficient feature fusion module and a comprehensive attention module. During the upsampling process, the multi-scale information output by the encoder contains important channel-level features. Especially in deep networks, each channel stores directional information, which plays an important role in discovering the key lesion location in the image. The channel-based efficient feature fusion module includes a multi-path connectivity extraction module and a channel-level feature fusion module. The multi-path connectivity extraction module captures the connectivity information output by the deep network through the connection mask. The channel-level feature fusion module uses the attention method to use these directional information to select important features in the channel dimension for fusion, and passes the connectivity information to the shallow network until the final segmentation result, thereby improving the final prediction effect. The comprehensive attention module includes an enhanced converter module for extracting global context information, and combines the spatial and channel attention modules to capture local texture information. This design enables the model to retain global long-distance dependencies while capturing local features. In addition, the multi-scale features output by the decoder are hierarchically extracted through a multi-scale aggregation module, so that the final prediction result contains both semantic information from the deep layer and low-level structural features from the shallow layer, and is rich in multi-scale information. Based on the experimental results on the Synapse multi-organ dataset and the ACDC dataset, the model of the present invention demonstrates excellent effectiveness and robustness, and its performance is far superior to existing methods.

[0125] Optionally, obtaining a medical image segmentation result according to the medical image data and the comprehensive attention converter in S3 may include the following steps S31-S33:

[0126] S31. Input the medical image data into the first layer encoder to obtain the output features of the first layer encoder, and input the features into the second layer encoder to obtain the output features of the second layer encoder, then input it into the third layer encoder to obtain the output features of the third layer encoder, and finally input it into the fourth layer encoder to obtain the output features of the fourth layer encoder.

[0127] In a feasible implementation, the encoder adopts a convolution-based feature encoding path and an attention-based feature encoding path in parallel. In the field of medical image segmentation, it is necessary to pay attention not only to the accuracy of the overall contour of the image, but also to the accuracy of the local texture of the image. Although the converter-based encoder can effectively capture the global context information of the image, its performance in processing image detail features is not satisfactory. In contrast, convolutional neural networks have a natural advantage in capturing image detail features. Therefore, the dual-path encoder combines the global context capture capability of the converter with the local detail processing capability of the convolution method, making the encoding of medical images more ideal. In the convolution-based feature fusion module, in order to effectively learn high-frequency signals, we introduce a parallel structure of adaptive pooling and depthwise separable convolution. This design takes into account the advantages of adaptive pooling in perceiving surrounding neighbor features and the ability of depthwise separable convolution in extracting detail textures. As Figure 3 As shown in , adaptive pooling can dynamically adjust the pooling window to focus on features of different scales, thereby enhancing the ability to capture local information. At the same time, the depthwise separable convolution significantly reduces the computational complexity by decomposing the standard convolution into depthwise convolution and point convolution, and more finely extracts the details and texture information in the image. Through this parallel structure, our feature fusion module can more comprehensively capture and fuse high-frequency and low-frequency signals, thereby improving the overall performance of the model. Figure 4 As shown, the dual-path encoder proposed in the present invention includes at least: a convolution-based feature encoding path, an attention-based feature encoding path, a feature fusion module, and adaptive pooling. The features are converted into frequency domain signals through fast Fourier transform. High-frequency signals usually represent details and edges in the image, so the ability to obtain local and texture features can be enhanced through the convolution-based feature encoding path, while low-frequency signals usually represent the overall structure and basic contours of the image, so the ability to obtain global context dependencies can be enhanced through the attention-based feature encoding path.

[0128] Optionally, considering the perception of adaptive pooling for the features that need to be paid attention to by the surrounding neighbors and the extraction of detailed textures by deep separable convolution, the convolution-based feature fusion module adopts adaptive pooling and deep separable convolution in parallel to learn high-frequency signals, as shown in the following formula:

[0129] AdaptivePool(X)=attn ·X

[0130] Y h1 =FC(AdaptivePool(X h1 ))

[0131] Y h2 =DSConv(FC(X h2 ))

[0132] In the formula, AdaptivePool represents adaptive pooling, X represents the input feature, and att n represents the attention score of the neighbors, X h1 represents the part of the high-frequency signal used for adaptive pooling, X h2 represents the part of the high-frequency signal used for depthwise separable convolution, FC represents the linear layer, DSConv represents the depthwise separable convolution including depthwise convolution and pointwise convolution, and Y h1 and Y h2 Represent the outputs of adaptive pooling and depth-wise separable convolution, respectively.

[0133] Optionally, self-attention has the ability to extract global context information. Considering that the visual converter uses local attention, that is, self-attention is used in the local window to reduce the computational complexity, but at the same time it also loses some performance in obtaining long-distance dependencies, and the sliding window can perceive the features of the surrounding windows, and the use of adaptive pooling before applying attention also helps the local window obtain important information about neighbors. Therefore, the attention-based feature fusion module uses dual self-attention to learn low-frequency signals, that is, dual attention based on local windows and sliding windows, as shown in the following formula:

[0134] X1=W-MSA(LN(AdaptivePool(X l )))+AdaptivePool(X l )

[0135] X2=MLP(LN(X1))+X1

[0136] X3=SW-MSA(LN(X2))+X2

[0137] Y=MLP(LN(X3))+X3

[0138] Where, X l represents low-frequency signal, LN represents layer normalization, W-MSA represents local window self-attention, SW-MSA represents sliding window self-attention, MLP represents multi-layer perceptron, X i (i=1,2,3) represents the output of each stage. The whole process uses a residual structure to improve performance. lrepresents the output of the attention-based feature encoding pathway.

[0139] Optionally, the features output by the convolution-based feature encoding path and the attention-based feature encoding path are fused in the channel dimension. In order to avoid excessive smoothing of the fusion result, a depthwise separable convolution is used to exchange information between blocks channel by channel and point by point, as shown in the following formula:

[0140] Y c =Concat(Y l ,Y h1 ,Y h2 )

[0141] Y=FC(Y c +DSConv(Y c ))

[0142] Where Y l ,Y h1 ,Y h2 They represent the output of the attention-based feature encoding path and the convolution-based feature encoding path, Concat represents the merging of channel dimensions, DSConv represents depth-wise separable convolution, FC represents the linear layer, and Y c represents the merged features, and Y is the output.

[0143] Therefore, the encoder of the present invention does not rely solely on a single convolutional neural network or converter network, but rather uses a convolutional neural network and a converter network for parallel processing, respectively, based on the characteristics of high-frequency signals and low-frequency signals. Convolutional neural networks are good at perceiving local features, while converter networks excel in acquiring long-distance dependencies. Through this parallel processing approach, more efficient feature encoding is achieved. In order to overcome the shortcomings of convolutional neural networks in terms of limited perceptual field, the present invention innovatively introduces adaptive pooling and depth-separable convolution, further enhancing the ability of the convolution path to extract image texture details. On the other hand, converter networks have always faced challenges in balancing computational complexity and global context capture capabilities. To address this problem, the present invention adopts a windowing strategy to reduce computational complexity, while enhancing the perception of global information through sliding window technology.

[0144] S32. Input the first layer encoder output features, the second layer encoder output features, the third layer encoder output features and the fourth layer encoder output features into the first hybrid attention converter module respectively to obtain the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module and the fourth layer output of the first hybrid attention converter module.

[0145] In one possible implementation, a hybrid attention transformer module is proposed as a bridge between the encoder and the decoder, and between the decoder and the hierarchical prediction link, leveraging the transformer's ability to capture long-distance relationships to extract key features with rich multi-scale information. This module helps simplify the setup of skip connections and helps the model learn the best way to connect the encoder and decoder layers on a given dataset. To ensure that it can effectively extract the relationships between features with rich multi-scale information in the hierarchical encoder, a channel-level cross-attention mechanism is introduced before the traditional transformer's self-attention mechanism to fully extract multi-scale features, such as Figure 5 This design enables it to obtain the dependencies between channels before capturing the relationships between sequences, thereby better focusing on the important channel information in the sequence.

[0146] like Figure 6 As shown, the hybrid attention converter proposed in this invention includes four input Similar to the classic visual transformer process, after block embedding, it becomes First, the conventional self-attention formula is:

[0147]

[0148] Among them, Q, K, and V have the same shape N×C, and N=H×W, d head Represents the number of multi-head attention heads. Unlike self-attention, the present invention not only applies the attention mechanism block by block, but also applies the attention mechanism channel by channel. i The features are combined to obtain X Σ As Key and Value:

[0149] K=X Σ ×W K ,V=X Σ ×W V

[0150] in is the weight matrix, d is the number of blocks, C i (i=1, 2, 3, 4) are the channel dimensions of the four skip link inputs, and in the design of the present invention, the values ​​are C1=64, C2=128, C3=320, and C4=512. Thus, the channel-based multi-head attention calculation formula of the present invention is:

[0151]

[0152] In the formula, Q, K, and V represent query, key, and value respectively. K=X Σ ×WK , V = X Σ ×W V ,in represents the data of the input of the first hybrid attention converter module after block embedding, B represents batch, H represents height, W represents width, C represents the number of channels, and X represents Σ Expressed as Feature merging at the channel level, represents the weight matrix, where R represents a set of real numbers, represents the number of channels of the four skip link inputs, C Σ represents the sum of the number of channels of the four input features, T represents the transpose of the matrix, and d represents the number of blocks.

[0153] In this way, the model of the present invention can first focus on important areas at the channel level, so that in the subsequent self-attention module, it can give priority to channel-level features with greater weights, thereby better performing multi-scale feature fusion.

[0154] S33. Input the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module, and the fourth layer output of the first hybrid attention converter module into the decoder to obtain the medical image segmentation result.

[0155] Optionally, in order to effectively fuse the multi-scale semantic information output by the hybrid attention converter module in the decoder, and to learn comprehensive attention in the decoder, the decoder of the present invention mainly includes a channel-based efficient feature fusion module, a comprehensive attention module (enhanced converter module, channel attention module, spatial attention module), and a cascade method is used to more efficiently connect each attention module. Different from the traditional addition or splicing used for jump connection in upsampling, the channel-based efficient feature fusion module can better focus on the useful channel information from the encoder and the directional information contained in the channel dimension from the deep encoder. In general, the multi-scale feature fusion decoder is not only used to focus on local texture and global context, but also for capturing, transmitting and fusing features of directional information in the channel dimension.

[0156] Specifically, Figure 7 and Figure 8 As shown, the channel-based efficient feature fusion module includes: a multi-path connectivity extraction module, a first channel-level feature fusion module, a second channel-level feature fusion module, and a third channel-level feature fusion module.

[0157] The comprehensive attention module includes: a first comprehensive attention module, a second comprehensive attention module, a third comprehensive attention module and a fourth comprehensive attention module.

[0158] The above step S33 may include the following steps S331-S335:

[0159] S331. Input the fourth layer output of the first hybrid attention converter module into the multi-path connectivity extraction module to obtain the output of the multi-path connectivity extraction module, and input the output of the multi-path feature extraction module into the first comprehensive attention module to obtain the output of the first comprehensive attention module.

[0160] S332. Input the third layer output of the first hybrid attention converter module and the output of the first comprehensive attention module into the first channel-level feature fusion module to obtain the output of the first channel-level feature fusion module, and input the output of the first channel-level feature fusion module into the second comprehensive attention module to obtain the output of the second comprehensive attention module.

[0161] In a feasible implementation, the present invention selects a multi-way connectivity extraction module in the fourth layer and a channel-level feature fusion module in the other three layers. The multi-way feature extraction module uses a connection mask to capture the directional information contained in the channel, where different channels represent different directions of pixel connections. As the network deepens, the information stored in each channel gradually increases, naturally containing rich directional information. Based on this feature, the present invention uses a channel-splitting method to capture and process these connectivity information in multiple parallel channels. Subsequently, the channel-level feature fusion module transmits and utilizes the connectivity information obtained by the multi-way connectivity extraction module to the shallow network, and selects important features at the channel level through the attention mechanism for merging, thereby improving the overall representation capability of the features.

[0162] The output of the fourth layer of the hybrid attention converter module is represented as X, and then the connection mask is obtained by a dual-path method of global maximum pooling and global average pooling. Then, it is split at the channel level with X, and the directional information is extracted in a multi-path way, as shown in the following formula:

[0163] X f =δ(Conv(GAP(Conv(X)))+Conv(GMP(Conv(X))))

[0164] Y=X fi ×(SA(X i )+CA(X i ))+X i

[0165] In the formula, GAP and GMP represent global average pooling and global maximum pooling, X f represents the connection mask, δ represents the activation function, X fi represents the link mask based on channel segmentation, X iRepresents the input features based on channel segmentation. SA and CA represent the spatial attention module and the channel attention module.

[0166] In order to utilize the captured connectivity information in the decoding layer without losing it, the channel-level feature fusion module is used to merge the directional information at the channel level. The output layer i of the hybrid attention converter module is represented as And the feature map of the decoder layer i is As the input of the feature fusion module of the present invention, H, W, C are the height, width and number of channels respectively. The present invention adopts the dual-path method of global average pooling and global maximum pooling to perform channel-level fusion:

[0167]

[0168] GMP(X)=MAX(X(i,j))(i∈{1,2,3…H},j∈{1,2,3…W})

[0169] y1=δ(L1(GAP(O i ))+L2(GAP(D i )))+δ(L3(GMP(O i ))+L4(GMP(D i )))

[0170]

[0171] In the formula, GAP represents global average pooling, GMP represents global maximum pooling, X represents input features, H represents height, W represents width, X(i,j) represents features at height i and width j, y represents the area of ​​interest in the feature, δ(·) represents the ReLU activation function, and L i (i∈1,2,3,4) represents the linear layer, O i represents the output layer i of the first hybrid attention transformer module, D i represents the features of the i-th layer of the decoder, represents the feature obtained by weighting by δ(y). Then merge it with the features sampled from the decoder at the same layer.

[0172] Connectivity information is crucial for discovering the location of lesions in medical images. The directional features are captured by the multi-way connectivity extraction module, and these features are transferred and fused through the channel-level feature fusion module, which greatly improves the detection effect. Furthermore, despite the great success of visual transformers in the field of computer vision, their poor performance in extracting local features has caused researchers to once again turn their attention to convolutional neural networks. Recently, the successful application of dynamic convolution has rekindled the interest of researchers. In order to extract local textures and long-distance dependencies in images, an integrated attention module is proposed. The integrated attention module consists of three parts: an enhanced transformer module and two classic and efficient convolutional attention methods based on space and channels, such as Fig. 9 As shown in the figure. Many successful networks have tried to introduce convolution operations between the fully connected layers of the converter's feedforward neural network to make up for the converter's shortcomings in acquiring local features. Although they have improved these defects to a certain extent, there are still some problems in medical image segmentation, such as medical images are often more blurred and complex. Therefore, dynamic convolution is used to replace convolution in feedforward neural networks to achieve better local feature extraction. After this module, two classic and efficient convolution attention methods based on space and channels are added to better extract local details of the image, thereby improving the segmentation effect.

[0173] The enhanced transformer module mainly consists of two parts, self-attention and feedforward neural network. The self-attention part is the same as previous work, assuming that the feature Where H, W, and C are the height, width, and number of channels respectively. The multi-head attention calculation formula is:

[0174]

[0175] Where Q, K, and V have the same shape N×C, and N=H×W. In order to reduce the computational complexity of self-attention, the present invention uses a spatial reduction rate to reduce the spatial resolution, and then restores the original shape through linear mapping. The design of the feedforward neural network, the present invention uses a more effective dynamic convolution on the basis of the past, and then, through spatial attention and channel attention to better enhance the expression ability of features for local information, the formula is:

[0176]

[0177] out = FC(GELU(y)+x in )

[0178] where x in is the output of self-attention, FC is the fully connected layer, IdynamicConv represents dynamic convolution, is the layer normalization. The formula for channel attention is:

[0179]

[0180] where x in is the output of the feedforward neural network layer, GAP and GMP represent global average pooling and global maximum pooling respectively, conv is 1×1 convolution, and δ is the ReLu activation function. The result is then input into the spatial attention:

[0181]

[0182] where x in is the output of spatial attention, C a , c m It means that in the channel-based average and maximum operation, conv is 7×7 convolution. The final formula of the module of the present invention is:

[0183] out=SA(CA(FFN(SelfAtention(x input ))))

[0184] Where out represents the output of the enhanced converter module, SA represents the spatial attention module, CA represents the channel attention module, FFN represents the feedforward neural network, and x input Represents the module input.

[0185] S333. Input the second layer output of the first hybrid attention converter module and the output of the second comprehensive attention module into the second channel-level feature fusion module to obtain the output of the second channel-level feature fusion module, and input the output of the second channel-level feature fusion module into the third comprehensive attention module to obtain the output of the third comprehensive attention module.

[0186] S334. Input the first layer output of the first hybrid attention converter module and the output of the third comprehensive attention module into the third channel-level feature fusion module to obtain the output of the third channel-level feature fusion module, and input the third channel-level feature fusion module into the fourth comprehensive attention module to obtain the output of the fourth comprehensive attention module.

[0187] S335. Input the output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module and the output of the fourth comprehensive attention module into a multi-scale aggregation module to obtain a medical image segmentation result.

[0188] In a feasible implementation, a multi-scale aggregation module is proposed to extract features with rich multi-scale details from the decoder. Similarly, the hybrid attention converter module mentioned above is also used here for feature extraction. Unlike the method that usually only uses the last layer output of the decoder as the segmentation result, the present invention combines global semantic features with local structural features through a hierarchical prediction link, thereby making the segmentation result more accurate.

[0189] Optionally, the above step S335 may include the following steps S3351-S3352:

[0190] S3351. Input the output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module and the output of the fourth comprehensive attention module into the second hybrid attention converter module respectively to obtain the first layer output of the second hybrid attention converter module, the second layer output of the second hybrid attention converter module, the third layer output of the second hybrid attention converter module and the fourth layer output of the second hybrid attention converter module.

[0191] S3352. The first layer output of the second hybrid attention converter module, the second layer output of the second hybrid attention converter module, the third layer output of the second hybrid attention converter module, and the fourth layer output of the second hybrid attention converter module are respectively input into the multi-layer prediction link to obtain the medical image segmentation result.

[0192] In a feasible implementation, in the past, the output of the last layer was often selected for the final prediction result, but the present invention adopts a multi-level prediction link. Fig.10 As shown, the present invention proposes a multi-scale aggregation module. Because the hierarchical result of the final output also has rich multi-scale information, the high-scale features have local contextual relationships and the low-scale features have global long-distance dependencies. Therefore, before the final prediction, the present invention adds a hybrid attention converter module. The hybrid attention converter module extracts long-distance dependencies and local relationships to help features have better representation capabilities. Conventional methods usually only use the output of the last layer of the decoder as the final prediction result, ignoring the multi-scale information in the decoder. Here, the use of a multi-link aggregation network can well preserve these multi-scale information for the final prediction result. The present invention assumes that x1, x2, x3, x4 are the outputs of the decoder, then the formula is:

[0193] p1,p2,p3,p4=MATrans(x1,x2,x3,x4)

[0194] Wherein MATrans is the hybrid attention converter module mentioned above. After MATrans encoding, the final prediction value calculated by the present invention is:

[0195] output=w×p1+x×p2+y×p3+z×p4

[0196] Where p1, p2, p3, p4 are the predicted values ​​of MATrans in four directions, and w, x, y, z are the weights of each predicted value. In the design of the present invention, each weight value is set to 1.0. For the binary segmentation task, the present invention uses the Sigmoid activation function for the final predicted value, and for the multi-classification segmentation task, the present invention uses the SoftMax activation function.

[0197] The final total loss of the present invention uses the formula:

[0198] loss=αloss1+βloss2+γloss3+δloss4

[0199] Among them, loss i (i=1, 2, 3, 4) is the loss value of each prediction direction, and α, β, γ, δ are the weights of each loss value. In the design of the present invention, each weight is also set to 1.0.

[0200] Before the final prediction, a multi-scale aggregation module is introduced to learn high-level semantic information to capture the complex context and relationships in the image, while obtaining low-level structural features to obtain texture and details in the image. Then, the hierarchical prediction values ​​are aggregated to obtain the final score. For the binary segmentation task, the Sigmoid function is used, while for the multi-class segmentation task, the SoftMax function is used to calculate the loss.

[0201] In an embodiment of the present invention, a new method, a dual-path comprehensive attention converter, is proposed. The encoder uses a dual path based on convolution and converter to encode features in parallel, while the decoder uses a cascaded method. Different from the common single-path encoder, the present invention uses the advantage of convolutional networks to extract local information to process high-frequency signals, and uses the ability of converters to capture long-distance dependencies to process low-frequency signals, thereby giving full play to their respective advantages. Different from using attention gates in the decoder to obtain important areas of the image, the present invention uses an enhanced converter module and two classic and effective convolutional attention methods based on space and channels. These methods retain long-distance relationships while extracting local features, thereby improving the feature extraction performance of global context and local texture in the decoder. In order to capture the multi-scale features output by the encoding layer and the directional information contained in the deep network during upsampling, an efficient feature fusion module based on channels is introduced to fuse and pass these important features to the upper layer.

[0202] At the same time, the present invention introduces a hybrid attention converter module between the encoder and the corresponding decoder to set the jump connection more efficiently. The hybrid attention converter module adds channel-level multi-head cross attention before the multi-head self-attention, so as to effectively obtain the relationship between channel dimensions with rich multi-scale features, and then capture the relationship between different sequences through self-attention. Before the final prediction, the present invention adopts hierarchical prediction links. For the features from the decoder, the hybrid attention converter module is used again to extract features with rich multi-scale features, so that the final prediction result contains both semantic information from the deep layer and low-level structural features from the shallow layer.

[0203] Fig.11 1 is a block diagram of a dual-path medical image segmentation device based on multi-level comprehensive attention according to an exemplary embodiment, and the device is used for a dual-path medical image segmentation method based on multi-level comprehensive attention. Fig.11 The device includes an acquisition module 310, an input module 320 and an output module 330.

[0204] in:

[0205] The acquisition module is used to acquire the medical image data to be segmented.

[0206] Input module, used to input medical image data into the constructed comprehensive attention transformer.

[0207] The output module is used to obtain the medical image segmentation result according to the medical image data and the comprehensive attention converter.

[0208] Among them, the comprehensive attention converter includes: a dual-path encoder, a first hybrid attention converter module and a decoder.

[0209] The dual-path encoder includes: a first-layer dual-path encoder, a second-layer dual-path encoder, a third-layer dual-path encoder and a fourth-layer dual-path encoder.

[0210] The decoder includes: a multi-scale feature fusion decoder and a multi-scale aggregation module.

[0211] The multi-scale feature fusion decoder includes: a channel-based efficient feature fusion module and a comprehensive attention module.

[0212] The channel-based efficient feature fusion module includes: a multi-path connectivity extraction module and a channel-level feature fusion module.

[0213] The comprehensive attention module includes: enhanced converter module, channel attention module, spatial attention module and convolution block.

[0214] The multi-scale aggregation module includes: a second hybrid attention converter module and a multi-layer prediction chain.

[0215] In an embodiment of the present invention, a new method is proposed to integrate attention converters, which are cascaded in the decoder. Different from using attention gates in the decoder to obtain important areas of the image, the present invention uses an enhanced converter module and two classic and effective convolutional attention methods based on space and channels. These methods retain long-range relationships while extracting local features, thereby improving the feature extraction performance of global context and local texture in the decoder.

[0216] At the same time, the present invention introduces a hybrid attention converter module between the encoder and the corresponding decoder to set the jump connection more efficiently. The hybrid attention converter module adds channel-level multi-head cross attention before the multi-head self-attention, so as to effectively obtain the relationship between channel dimensions with rich multi-scale features, and then capture the relationship between different sequences through self-attention. Before the final prediction, the present invention adopts hierarchical prediction links. For the features from the decoder, the hybrid attention converter module is used again to extract features with rich multi-scale features, so that the final prediction result contains both semantic information from the deep layer and low-level structural features from the shallow layer.

[0217] Fig.12 is a schematic diagram of the structure of a medical image segmentation device provided by an embodiment of the present invention, such as Fig.12 As shown, the medical image segmentation device may include the above Fig.11 The dual-path medical image segmentation apparatus based on multi-level comprehensive attention is shown. Optionally, the medical image segmentation device 410 may include a first processor 2001 .

[0218] Optionally, the medical image segmentation device 410 may further include a memory 2002 and a transceiver 2003 .

[0219] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0220] Combine the following Fig.12 The components of the medical image segmentation device 410 are specifically introduced as follows:

[0221] The first processor 2001 is the control center of the medical image segmentation device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0222] Optionally, the first processor 2001 may perform various functions of the medical image segmentation device 410 by running or executing a software program stored in the memory 2002 , and calling data stored in the memory 2002 .

[0223] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Fig.12 CPU0 and CPU1 are shown in FIG.

[0224] In a specific implementation, as an embodiment, the medical image segmentation device 410 may also include multiple processors, such as Fig.12 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0225] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0226] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Fig.12 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0227] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dual-path medical image segmentation method based on multi-level comprehensive attention, characterized in that: The method comprises: S1. Obtain medical image data to be segmented; S2, inputting the medical image data into the constructed comprehensive attention converter; S3, obtaining a medical image segmentation result according to the medical image data and the comprehensive attention converter; The integrated attention converter comprises: a dual-path encoder, a first hybrid attention converter module and a decoder; The dual-path encoder comprises: a first-layer dual-path encoder, a second-layer dual-path encoder, a third-layer dual-path encoder and a fourth-layer dual-path encoder; The decoder comprises: a multi-scale feature fusion decoder and a multi-scale aggregation module; The multi-scale feature fusion decoder includes: a channel-based efficient feature fusion module and a comprehensive attention module; The channel-based efficient feature fusion module includes: a multi-path connectivity extraction module and a channel-level feature fusion module; The comprehensive attention module includes: an enhanced converter module, a channel attention module, a spatial attention module and a convolution block; The multi-scale aggregation module includes: a second hybrid attention converter module and a multi-layer prediction link.

2. The dual-path medical image segmentation method based on multi-level comprehensive attention according to claim 1, characterized in that: The step of obtaining a medical image segmentation result according to the medical image data and the comprehensive attention converter in S3 includes: S31, inputting the medical image data into the first layer dual path encoder after block embedding to obtain the first layer dual path encoder output feature, and inputting the feature into the second layer dual path encoder to obtain the second layer dual path encoder output feature, then inputting into the third layer dual path encoder to obtain the third layer dual path encoder output feature, and finally inputting into the fourth layer dual path encoder to obtain the fourth layer dual path encoder output feature; S32, inputting the first layer dual path encoder output features, the second layer dual path encoder output features, the third layer dual path encoder output features and the fourth layer dual path encoder output features into the first hybrid attention converter module respectively, to obtain the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module and the fourth layer output of the first hybrid attention converter module; S33. Input the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module, and the fourth layer output of the first hybrid attention converter module into the decoder to obtain a medical image segmentation result.

3. The dual-path medical image segmentation method based on multi-level comprehensive attention according to claim 2, characterized in that: The dual-path encoder includes at least: a convolution-based feature encoding path, an attention-based feature encoding path, a feature fusion module, and adaptive pooling; The convolution-based feature encoding path uses adaptive pooling and depth-separable convolution in parallel to learn high-frequency signals, as shown below: AdaptivePool(X)=to n ·X (1) Y h1 =FC(AdaptivePool(X h1 )) (2) Y h2 =DSConc(FC(X h2 )) (3) In the formula, AdaptivePool represents adaptive pooling, X represents the input feature, and att n represents the attention score of the neighbors, X h1 represents the part of the high-frequency signal used for adaptive pooling, X h2 represents the part of the high-frequency signal used for depthwise separable convolution, FC represents the linear layer, DSConc represents the depthwise separable convolution including depthwise convolution and pointwise convolution, and Y h1 and Y h2 Represent the outputs of adaptive pooling and depth-wise separable convolution respectively; The attention-based feature encoding path uses dual self-attention to learn low-frequency signals, that is, dual attention based on local windows and sliding windows, as shown below: X1=W-MSA(LN(AdaptivePool(X l )))+AdaptivePool(X l ) (4) X2=MLP(LN(X1))+X1 (5) X3=SW-MSA(LN(X2))+X2 (6) Y l =MLP(LN(X3))+X3 (7) Where, X l represents low-frequency signal, LN represents layer normalization, W-MSA represents local window self-attention, SW-MSA represents sliding window self-attention, MLP represents multi-layer perceptron, X i (i=1,2,3) represents the output of each stage. The whole process uses a residual structure to improve performance. l represents the output of the attention-based feature encoding pathway; The feature fusion module fuses the features output by the convolution-based feature encoding path and the attention-based feature encoding path in the channel dimension, and uses depthwise separable convolution to exchange information between blocks channel by channel and point by point, as shown in the following formula: AND C =Concat(And l ,AND h1 ,AND h2 ) (8) Y=FC(Y c +DSConv(Y c )) (9) Where Y l ,Y h1 ,Y h2 They represent the output of the attention-based feature encoding path and the convolution-based feature encoding path, Concat represents the merging of channel dimensions, DSConv represents depth-wise separable convolution, FC represents the linear layer, and Y c represents the merged features, and Y is the output; The first hybrid attention converter module at least includes: a channel-level interleaved attention module, a self-attention module and a feedforward neural network; the channel-level interleaved attention module is introduced before the self-attention module to first capture long-distance features at the channel level, and then on this basis, perceive key local features at the sequence level through the self-attention mechanism to realize automatic learning of the connection mode of the jump link; The channel-based multi-head attention calculation formula of the first hybrid attention converter module is shown in the following formula (10): In the formula, Q, K, and V represent query, key, and value respectively. K=X Σ ×W K , V = X Σ ×W V ,in represents the data of the input of the first hybrid attention converter module after block embedding, B represents batch, H represents height, W represents width, C represents the number of channels, and X represents Σ Expressed as Feature merging at the channel level, represents the weight matrix, where R represents a set of real numbers, represents the number of channels of the four skip link inputs, C Σ represents the sum of the number of channels of the four input features, T represents the transpose of the matrix, and d represents the number of blocks.

4. The dual-path medical image segmentation method based on multi-level comprehensive attention according to claim 2, characterized in that: The channel-based efficient feature fusion module includes: a multi-path connectivity extraction module, a first channel-level feature fusion module, a second channel-level feature fusion module, and a third channel-level feature fusion module; The comprehensive attention module includes: a first comprehensive attention module, a second comprehensive attention module, a third comprehensive attention module and a fourth comprehensive attention module; The step S33 inputs the first layer output of the first hybrid attention converter module, the second layer output of the first hybrid attention converter module, the third layer output of the first hybrid attention converter module, and the fourth layer output of the first hybrid attention converter module into the decoder to obtain a medical image segmentation result, including: S331, inputting the fourth layer output of the first hybrid attention converter module into the multi-path connectivity extraction module to obtain the output of the multi-path connectivity extraction module, and inputting the output of the multi-path feature extraction module into the first comprehensive attention module to obtain the first comprehensive attention module output; S332, inputting the third layer output of the first hybrid attention converter module and the output of the first comprehensive attention module into the first channel-level feature fusion module to obtain the first channel-level feature fusion module output, and inputting the output of the first channel-level feature fusion module into the second comprehensive attention module to obtain the second comprehensive attention module output; S333, inputting the second layer output of the first hybrid attention converter module and the second comprehensive attention module output into the second channel-level feature fusion module to obtain the second channel-level feature fusion module output, and inputting the second channel-level feature fusion module output into the third comprehensive attention module to obtain the third comprehensive attention module output; S334, inputting the first layer output of the first hybrid attention converter module and the output of the third comprehensive attention module into the third channel-level feature fusion module to obtain the output of the third channel-level feature fusion module, and inputting the third channel-level feature fusion module into the fourth comprehensive attention module to obtain the output of the fourth comprehensive attention module; S335. Input the output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module and the output of the fourth comprehensive attention module into the multi-scale aggregation module to obtain a medical image segmentation result.

5. The dual-path medical image segmentation method based on multi-level comprehensive attention according to claim 4, characterized in that: The multi-path connectivity extraction module captures the directional information of the channel dimension in the deep network, represents the output of the fourth layer of the hybrid attention converter module as X, and then obtains the connection mask through the dual-path method of global maximum pooling and global average pooling, and then splits X at the channel level, and extracts the directional information in a multi-path manner, as shown in the following formula: X f =δ(Conv(GAP(Conv(X)))+Conv(GMP(Conv(X)))) (11) Y=X fi ×(SA(X i )+CA(X i ))+X i (12) In the formula, GAP and GMP represent global average pooling and global maximum pooling, X f represents the connection mask, δ represents the activation function, X fi represents the link mask based on channel segmentation, X i represents the input features based on channel segmentation, CA and SA represent the channel attention module and the spatial attention module; The channel-level feature fusion module transfers and fuses directional information. Each module uses a dual-path approach of global average pooling and global maximum pooling to perform channel-level fusion, as shown in the following equations (13)-(16): GMP(X)=MAX(X(i,j))(i∈{1,2,3…H},j∈{1,2,3…W}) (14) y16δ(L1(GAP(O). i ))+L2(GAP(D). i )))+δ(L3(GMP(O). i ))+L4(GMP(D). i ))) (15) In the formula, GAP represents global average pooling, GMP represents global maximum pooling, X represents input features, H represents height, W represents width, X(i,j) represents features at height i and width j, y1 represents the area of ​​interest in the feature, δ(·) represents the ReLU activation function, and L i (i∈1,2,3,4) represents the linear layer, O i represents the output layer i of the first hybrid attention transformer module, D i represents the features of the i-th layer of the decoder, Represents the feature weighted by δ(y).

6. The dual-path medical image segmentation method based on multi-level comprehensive attention according to claim 4, characterized in that: The comprehensive attention module includes an enhanced converter module, a channel attention module, a spatial attention module and a convolution block, which is used to retain local features while learning long-range dependencies; The enhanced converter module includes at least: a self-attention layer and a feed-forward neural network layer; The enhancement is in the feedforward neural network, which uses a dynamic convolution layer and layer normalization to three layers of residual layer normalization to improve the representation ability of local features, as shown in the following formulas (17) and (18): out1=FC(GELU(y)+x in1 ) (18) In the formula, y2 represents the output of dynamic convolution, x in1 represents the output of self-attention, FC represents the fully connected layer, IdynamicConv represents the dynamic convolution, represents layer normalization, GELU represents activation function, and out1 represents the output of feedforward neural network; The channel attention module is shown in formula (19): In the formula, x in2 represents the output of the feedforward neural network layer, GAP represents global average pooling, GMP represents global maximum pooling, conv represents 1×1 convolution, and δ represents the ReLu activation function; The spatial attention module is shown in equation (20): In the formula, x in3 represents the output of the channel attention module, C a represents the average value of the feature in the channel dimension, C m Represents the maximum value of the feature in the channel dimension, conv represents 7×7 convolution, and δ represents the ReLU activation function; The output of the comprehensive attention module is shown in the following formula (21): out2=SA(CA(FFN(SelfAttention(x input )))) (21) Where out2 represents the output of the comprehensive attention module, SA represents the spatial attention module, CA represents the channel attention module, FFN represents the feedforward neural network, Selfattention represents the self-attention, and x input Represents the input features of the comprehensive attention module.

7. The dual-path medical image segmentation method based on multi-level comprehensive attention according to claim 4, characterized in that: The step S335 inputs the output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module, and the output of the fourth comprehensive attention module into the multi-scale aggregation module to obtain a medical image segmentation result, including: Inputting the output of the first comprehensive attention module, the output of the second comprehensive attention module, the output of the third comprehensive attention module, and the output of the fourth comprehensive attention module into the second hybrid attention converter module respectively, obtaining the first layer output of the second hybrid attention converter module, the second layer output of the second hybrid attention converter module, the third layer output of the second hybrid attention converter module, and the fourth layer output of the second hybrid attention converter module; The first layer output of the second hybrid attention converter module, the second layer output of the second hybrid attention converter module, the third layer output of the second hybrid attention converter module, and the fourth layer output of the second hybrid attention converter module are respectively input into the multi-layer prediction chain to obtain the medical image segmentation result, as shown in the following equations (22)(23): p1,p2,p3,p4=MATrans(x1,x2,x3,x4) (22) output=w×p1+x×p2+y×p3+z×p4 (23) In the formula, x i (i∈(1,2,3,4)) represents the output of the comprehensive attention module, p i (i∈(1,2,3,4)) represents the predicted values ​​of the four prediction links, w,x,y,z represent the weights of each predicted value, and output represents the final prediction result.

8. A dual-path medical image segmentation device based on multi-level comprehensive attention, the dual-path medical image segmentation device based on multi-level comprehensive attention is used to implement the dual-path medical image segmentation method based on multi-level comprehensive attention as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used for acquiring medical image data to be segmented; An input module, used for inputting the medical image data into the constructed comprehensive attention converter; An output module, used for obtaining a medical image segmentation result according to the medical image data and the comprehensive attention converter; The integrated attention converter comprises: a dual-path encoder, a first hybrid attention converter module and a decoder; The dual-path encoder comprises: a first-layer dual-path encoder, a second-layer dual-path encoder, a third-layer dual-path encoder and a fourth-layer dual-path encoder; The decoder comprises: a multi-scale feature fusion decoder and a multi-scale aggregation module; The multi-scale feature fusion decoder includes: a channel-based efficient feature fusion module and a comprehensive attention module; The channel-based efficient feature fusion module includes: a multi-path connectivity extraction module and a channel-level feature fusion module; The comprehensive attention module includes: an enhanced converter module, a channel attention module, a spatial attention module and a convolution block; The multi-scale aggregation module includes: a second hybrid attention converter module and a multi-layer prediction link.

9. A medical image segmentation device, characterized in that: The medical image segmentation device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images