Multi-attention mechanism collaborative medical image segmentation method

Abstract

The invention belongs to the technical field of computer image processing, and particularly relates to a multi-attention mechanism collaborative medical image segmentation method, which comprises the following steps: acquiring data of a multi-organ segmentation data set and an automatic heart diagnosis challenge data set, forming a multi-organ image data set, and performing data enhancement; constructing a medical image segmentation model based on layered multi-scale attention; and integrating the hybrid expert model into the medical image segmentation model. According to the method, a multi-attention mechanism collaborative medical image segmentation model is constructed through an integral method, a plurality of attention mechanisms are collaborative in a layered form in the preliminary construction of the model, the medical image segmentation model based on layered multi-scale attention is constructed, and then a mixed expert model is added into the model, so that the medical image segmentation efficiency is improved. Therefore, the most appropriate attention mechanism can be more activated for the current task, the medical image segmentation model is further optimized, and the final medical image segmentation model with multiple attention mechanisms cooperating is formed.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing technology, and in particular to a medical image segmentation method using a multi-attention mechanism collaboration. Background Art

[0002] Medical images can reflect the anatomical structures or functional tissues within the human body, generating discrete image representations through sampling or reconstruction. These images are characterized by low contrast, blurred boundaries, and inaccurate visual recognition. Commonly used medical imaging technologies include computed tomography (CT), ultrasound, magnetic resonance imaging (MRI), and X-rays. Doctors can obtain more detailed medical images, which makes early diagnosis, treatment, and prognosis of diseases possible. However, these medical images are often very complex and contain a wealth of detailed information, making manual analysis and diagnosis difficult. Therefore, accurately extracting and analyzing important structures or lesion areas from these images has become a challenge.

[0003] Medical image segmentation is the process of automatically identifying and extracting individual structures or abnormal regions within an image. By dividing an image into regions with distinct characteristics, it helps doctors quickly identify key areas within massive amounts of imaging data. For example, segmenting brain tumors, lung nodules, blood vessels, organs, and other regions can help doctors more accurately diagnose a condition. With the rapid development and widespread adoption of medical imaging equipment, imaging technology has been widely used in clinical practice, becoming an indispensable aid for disease diagnosis, surgical planning, prognosis assessment, and follow-up. Medical images often play a crucial role in diagnosis and treatment. Therefore, medical images have become one of the most important sources of evidence for clinical analysis and medical intervention.

[0004] Medical image segmentation extracts key information from images of specific tissues and is a crucial step in achieving medical image visualization. Segmented images are provided to physicians for tasks such as quantitative tissue volume analysis, diagnosis, localization of pathological changes, delineation of anatomical structures, and treatment planning. Medical images contain vast amounts of information, and manually outlining target regions in clinical images is a time-consuming and laborious task, placing a significant burden on clinicians' daily routines. Therefore, the development of automated, highly accurate medical image segmentation models is highly urgent.

[0005] While DNNs hold great potential for automated segmentation in medical image analysis tasks like disease diagnosis, significant challenges remain. Unlike regular images, which often feature prominent features, medical images require the recognition of subtle changes in density or shape, which necessitates highly accurate and reliable AI models. Consequently, DNNs for medical applications are typically built with extensive convolutional layers, applying a high receptive field to capture these nuances. This significantly increases computational requirements. Worse still, such models can easily become trapped in local optima, leading to overfitting. Inspired by the human visual system, the attention mechanism was proposed as an efficient way to address these limitations of neural networks by selectively focusing on informative features. Similar to how the human eye fixates on key regions in a scene, early attention research assigned higher weights to relevant parts of the input sequence, such as text or images, for downstream tasks. In the context of image segmentation, we refer to this type of attention as short-range attention, as it allows the neural network to capture dependencies within a single sample, essentially focusing on regions of interest within the image.

[0006] While short-range attention is good at capturing local features in an image, it often misses the bigger picture. Therefore, it is necessary to extract high-level features that define the entire scene to summarize the overall content, which has led to the exploration of new attention mechanisms. For example, the Transformer and its image-specific variant ViT use attention to capture global information. Not surprisingly, the wider perspective helps improve performance compared to traditional neural networks. However, this attention still faces limitations because it does not consider the relationship between features across different images. In other words, it focuses on global features of the image rather than features of the entire dataset. Therefore, we define this type of attention as medium-range attention.

[0007] However, both short-range and medium-range attention are designed to extract features from a single image, without considering external information that can enrich image understanding. To address this issue, we define long-range attention, which extends the concept beyond a single example, allowing the neural network to capture relationships between features across images in the entire dataset. Ideally, it provides an additional field of view, leveraging this additional range of information to achieve more accurate and nuanced results.

[0008] Existing medical image segmentation methods only use one or two attention mechanisms, which affects the further improvement of the segmentation accuracy of medical image segmentation models.

[0009] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0010] The purpose of the present invention is to solve the technical problem existing in the background technology: the multi-organ segmentation accuracy of the existing medical image segmentation model is low. To this end, a medical image segmentation method with a multi-attention mechanism collaboration is provided.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0012] A multi-attention mechanism collaborative medical image segmentation method includes the following steps:

[0013] Step 1: Obtain the multi-organ segmentation dataset and the automatic cardiac diagnosis challenge dataset to form a multi-organ imaging dataset;

[0014] Step 2: Preprocess the multi-organ imaging dataset;

[0015] Step 3. Construct a network structure for a medical image segmentation model based on hierarchical multi-scale attention. The network structure of the medical image segmentation model adopts an encoder-decoder structure, including five convolutional layers and one bottleneck layer. The first three layers of the five convolutional layers are convolutional modules with a short-range attention mechanism, which are used to strengthen convolution and enable the network structure to focus on local and important areas of the input image. The last two layers of the five convolutional layers are a mixture of convolutional modules and two layers of LGE modules, which are used to capture and model the global feature information of a single sample and the feature information between different samples. The LGE module consists of a medium-range attention mechanism module and a long-range attention mechanism module. Finally, the bottleneck layer is composed of an LGE module.

[0016] Step 4: Add the hybrid expert model to optimize the collaborative strategy among the short-distance attention mechanism, medium-distance attention mechanism, and long-distance attention mechanism of the hierarchical multi-scale attention medical image segmentation model to form the final multi-attention mechanism collaborative medical image segmentation model.

[0017] The following is a technical solution further defined by the present invention. In step 2, the images in the multi-organ imaging dataset are normalized, and then standardized, the mean is subtracted, and the image is divided by the standard deviation. Secondly, the size of the image is unified. Finally, random flipping and random rotation are used to expand the multi-organ imaging dataset.

[0018] The following is a technical solution further defined by the present invention, in step 3, comprising the following steps:

[0019] Step 301: The pre-processed multi-organ image dataset enters the fusion module of the short-range attention mechanism module and the convolution module for processing, and outputs the processed feature map;

[0020] Step 302: Based on the feature map output in step 301, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map;

[0021] Step 303: repeat step 301 and step 302, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 2;

[0022] Step 304: Use the medium-range attention mechanism module to capture the global information in the medical image, that is, the relationship between organs, and use the long-range attention mechanism module to capture the relationship between medical images, that is, the connection between images. The process includes the following:

[0023] Based on the feature map finally processed and outputted in step 303, it enters the mid-range attention mechanism module of the LGE module and is divided into n windows, where n≥4. In each window, the local attention calculates the self-affinity, thereby generating a weighted local feature map U local :

[0024] U local =LA(X)

[0025] Global Tokens captures the global feature information within each window, that is, through the global attention algorithm, a global feature map U based on global Tokens is obtained. Global :

[0026] U Global =GA(DLConv(U local ))

[0027] For the global feature map U Global Upsample and combine it with the local feature map U local Splicing to minimize information loss and calculate the final feature map U of the entire image * :

[0028] U * =Concat(U local ,Upsample(U Global ))

[0029] After that, the feature map U * Entering the long-range attention mechanism module of the LGE module: First, the input feature map is linearly projected into the query matrix Q, the key matrix K, and the value matrix V; then, the similarity between the query matrix Q and the external key memory unit Mk is calculated to obtain the attention map A; finally, the input features are updated by multiplying the attention map A with the external value memory unit Mv to obtain the processed feature map;

[0030] The obtained feature map repeatedly enters the medium-distance attention mechanism module and the long-distance attention mechanism module of the LGE module for processing, and then passes through the convolution module for convolution processing to obtain the processed feature map;

[0031] Step 305: Based on the feature map output in step 304, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map;

[0032] Step 306: repeat step 304 and step 305, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 1;

[0033] Step 307: Based on the feature map output in step 306, the feature map is processed into the medium-range attention mechanism module and the long-range attention mechanism module of the LGE module to obtain a processed feature map without the need for convolution processing in the convolution module.

[0034] Step 308: Based on the feature map output in step 307, the bottleneck layer of the model is upsampled to reach the decoding network, and the final segmentation result is obtained through the decoding network, thereby completing the construction of the network structure of the medical image segmentation model based on hierarchical multi-scale attention.

[0035] The following is a technical solution further defined by the present invention. In step 301, the fusion module is a series connection of two ConvBNReLU convolution modules and a short-distance attention mechanism module, which gives full play to the role of the short-distance attention mechanism module in strengthening convolution. Each ConvBNReLU convolution module contains a 3×3 convolution kernel, batch normalization and ReLu activation function, forming a complete convolution operation. The short-distance attention mechanism module infers 3-D attention weights for the feature maps in the layer without adding additional parameters to the original network, that is, directly infers the spatial and channel dimension attention weights of the feature maps, thereby enhancing the feature extraction capability, focusing on the key areas of the medical image to be segmented, making the model pay more attention to the organ tissue to be segmented, and outputting the processed feature maps.

[0036] The following is a technical solution further defined by the present invention. In step 302, the feature map first enters the lossless feature coding block and performs the following processing steps in sequence: using low-pass and high-pass filters to extract low-frequency and high-frequency information; performing two downsampling operations to divide the image into low-frequency components, horizontal components, vertical components and diagonal components; fusing to form a processed feature map; thereafter, the feature map enters the feature representation learning block composed of a 1×1 convolutional layer, batch normalization and ReLu activation function to form the final processed feature map.

[0037] The following is a technical solution further defined by the present invention. In step 308, the feature map output by the bottleneck layer is deconvolved to expand the spatial resolution of the feature map, and the feature map is input into the module corresponding to the encoding network. During this period, each level splices the feature map of the corresponding layer in the encoding network with the upsampled feature map, and finally restores the feature map to its original size. After segmentation head processing and softmax activation function, the network output is converted into a probability distribution to obtain the final multi-organ segmentation result.

[0038] The following is a technical solution further defined by the present invention, in step 4, comprising the following steps:

[0039] Step 401: The pre-processed multi-organ image dataset enters a coding network composed of partial convolution modules, extracts spatial features using partial convolution, and outputs a processed feature map.

[0040] Step 402: Based on the feature map output in step 401, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map;

[0041] Step 403: repeat step 401 and step 402, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 2;

[0042] Step 404: Design an AMOE module to improve the coordination strategy among the three attention mechanisms: short-range attention mechanism, medium-range attention mechanism, and long-range attention mechanism. The AMOE module contains two MoE layers: the attention head mixture of experts layer and the MLP mixture of experts layer, including the following processing steps:

[0043] Based on the feature map finally processed and outputted in step 403, first, the feature map is divided into two branches;

[0044] First, after passing through the layer normalization layer, it enters the attention head mixed expert layer. After being processed by the gating network using the Top-k algorithm, three attention experts are sparsely activated, namely the short-range attention mechanism, the medium-range attention mechanism, and the long-range attention mechanism. Among them, layer normalization ensures training stability and smooth gradient propagation. The gating network assigns different activation weights to different attention modules based on the distribution of input features, realizing sparse dynamic selection of attention experts, allowing the model to use the most appropriate attention strategy under different input conditions;

[0045] The second one is fused with the output of one branch through residual connection to form a new feature map;

[0046] Then, the feature map is divided into two branches;

[0047] After passing through the layer normalization layer again and again, it enters the MLP mixed expert layer. The input features first pass through the gating network, which dynamically selects and activates the most appropriate one or more experts based on the feature distribution of the input content;

[0048] After the activated MLP experts process the input, they aggregate the results to form the output;

[0049] Similarly, the second one will be fused with the output of one branch through residual connection to form a new feature map;

[0050] After obtaining the feature map, the processing operation of the AMOE module is repeated again, and the final feature map is obtained after a layer of partial convolution module;

[0051] Step 405: Based on the feature map output in step 404, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map;

[0052] Step 406: repeat step 404 and step 405, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 1;

[0053] Step 407: Based on the feature map outputted in step 406, the processing operation of the AMOE module is repeated to obtain a processed feature map without the need for partial convolution processing by the partial convolution module.

[0054] Step 408: Based on the feature map output in step 407, the bottleneck layer of the model is upsampled to the decoding network, and the final segmentation result is obtained through the decoding network, thereby completing the construction of the medical image segmentation model with multi-attention mechanism collaboration.

[0055] The following is a technical solution further defined by the present invention. In step 401, the images of the preprocessed multi-organ imaging dataset are divided into two branches, one branch is used for the final splicing, and the other branch enters the partial convolution with a convolution kernel of 3×3, and then passes through a convolution with a convolution kernel of 1×1, and then passes through batch normalization and ReLu activation function, and finally passes through the convolution with a convolution kernel of 1×1 and is spliced ​​with the feature map of the first branch, performing the convolution operation on a partial area of ​​the input image rather than the entire image.

[0056] Compared with the prior art, the present invention has the following technical effects:

[0057] The present invention constructs a medical image segmentation model with multiple attention mechanisms working together through a holistic approach. In the initial construction of the model, multiple attention mechanisms are coordinated in a layered form to construct a medical image segmentation model based on hierarchical multi-scale attention. Then, by adding a hybrid expert model to the model, the most appropriate attention mechanism is activated more for the current task, and the medical image segmentation model is further optimized to form the final medical image segmentation model with multiple attention mechanisms working together.

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 This is the overall model architecture diagram of the medical image segmentation model based on hierarchical multi-scale attention in the present invention.

[0061] Figure 2 It is a structural diagram of SimAM and convolution fusion described in the present invention.

[0062] Figure 3 It is a calculation flow chart of the Local-Global Attention described in the present invention.

[0063] Figure 4 This is a structural diagram of the LGE module described in the present invention.

[0064] Figure 5 This is the architecture diagram of the medical image segmentation model that combines MoE with the attention mechanism.

[0065] Figure 6 This is the AMOE module architecture diagram. DETAILED DESCRIPTION

[0066] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0067] like Figure 1-6As shown, this embodiment provides a medical image segmentation method with multiple attention mechanisms coordinating attention, thereby constructing a medical image segmentation model with multiple attention mechanisms coordinating attention. Specifically, the following steps are included:

[0068] Step 1: Obtain the Synapse multi-organ segmentation dataset and the automatic cardiac diagnosis challenge dataset data.

[0069] The multi-organ segmentation image data used in the present invention is a public dataset. The Synapse multi-organ segmentation dataset includes 3779 axial-enhanced abdominal CT images from 30 contrast-enhanced clinical CT cases. The data is divided into a training set (18 cases, 2212 axial slices) and a test set (12 cases, 1567 axial slices). Each image includes segmentation labels of eight abdominal organs: aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas and stomach. The Automatic Cardiac Diagnosis Challenge Dataset (ACDC) provides 70 training images, 10 verification images and 20 test images. Each image includes segmentation labels of three cardiac structures: left ventricle, right ventricle and myocardium. The relationship with the latter step is to provide training data.

[0070] Step 2: Preprocess the multi-organ imaging dataset.

[0071] First, normalize the images in the dataset obtained in step 1, scaling the pixel values ​​to [0, 1]. Then, standardize by subtracting the mean and dividing by the standard deviation. Next, resize the images to 224×224. Finally, perform data augmentation, including random flipping and rotation, to generate more synthetic samples, thereby improving the model's generalization capabilities. This step is related to the previous step in preprocessing the multi-organ imaging dataset. It also facilitates data processing in the following step.

[0072] Step 3: Design the network architecture of RadioNet, a medical image segmentation model based on hierarchical multi-scale attention. This model employs a classic encoder-decoder architecture. The first three layers of the network are convolutional modules with a parameterless attention mechanism. This strengthens the convolutions, allowing the network to focus more on local and important regions of the input image, such as the organs to be segmented in patient images. The last two layers of the network are a hybrid of convolutional modules and two layers of LGE modules. Their design aims to capture and model both global feature information within a single sample and feature information across different samples. The LGE module is a specially designed module consisting of two components: Local-Global Attention and External Attention. Finally, the bottleneck layer of the network also consists of an LGE module. Haar wavelet downsampling is used in the downsampling phase of the feature extraction network to prevent information loss. Furthermore, skip connections are used in the decoding phase to preserve low-level features. The network's relationship with the previous step is that it requires data from the previous step for training. Its relationship with the subsequent step is that it provides a medical image segmentation model that can be optimized.

[0073] like Figure 1 As shown, it mainly includes the following steps:

[0074] Step 301: The non-parametric attention mechanism (short distance) SimAM captures local key information in the medical image. The input of this step is the medical image to be segmented after data enhancement. The processing process is that the medical image will enter the fusion module of the non-parametric attention mechanism (short distance) SimAM and the convolution module. Its core structure is the series connection of two ConvBNReLU convolution modules and SimAM modules, which can give full play to the role of the SimAM module in strengthening convolution. Figure 2 As shown. Each ConvBNReLU convolution module contains a 3×3 convolution kernel, batch normalization and ReLu activation function, forming a complete convolution operation. The SimAM module can infer 3-D attention weights for the feature maps in the layer without adding additional parameters to the original network, that is, directly inferring the attention weights of the spatial and channel dimensions of the feature maps, thereby enhancing the feature extraction capability and making the network pay more attention to some important areas, such as organs and tissues to be segmented. This processing process will focus on the key areas of the medical image to be segmented, so that the model will pay more attention to the organs and tissues to be segmented. The output is the processed feature map, which is convenient for subsequent other attention mechanisms to process the feature map. The relationship with the previous step is to train the data pre-processed in step 2. The relationship with the next step is to use the short-distance attention mechanism to enhance feature extraction, focus on key organs (organs to be segmented) to form feature maps of this scale, and facilitate downsampling.

[0075] Step 302: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input to this step is the feature map generated after step 301. Haar wavelet downsampling consists of two main components: a lossless feature encoding block that transforms features and reduces spatial resolution, and a feature representation learning block that extracts discriminative features. The feature map first enters the lossless feature encoding block and undergoes three steps: 1) Low-pass and high-pass filters are used to extract low-frequency and high-frequency information. 2) Two downsampling operations are performed, dividing the image into a low-frequency component A, a horizontal component H, a vertical component V, and a diagonal component D. 3) The resulting components are fused to form the final feature map. The Haar wavelet transformation increases the number of channels in the feature map by a factor of four, while reducing the resolution to H / 2 × W / 2. This means that information is converted from the spatial dimension to the channel dimension, thus avoiding information loss. The feature map then enters the feature representation learning block, which consists of a 1×1 convolutional layer, batch normalization, and a Reinforced Lu (ReLU) activation function, to form the final feature map. The result is a reduced-size feature map. This step extracts higher-level semantic features, providing multi-scale features for subsequent upsampling and skip connections. This is related to the previous step, which downsamples the feature map processed in step 301, reducing spatial resolution and computational complexity. It also relates to the subsequent step, which expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0076] Step 303: Further capture local key information from the medical image. The input to this step is the feature map processed in step 302. The processing and output are the same as in step 301. This step is related to the previous step in that it performs SimAM processing on the feature map processed in step 302 at a deeper level. It is also related to the subsequent step in that it facilitates downsampling.

[0077] Step 304: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 303. The processing and output are the same as those in step 302. This step is related to the previous step in that it downsamples the feature map processed in step 303, reducing spatial resolution and reducing computational complexity. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0078] Step 305: Further captures key local information from the medical image. After three SimAM processing passes (steps 301, 303, and this step), the feature map is now well focused on the organ to be segmented. The input to this step is the feature map processed in step 304. The processing and output are the same as in step 301. This step is related to the previous step in that the feature map processed in step 302 is further processed by SimAM. It also facilitates downsampling.

[0079] Step 306: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 305. The processing and output are the same as those in step 302. This step is related to the previous step in that it downsamples the feature map processed in step 305, reducing spatial resolution and computational complexity. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0080] Step 307: Local-Global Attention captures the global information in the medical image (the relationship between organs), and External Attention captures the relationship between medical images (the connection between images). Local-Global Attention and External Attention constitute the LGE module, such as Figure 4 The input of this step is the feature map formed after step 306, and the processing process is as follows:

[0081] like Figure 3 As shown, for the feature map X formed after step 306, we divide it into n windows (we set n = 4 in the experiment). In each window, the local attention calculates the self-affinity, thereby generating a weighted local feature map U local , see Equation 1 for details. In order to reduce the overhead in the global attention calculation process, local features represent the aggregate importance of key areas in the image.

[0082] U local =LA(X)(1)

[0083] Global Tokens captures the global feature information within each window, as shown in Formula 2. Here, we use lightweight and dynamic convolution (DLConv) instead of standard stride convolution for aggregation. DLConv combines global average pooling and lightweight linear transformation to efficiently aggregate and process input features. Through the global attention algorithm, the global feature map U based on global Tokens is Global can be calculated.

[0084] U global =GA(DLConv(U local ))(2)

[0085] The last stage starts with the global feature map U Global Upsample and then combine it with the local feature map U local Splicing to minimize information loss. Thus, the final feature map U of the entire image is calculated as shown in Formula 3 * .

[0086] U * =Concat(U local ,Upsample(U global )) (3)

[0087] The feature map formed through the above processing is not limited to the focus on local organs in step 301, but focuses more on the global information in a single image, that is, the relationship between organs.

[0088] After that, the feature map U * This module enters the External Attention portion of the LGE module. First, the input feature map is linearly projected onto the query matrix Q, the key matrix K, and the value matrix V. Next, the similarity between the query matrix Q and the external key memory unit Mk is calculated to obtain the attention map A. Finally, the input features are updated by multiplying the attention map A with the external value memory unit Mv, resulting in the final feature map, which is the output of this step. This feature map is then fed into a Local-GlobalAttention and an ExternalAttention module, and this process is repeated again. Finally, a ConvBNReLU convolutional module is used to obtain the final feature map. This step primarily allows the model to model the connections between images. Due to the presence of two external, learnable shared memory units, each image undergoing the external attention process can consistently process features belonging to the same category in previous images. This means that the two external memory units Mk and Mv can learn the most discriminative features in the entire dataset, thereby improving model performance. This step is related to the previous step in that it extracts global information from a single medical image (relationships between organs) and models connections between different images (image-to-image connections). The relationship with the back step is to facilitate downsampling.

[0089] Step 308: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 307. The processing and output are the same as those in step 302. This step is related to the previous step in that it downsamples the feature map processed in step 307, reducing spatial resolution and computational complexity. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0090] Step 309: The feature map is further processed using the LGE module, with the input being the feature map processed in step 308. The processing and output are the same as in step 307. This step is related to the previous step in that it extracts global information from a single medical image (relationships between organs) and models connections between different images (relationships between images) at a deeper level. It is also related to the subsequent step in that it facilitates downsampling.

[0091] Step 310: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 309. The processing and output are the same as those in step 302. This step is related to the previous step in that it downsamples the feature map processed in step 309, reducing spatial resolution and computational complexity. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0092] Step 311: The feature map will enter the bottleneck layer of the model, and the input is the feature map processed in step 310. The processing process is mostly the same as step 307, except that it does not need to pass through a layer of ConvBNReLU convolution module at the end, and focuses on the processing of two attention mechanisms to reduce redundant calculations. The bottleneck layer serves as the information compression stage of the encoder. The pure LGE module can more efficiently capture cross-regional and cross-scale dependencies and provide more global context guidance for the decoder. The relationship with the previous step is that the deepest network extracts the global information of a single medical image (the relationship between organs) and models the connection between different images (the connection between images), providing deep feature compression and multi-scale information aggregation for the model, and providing high semantic density and low redundancy feature representation for the subsequent decoding process. The relationship with the latter step is to facilitate upsampling input into the decoding network.

[0093] Step 312: Obtain the final segmentation result through the decoding network.

[0094] The feature map obtained after step 311 reaches the decoding network from the bottleneck of the model (i.e. the last layer). The input of this step is the output feature map of the bottleneck layer. The processing process is to gradually expand the spatial resolution of the feature map through deconvolution, and input the feature map into the module corresponding to the encoding network. In this process, each level concatenates (Concatenate) the feature map of the corresponding layer in the encoding network (with the same spatial resolution) with the upsampled feature map. Finally, the feature map is restored to its original size, processed by the segmentation head (1×1 convolution) and the Softmax activation function, and the network output is converted into a probability distribution to obtain the final multi-organ segmentation result. The relationship with the previous step is that the deep features of the bottleneck layer of the encoding network are concatenated. Figure 1 The image is restored to its original size and the segmentation result is output. The relationship with the latter step is to provide a medical image segmentation model that can be optimized with the collaboration of multiple attention mechanisms.

[0095] In general, steps 301 to 306 mainly use the short-range attention mechanism (SimAM) to focus on the local key information of a single image (such as important organs and tissues). Steps 307 to 311 mainly use the medium-range attention mechanism (Local-Global Attention) to capture the global information of a single image (the relationship between organs) and use the long-range attention mechanism (External Attention) to model the relationship between different images. Step 312 outputs the final segmentation result through the decoding network.

[0096] Step 4: Add the mixed expert model MoE to optimize the collaborative strategy between multiple attention mechanisms, and then optimize the medical image segmentation model in step 3, such as Figure 5 As shown in the figure. The relationship with the previous step is to optimize the multi-attention mechanism collaborative strategy of the hierarchical multi-scale medical image segmentation model. The relationship with the next step is to form the final multi-attention mechanism collaborative medical image segmentation model. The specific steps are as follows:

[0097] Step 401: Extract spatial features using partial convolution. The input for this step is the preprocessed medical image to be segmented. During the processing, the medical image enters the encoding network composed of the partial convolution module PConv. Specifically, the preprocessed image is divided into two branches. One branch is used for the final concatenation, while the other branch undergoes a 3×3 partial convolution with a 1×1 kernel. It then undergoes batch normalization and a ReLu activation function, and finally a 1×1 convolution with a kernel before being concatenated with the feature map from the previous branch. Convolution is performed on a portion of the input image rather than the entire image. This process reduces redundant computation and memory access. The output is the processed feature map, which facilitates subsequent processing by the multi-attention mechanism. This step is related to the previous step in that it trains on the preprocessed data from step 2. It is related to the subsequent step in that it facilitates downsampling.

[0098] Step 402: Haar wavelet downsampling avoids information loss during the downsampling process. The input of this step is the feature map formed after the processing of step 401. The processing process is to explicitly increase the number of channels of the feature map and use the Haar wavelet transform to reduce its resolution without losing information. The specific composition structure of the Haar wavelet downsampling refers to step 302. The processing result is a feature map with reduced size. This step can extract higher-level semantic features and provide multi-scale features for subsequent upsampling and jump connections. The relationship with the previous step is to downsample the feature map processed by step 401, reduce the spatial resolution, and reduce the amount of calculation. The relationship with the latter step is to expand the receptive field, laying the foundation for processing the feature map in a deeper network.

[0099] Step 403: Further extract deeper features using partial convolution. The input is the feature map processed in step 402, and the processing and output are the same as in step 401. This step is similar to the previous step in that it extracts features from the downsampled feature map in step 402. It also facilitates downsampling.

[0100] Step 404: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 403. The processing and output are the same as those in step 402. This step is related to the previous step in that it downsamples the feature map processed in step 403, reducing spatial resolution and computational complexity. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0101] Step 405: Further extract deeper features using partial convolution. The input is the feature map processed in step 404, and the processing and output are the same as in step 401. This step is related to the previous step in that it extracts features from the downsampled feature map in step 404. It is also related to the subsequent step in that it facilitates downsampling.

[0102] Step 406: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 405. The processing and output are the same as those in step 402. This step is related to the previous step in that it downsamples the feature map processed in step 405, reducing spatial resolution and computational complexity. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0103] Step 407: Design the AMOE module to improve the coordination strategy of multiple attention mechanisms, so that multiple attention mechanisms can be activated differently. This step effectively combines MoE and attention mechanisms, optimizes the coordination mechanism between multiple attention mechanisms, and enables more appropriate attention mechanisms to be activated more. Figure 6 As shown in Figure 4, the AMOE module contains two MoE layers, namely the attention head mixture of experts layer and the MLP mixture of experts layer. The input of this step is the feature map formed after step 402, and the processing process is as follows:

[0104] First, the feature map is split into two branches. The first branch, after passing through a layer normalization layer, enters the attention head mixture-of-experts layer. After processing through a gating network using the Top-k algorithm, three attention experts—SimAM, Local-Global Attention, and External Attention—are sparsely activated. Layer normalization ensures training stability and smooth gradient propagation. The gating network assigns different activation weights to different attention modules based on the distribution of input features. This process enables sparse dynamic selection of attention experts, allowing the model to employ the most appropriate attention strategy under different input conditions. The second branch is fused with the output of the first branch through a residual connection to form a new feature map. The feature map then splits into two more branches. The first branch, after passing through another layer normalization layer, enters the MLP mixture-of-experts layer. The input features first pass through a gating network, which dynamically selects and activates one or more experts based on the feature distribution of the input content. The activated MLP experts process the input and aggregate the results to form the output. Similarly, the second branch is fused with the output of the first branch through a residual connection to form a new feature map. This feature map is the one generated after processing with the attention mechanism most suitable for this image. After obtaining the feature map, the steps of the AMOE module are repeated, and then a layer of partial convolution module PConv is passed through to obtain the final feature map of this step. This step fully integrates the design principles of MoA with the construction requirements of the AMOE module, optimizing the synergy between the three attention mechanisms. This allows for more activation of the most appropriate attention mechanism for complex medical images. It is related to the previous step by sparsely activating multiple attention mechanisms on the downsampled feature map in step 406. It is related to the subsequent step by facilitating downsampling.

[0105] Step 408: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 407. The processing and output are the same as those in step 402. This step is related to the previous step in that it downsamples the feature map processed in step 407, reducing spatial resolution and reducing computational effort. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0106] Step 409: Further process the feature map using the AMOE module in a deeper network layer. The input of this step is the feature map processed in step 408. The processing and output are the same as in step 407. This step is related to the previous step in that it performs sparse activation of the multi-attention mechanism on the downsampled feature map in step 408. It is also related to the subsequent step in that it facilitates downsampling.

[0107] Step 410: Haar wavelet downsampling reduces the resolution of the feature map and avoids information loss during the downsampling process. The input of this step is the feature map processed in step 409. The processing and output are the same as those in step 402. This step is related to the previous step in that it downsamples the feature map processed in step 409, reducing spatial resolution and reducing computational complexity. It is related to the subsequent step in that it expands the receptive field, laying the foundation for feature map processing by deeper networks.

[0108] Step 411: In this step, the feature map enters the bottleneck layer of the model. The input is the feature map processed in step 410. The processing is largely the same as step 407, except that the final layer of partial convolution (PConv) is not required. Instead, the focus is on the attention mechanism, reducing redundant computation. This step is related to the previous step in that it performs sparse activation of multiple attention mechanisms on the downsampled feature map in step 410, providing the model with deep feature compression and multi-scale information aggregation, providing a high-density and low-redundancy feature representation for the subsequent decoding process. This step is related to the subsequent step in that it facilitates the upsampled input to the decoding network.

[0109] Step 412: Decode the network to obtain the final segmentation result.

[0110] The feature map obtained after step 411 will be upsampled to reach the decoding network. The input of this step is the output feature map of the bottleneck layer. The processing process is to gradually expand the spatial resolution of the feature map through deconvolution, and input the feature map into the module corresponding to the encoding network. In this process, each level will concatenate the feature map of the corresponding layer in the encoding network (with the same spatial resolution) with the upsampled feature map. Finally, the feature map is restored to its original size, and the final segmentation result is obtained after processing with the segmentation head (1x1 convolution) and the Softmax activation function. The relationship with the previous step is to concatenate the deep features of the bottleneck layer of the encoding network. Figure 1 Each step is restored to its original size and the segmentation result is output. The relationship with the subsequent steps forms the final multi-attention mechanism collaborative medical image segmentation model. So far, the present invention has successfully constructed a multi-attention mechanism collaborative medical image segmentation model.

[0111] In general, steps 401 to 406 mainly use partial convolution to extract features at low computational cost. Steps 407 to 411 mainly use the AMOE module to optimize the coordination mechanism between multiple attention mechanisms, so that the most appropriate attention mechanism is more activated. Step 412 outputs the final segmentation result through the decoding network.

[0112] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any person skilled in the art can utilize the methods and technical contents disclosed above to make many possible variations and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments with equivalent variations. Therefore, any equivalent variations made in accordance with the shape, structure, and principles of the present invention without departing from the content of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on multi-attention mechanism collaboration, characterized in that: The following steps are involved: Step 1: Obtain the multi-organ segmentation dataset and the automatic cardiac diagnosis challenge dataset to form a multi-organ imaging dataset; Step 2: Preprocess the multi-organ imaging dataset; Step 3. Construct a network structure for a medical image segmentation model based on hierarchical multi-scale attention. The network structure of the medical image segmentation model adopts an encoder-decoder structure, including five convolutional layers and one bottleneck layer. The first three layers of the five convolutional layers are convolutional modules with a short-range attention mechanism, which are used to strengthen convolution and enable the network structure to focus on local and important areas of the input image. The last two layers of the five convolutional layers are a mixture of convolutional modules and two layers of LGE modules, which are used to capture and model the global feature information of a single sample and the feature information between different samples. The LGE module consists of a medium-range attention mechanism module and a long-range attention mechanism module. Finally, the bottleneck layer is composed of an LGE module. Step 4: Add the hybrid expert model to optimize the collaborative strategy among the short-distance attention mechanism, medium-distance attention mechanism, and long-distance attention mechanism of the hierarchical multi-scale attention medical image segmentation model to form the final multi-attention mechanism collaborative medical image segmentation model.

2. The method according to claim 1, wherein In step 2, the images in the multi-organ imaging dataset are normalized, then standardized, the mean is subtracted, and the image size is unified. Finally, random flipping and random rotation are used to expand the multi-organ imaging dataset.

3. The method according to claim 1, wherein In step 3, the following steps are included: Step 301: The pre-processed multi-organ image dataset enters the fusion module of the short-range attention mechanism module and the convolution module for processing, and outputs the processed feature map; Step 302: Based on the feature map output in step 301, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map; Step 303: repeat step 301 and step 302, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 2; Step 304: Use the medium-range attention mechanism module to capture the global information in the medical image, that is, the relationship between organs, and use the long-range attention mechanism module to capture the relationship between medical images, that is, the connection between images. The process includes the following: Based on the feature map finally processed and outputted in step 303, it enters the mid-range attention mechanism module of the LGE module and is divided into n windows, where n≥4. In each window, the local attention calculates the self-affinity, thereby generating a weighted local feature map U local : U local =LA(X) Global Tokens captures the global feature information within each window, that is, through the global attention algorithm, a global feature map U based on global Tokens is obtained. Global : U Global =GA(DLConv(U locol )) For the global feature map U Global Upsample and combine it with the local feature map U local Splicing to minimize information loss and calculate the final feature map U of the entire image * : U * =Concat(U local ,Upsample(U Global )) After that, the feature map U * Entering the long-range attention mechanism module of the LGE module: First, the input feature map is linearly projected into the query matrix Q, the key matrix K, and the value matrix V; then, the similarity between the query matrix Q and the external key memory unit Mk is calculated to obtain the attention map A; finally, the input features are updated by multiplying the attention map A with the external value memory unit Mv to obtain the processed feature map; The obtained feature map repeatedly enters the medium-distance attention mechanism module and the long-distance attention mechanism module of the LGE module for processing, and then passes through the convolution module for convolution processing to obtain the processed feature map; Step 305: Based on the feature map output in step 304, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map; Step 306: repeat step 304 and step 305, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 1; Step 307: Based on the feature map output in step 306, the feature map is processed into the medium-range attention mechanism module and the long-range attention mechanism module of the LGE module to obtain a processed feature map without the need for convolution processing in the convolution module. Step 308: Based on the feature map output in step 307, the bottleneck layer of the model is upsampled to reach the decoding network, and the final segmentation result is obtained through the decoding network, thereby completing the construction of the network structure of the medical image segmentation model based on hierarchical multi-scale attention.

4. The method according to claim 3, wherein In step 301, the fusion module is a series connection of two ConvBNReLU convolution modules and a short-distance attention mechanism module, which gives full play to the role of the short-distance attention mechanism module in strengthening convolution. Each ConvBNReLU convolution module contains a 3×3 convolution kernel, batch normalization and ReLu activation function, forming a complete convolution operation. The short-distance attention mechanism module infers 3-D attention weights for the feature maps in the layer without adding additional parameters to the original network, that is, directly infers the spatial and channel dimension attention weights of the feature maps, thereby enhancing the feature extraction capability, focusing on the key areas of the medical image to be segmented, making the model pay more attention to the organ tissue to be segmented, and outputting the processed feature maps.

5. The method according to claim 3, wherein In step 302, the feature map first enters the lossless feature coding block and performs the following processing steps in sequence: using low-pass and high-pass filters to extract low-frequency and high-frequency information; performing two downsampling operations to divide the image into low-frequency components, horizontal components, vertical components and diagonal components; fusion to form a processed feature map; then, the feature map enters the feature representation learning block composed of a 1×1 convolution layer, batch normalization and ReLu activation function to form the final processed feature map.

6. The method according to claim 3, wherein In step 308, the feature map output by the bottleneck layer is deconvolved to expand the spatial resolution of the feature map, and the feature map is input into the module corresponding to the encoding network. During this period, each level splices the feature map of the corresponding layer in the encoding network with the upsampled feature map, and finally restores the feature map to its original size. After processing by the segmentation head and the Softmax activation function, the network output is converted into a probability distribution to obtain the final multi-organ segmentation result.

7. The method according to claim 1, wherein In step 4, the following steps are included: Step 401: The pre-processed multi-organ image dataset enters a coding network composed of partial convolution modules, extracts spatial features using partial convolution, and outputs a processed feature map. Step 402: Based on the feature map output in step 401, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map; Step 403: repeat step 401 and step 402, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 2; Step 404: Design an AMOE module to improve the coordination strategy among the three attention mechanisms: short-range attention mechanism, medium-range attention mechanism, and long-range attention mechanism. The AMOE module contains two MoE layers: the attention head mixture of experts layer and the MLP mixture of experts layer, including the following processing steps: Based on the feature map finally processed and outputted in step 403, first, the feature map is divided into two branches; First, after passing through the layer normalization layer, it enters the attention head mixed expert layer. After being processed by the gating network using the Top-k algorithm, three attention experts are sparsely activated, namely the short-range attention mechanism, the medium-range attention mechanism, and the long-range attention mechanism. Among them, layer normalization ensures training stability and smooth gradient propagation. The gating network assigns different activation weights to different attention modules based on the distribution of input features, realizing sparse dynamic selection of attention experts, allowing the model to use the most appropriate attention strategy under different input conditions; The second one is fused with the output of one branch through residual connection to form a new feature map; Then, the feature map is divided into two branches; After passing through the layer normalization layer again and again, it enters the MLP mixed expert layer. The input features first pass through the gating network, which dynamically selects and activates the most appropriate one or more experts based on the feature distribution of the input content; After the activated MLP experts process the input, they aggregate the results to form the output; Similarly, the second one will be fused with the output of one branch through residual connection to form a new feature map; After obtaining the feature map, the processing operation of the AMOE module is repeated again, and the final feature map is obtained after a layer of partial convolution module; Step 405: Based on the feature map output in step 404, perform Haar wavelet downsampling to reduce the resolution of the feature map and avoid information loss during the downsampling process, and output the processed feature map; Step 406: repeat step 404 and step 405, and the output of the previous step is the input of the next step, wherein the number of repetitions is at least 1; Step 407: Based on the feature map outputted in step 406, the processing operation of the AMOE module is repeated to obtain a processed feature map without the need for partial convolution processing by the partial convolution module. Step 408: Based on the feature map output in step 407, the bottleneck layer of the model is upsampled to the decoding network, and the final segmentation result is obtained through the decoding network, thereby completing the construction of the medical image segmentation model with multi-attention mechanism collaboration.

8. The method according to claim 7, wherein In step 401, the images of the preprocessed multi-organ imaging dataset are divided into two branches, one branch is used for the final splicing, and the other branch enters the partial convolution with a convolution kernel of 3×3, and then passes through a convolution with a convolution kernel of 1×1, and then passes through batch normalization and ReLu activation function, and finally passes through a convolution with a convolution kernel of 1×1 and is spliced ​​with the feature map of the first branch, performing the convolution operation on a partial area of ​​the input image rather than the entire image.

Images