Similarity memory prior network segmentation method and system for medical image segmentation

Abstract

The invention provides a similarity memory prior network segmentation method and system for medical image segmentation, and particularly relates to the technical field of image processing. A similarity memory prior network segmentation method for medical image segmentation comprises the following steps: (1) completing feature extraction through a first-stage encoder and a second-stage encoder, and extracting key features from a medical image; (2) fusing stage characteristics corresponding to the first-stage encoder and the second-stage encoder through a network, and transmitting the stage characteristics to a jump connection and decoder; (3) in jump connection, channel attention and space attention are used to fuse corresponding characteristics of a first-stage encoder and a second-stage encoder; and (4) decoding: using DMW-LA to extract deep semantic information. The method has the beneficial effects that key information in a medical image can be efficiently captured and utilized, so that higher accuracy and robustness are achieved in a segmentation task, the global relevance and internal consistency among features are improved, and the representation capability of a network is enhanced.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of image processing, and in particular to a similarity memory prior network segmentation method and system for medical image segmentation. Background Art

[0002] Artificial neural networks are computational models inspired by biological nervous systems. They solve complex problems by simulating the connections between neurons and how information is processed in the brain. With the rise of computer vision and deep learning, deep neural networks with multiple hidden layers have become the mainstream technology for handling complex pattern recognition tasks. The principle of computer vision is to simulate the functions of the human visual system, allowing computers to process, analyze and understand information in images and videos, and extract useful data or perform specific tasks from them. The visual system is of great significance to primates.

[0003] Some studies have found that some cells in the V1 area of ​​the primary visual cortex of macaques can recognize visual inputs with complex shapes. This makes us think about the inspiration of the discovery of these "grandmother cells" for medical image segmentation technology. First, we believe that these cells can be used as prototype memory priors to guide the network to explore more complex and multi-dimensional distributed features during the process of medical image segmentation. Second, these prototype memory priors are learned by online dynamic clustering based on loss difference and weights, and are effectively utilized through a memory-based attention mechanism.

[0004] After the visual stimulus is received by the retina, it will first be projected to the primary visual cortex V1 area in the cerebral cortex, where there are a large number of neurons that can perceive visual shapes. Studies have found that simple cells and complex cells in the primary visual cortex V1 area can encode the orientation information of straight-line visual signals. Another type of super-complex cells can encode the orientation information and length information of the straight line at the same time. Shapes that are more complex than straight lines, such as arcs, broken lines, and crosses, are usually further transmitted to higher-level visual cortices before they are represented. However, recently, researchers used two-photon calcium imaging technology to discover that there are arc-recognition neurons in the primary visual cortex V1 area of ​​macaques that specifically respond to arc-shaped visual stimuli, confirming that the encoding selection of some neurons in the visual cortex V1 area is highly specific. So can this specificity also be applied to neural networks?

[0005] The answer is yes. These cells in the V1 area can be regarded as a special type of feature filter. This filter can show extremely high sensitivity and specificity to specific stimuli. And this type of filter can perform fine analysis of the input, identify and respond to specific features in the image, such as boundaries, textures, categories and other information. In this way, a multi-level feature extraction mechanism can be constructed to transform the original visual data into a more abstract and meaningful representation, which can help further advanced visual processing and recognition functions. The author calls these rare feature-sensing cells in the V1 area that show strong and specific responses to specific visual patterns "grandmother cells" to emphasize their explicit encoding of specific prototypes. In addition, the author believes that these "grandmother cells" can serve as "prototype memories" before the image generation process to guide the current network operation. These "prototype memories" can be regarded as prior knowledge, which helps the network to transcend existing time and space limitations when synthesizing images. Therefore, in the memory attention of image generation, "grandmother cells" play a pioneering role in the conceptual framework. However, in the field of medical image segmentation, the application of this characteristic of the primary visual cortex V1 area has not been explored yet. Therefore, the present invention provides a similarity memory prior network segmentation method and system for medical image segmentation. Summary of the invention

[0006] In order to overcome the deficiencies of the prior art, the present invention provides a similarity memory prior network segmentation method and system for medical image segmentation.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A similarity memory prior network segmentation method for medical image segmentation comprises the following steps:

[0009] (1) Feature extraction is completed through the first-stage encoder and the second-stage encoder to extract key features from medical images;

[0010] (2) The network fuses the stage features corresponding to the first-level encoder and the second-level encoder and passes them to the skip connection and decoder;

[0011] (3) In the skip connection, channel attention and spatial attention are used to fuse the corresponding features of the first-level encoder and the second-level encoder;

[0012] (4) Decoding uses DMW-LA to extract deep semantic information and uses bilinear interpolation to restore the image to the same resolution as the input image to achieve pixel-level segmentation.

[0013] Optionally, in step 1, the first-level encoder consists of four levels of DMW-LA, and DMW-LA and DS-GIM are alternately stacked at the third level, DMW-LA is a dynamic memory weight loss attention module, DS-GIM is a dual similarity global internal enhancement module, DMW-LA extracts key features from medical images, and inputs X∈RH×w×c, where H, W and C refer to the length, width and channel dimensions of the input image, respectively. X is linearly transformed with weights W4Wk and W to generate Query, Key and Value for self-attention calculation, and the output attention result is X'.

[0014] Optionally, in step 1, K-means is performed on the self-attention result X' to fill the memory bank through grid training. The generated semantic clusters are called prototype prior units, and the center point of each cluster is called a semantic core unit. The prototype prior unit can be expressed as {{E11, E12, ..., E1c}, ..., {Ek1, Ek2, ..., Ekc}}:

[0015] The semantic core unit can be represented as nE1, E2, E3, …, Ek.

[0016] After the feature memory is completed, the prior knowledge is supplemented for network training. First, the cosine similarity between the query and the semantic core unit is calculated to match the prototype prior unit corresponding to xthe. Assuming that there is Xi∈RN×C,N≤H×WEi=Ei1,Ei2,…,Eic|EiT∈RC×M matching the prototype prior unit in the Query, then Ei will self-focus on Xi. This step simulates the process of "grandmother cells" recognizing specific visual stimuli. Finally, the semantic features of the high-level category are added to the global features calculated by self-attention, and then mapped to the original feature space through the convolution operation ψ();

[0017] After the memory bank is initialized, a W-LD update strategy is proposed to dynamically update the memory bank. Assuming that Xi∈RN×c in X∈pH×w×c matches the prototype prior unit E1∈RM×C, the Softmax activation function is used to calculate the weights of Xi and E respectively, and sort them according to the weight of each column. M represents the feature dimension of the prototype prior unit.

[0018] Optionally, the Dual Similarity Global Internal Enhancement Module (DS-GIM) assumes that the input size is X∈RH×W×c and performs division:

[0019] o X becomes

[0020] P×P windows are used to calculate the internal cosine similarity of these windows. Different decay coefficients are assigned to the windows according to the number of layers of the network at the current stage. The purpose is to use the decay coefficient to increase the difference between windows. The calculation formula of the decay coefficient is as follows.

[0021]

[0022] In the above formula, 1 represents the total number of layers in the current stage, n = {0, 1, 2, ..., 1-1}, and the attenuation coefficient of each layer in the current stage of the network is obtained from the above equation. The windows are connected in series to form a new feature representation, and self-attention is performed to obtain the attention matrix X1∈RHW×Hw. The columns of X1 are averaged to obtain Xn∈RHW×1, and then X is copied to obtain a new matrix S∈RHW×HW. At this time, the elements in S can be regarded as a set of spatial positioning elements. The Euclidean distance between elements is calculated using the formula Euc=S-ST to measure their proximity in the feature space. According to Euc, the internal elements in X1 are divided into two parts, and the two parts are calculated separately by the Softmax function. Finally, the two parts are spliced ​​together to form a new matrix and multiplied with Value.

[0023] Optionally, in step 2, the decoder uses a four-stage DMW-LA to mine and preserve deep semantic information. In the skip connection, channel attention and spatial attention are used to fuse corresponding features from the encoder and decoder, and the fused features are added to the decoder.

[0024] A similarity memory prior network system for medical image segmentation includes a first-level encoder, a second-level encoder, a decoder and a skip connection, wherein the first-level encoder consists of four-level DMW-LA, and DMW-LA and DS-GIM are alternately stacked at the third level, DMW-LA is a dynamic memory weight loss attention module, the first-level encoder has a dynamic memory weight loss attention, DS-GIM is a dual similarity global internal enhancement module, the second encoder consists of a four-level visual backbone MaxViT, and the decoder consists of DMW-LA. After the encoder completes feature extraction, the network will fuse the stage features corresponding to the two encoders and pass them to the skip connection and the decoder. In the skip connection, channel attention and spatial attention are used to fuse the corresponding features of the encoder and decoder. The decoder also uses DMW-LA to extract deep semantic information and uses bilinear interpolation to restore the image to the same resolution as the input image to achieve pixel-level segmentation.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] 1. Inspired by the information processing of primate visual systems, the present invention designs Dynamic-Memory Concept Attention (DMW-LA). DMW-LA uses the prototype memory library to supplement the information in the time dimension and dynamically adjusts the semantic cluster features therein, so as to efficiently capture and utilize the key information in medical images, thereby achieving higher accuracy and robustness in the segmentation task.

[0027] 2. A dual similarity global internal enhancement module (DS-GIM) is designed to enhance the depth and breadth of network feature extraction. DS-GIM is based on two metrics, cosine similarity and Euclidean distance, to improve the global correlation and internal consistency between features, thereby enhancing the representation ability of the network.

[0028] 3. We designed a novel similarity memory prior network specifically for medical image segmentation tasks. Experimental results on three public datasets show that Sim-MPNet can fully integrate and utilize information in the temporal and spatial dimensions to improve segmentation accuracy. Moreover, the network imitates the cognitive process of the primate visual system, providing a new perspective for the design of medical image segmentation networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0030] Figure 2 It is a schematic diagram of the overall structure of DMW-LA and W-LD update strategy of the present invention;

[0031] Figure 3 It is a schematic diagram of the K change curve of the network training process of the present invention;

[0032] Figure 4 It is a schematic diagram of the overall structure of the DS-GIM of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the application.

[0034] like Figure 1-4 As shown, a similarity memory prior network segmentation method for medical image segmentation includes the following steps:

[0035] (1) Feature extraction is completed through the first-stage encoder and the second-stage encoder to extract key features from medical images;

[0036] (2) The network fuses the stage features corresponding to the first-level encoder and the second-level encoder and passes them to the skip connection and decoder;

[0037] (3) In the skip connection, channel attention and spatial attention are used to fuse the corresponding features of the first-level encoder and the second-level encoder;

[0038] (4) Decoding uses DMW-LA to extract deep semantic information and uses bilinear interpolation to restore the image to the same resolution as the input image to achieve pixel-level segmentation.

[0039] In step 1, the first-level encoder consists of four levels of DMW-LA, and DMW-LA and DS-GIM are alternately stacked in the third level. DMW-LA is a dynamic memory weight loss attention module, and DS-GIM is a dual-similarity global internal enhancement module. DMW-LA extracts key features from medical images. The input X∈RH×w×c, where H, W and C refer to the length, width and channel dimensions of the input image, respectively, is linearly transformed with weights W4Wk and W to generate Query, Key and Value for self-attention calculation, and the output attention result is X'.

[0040] In step 1, through grid training, K-means is performed on the result X' of self-attention to fill the memory bank. The generated semantic clusters are called prototype prior units, and the center point of each cluster is called a semantic core unit. The prototype prior unit can be expressed as {{E11, E12, ..., E1c}, ..., {Ek1, Ek2, ..., Ekc}}, and the semantic core unit can be expressed as nE1, E2, E3, ..., Ek;

[0041] After the feature memory is completed, the prior knowledge is supplemented for network training. First, the cosine similarity between the query and the semantic core unit is calculated to match the prototype prior unit corresponding to xthe. Assuming that there is Xi∈RN×C,N≤H×WEi=Ei1,Ei2,…,Eic|EiT∈RC×M matching the prototype prior unit in the Query, then Ei will self-focus on Xi. This step simulates the process of "grandmother cells" recognizing specific visual stimuli. Finally, the semantic features of the high-level category are added to the global features calculated by self-attention, and then mapped to the original feature space through the convolution operation ψ();

[0042] After the memory bank is initialized, it still needs to be continuously updated to ensure that it can adapt to complex medical images. A W-LD update strategy is proposed to dynamically update the memory bank. Assume that Xi ∈ RN×c in X ∈ pH×w×c matches the prototype prior unit E1 ∈ RM×C. The Softmax activation function is used to calculate the weights of Xi and E respectively, and the weights are sorted according to each column. M represents the feature dimension of the prototype prior unit. In addition, set M, 4 < k < 3m to avoid K falling into extreme cases and ensure the effective update of the prototype memory bank. The columns with larger weights in Xi contain more semantic information, while the columns with smaller weights in E contain less semantic information.

[0043] Dual Similarity Global Internal Enhancement Module (DS-GIM). Assume the _input size is X ∈ RH×W×c, and perform division:

[0044] o X is transformed into

[0045] a P×P window, calculate the internal cosine similarity of these windows, and assign different decay coefficients to the windows according to the number of layers of the network at the current stage. The purpose is to use the decay coefficient to increase the difference between the windows. The calculation formula of the decay coefficient is as follows.

[0046]

[0047] In the above formula, 1 represents the total number of layers at the current stage, n = {0, 1, 2,..., 1 - 1}. The decay coefficients of each layer of the network at the current stage are obtained from the above equation. The windows are concatenated to form a new feature representation, and self-attention is performed. After obtaining the attention matrix X1 ∈ RHW×Hw, the columns of X1 are averaged to obtain Xn ∈ RHW×1, and then X is copied to obtain a new matrix S ∈ RHW×HW. At this time, the elements in S can be regarded as a set of spatial positioning elements. The Euclidean distance between the elements is calculated by the formula Euc = S - ST to measure their closeness in the feature space. According to Euc, the internal elements in X1 are divided into two parts, and the two parts are calculated through the Softmax function respectively. Finally, the two parts are concatenated together to form a new matrix and multiplied by Value.

[0048] In step 2, the decoder uses a four-stage DMW-LA to mine and save deep semantic information. In the skip connection, channel attention and spatial attention are used to fuse the corresponding features from the encoder and the decoder, and the fused features are added to the decoder.

[0049] A similarity memory prior network system for medical image segmentation includes a first-level encoder, a second-level encoder, a decoder and a skip connection. The first-level encoder consists of four-level DMW-LA. DMW-LA and DS-GIM are alternately stacked at the third level. DMW-LA is a dynamic memory weight loss attention module. The first-level encoder has a dynamic memory weight loss attention. DS-GIM is a dual-similarity global internal enhancement module. The second encoder consists of a four-level visual backbone MaxViT. The decoder consists of DMW-LA. After the encoder completes feature extraction, the network will fuse the stage features corresponding to the two encoders and pass them to the skip connection and the decoder. In the skip connection, channel attention and spatial attention are used to fuse the corresponding features of the encoder and decoder. The decoder also uses DMW-LA to extract deep semantic information and uses bilinear interpolation to restore the image to the same resolution as the input image to achieve pixel-level segmentation.

[0050] Extensive experiments were conducted on four publicly available datasets: the Automatic Cardiac Diagnosis Challenge (ACDC, 4 categories) (Bernard et al., 2018), Segmentation of Multiple Osteoma Plasma Cells in Microscopic Images (SegPC-2021, 3 categories) (Gupta et al., 2020), the International Skin Imaging Collaboration 2018 Challenge (ISIC-2018, 2 categories) (Codella et al., 2019), and the MICCAI2015 Multi-Atlas Abdominal Labeling Challenge (Synapse, 9 categories). (1) The ACDC dataset consists of magnetic resonance imaging (MRI) images of the left ventricle (LV), right ventricle (RV), and myocardium (Myo). The training and test set divisions are consistent with TransUNe,t (Chen et al. 2021). (2) The SegPC-2021 dataset consists of myeloma plasma cell images. The segmentation targets are the cytoplasm and nucleus of the cells, and the training and test sets are segmented in a 4:1 ratio. (3) The ISIC-2018 dataset consists of dermoscopic images. The training set and test set are divided according to EMCAD (Rahman et al. 2024). (4) The synaptic dataset consists of axial clinical CT images of the abdomen. Each image contains part or all of the eight abdominal organs (aorta, gallbladder, left kidney, right kidney, liver, pancreas, spleen, and stomach). The training set and test set are divided according to TransUNet (Chen et al. 2021).

[0051] The present invention is compared with other state-of-the-art methods, including CNN-based methods: U-Net (Ronneberger et al., 2015), AT-UNET (Oktay et al., 2018), U-Net++ (Zhou et al., 2018), and EMCAD (Rahman et al., 2024); VIT-based methods: SwinUNet (Cao et al., 2022), TransDeepLab (Azad et al., 2022), and DAE-Former (Azad et al., 2023); and hybrid CNN-ViT methods TransUNet (Chen et al., 2021), MixTrans (Wan et al., 2022), TransClawUNet (Yao et al., 2022), TransAscade (Rahman and Marculescu 2023), HiFormer (Heidari et al., 2023), and PAG-TransYNet (Bougourzi et al., 2024).

[0052] In the experiments, the input resolutions of the two encoders are set to 224×224 and 256×256, respectively. In addition, we use the AdamW optimizer with a learning rate of 0.0001 and a weight decay of 1e-4 to train the model. The model is trained on a single RTX3090 GPU with a batch size of 20 and trained for 300 epochs using Dice (lice) and cross entropy (Lce) losses. The specific ratio of these two losses is Loss = 0.7-Ldce+0.3-L. Finally, all experiments were performed in Python 3.10.0 and Torch 2.3.0 environments. Only random rotation and flipping are used as data augmentation. We choose Dice score (DSC) and 95% Hausdorff distance (HD95) as evaluation metrics. HD95 represents the 95th percentile distance between the boundary of the model prediction map and the actual label.

[0053] The ACDC dataset tests the model's ability to explore local details, and VIT-based methods usually have difficulty in obtaining better segmentation results. Although the CNN-ViT hybrid method partially makes up for the shortcomings of ViT, its highest DSC is only 91.48%. In contrast, the present invention can directly capture the category characteristics of organs in ACDC through a dynamic memory prior mechanism, achieving a DSC of 92.18% and obtaining better segmentation results. The present invention outperforms existing methods and achieves a new SOTA on SegPC-2021.

Claims

1. A similarity memory prior network segmentation method for medical image segmentation, characterized in that: The following steps are involved: (1) Feature extraction is completed through the first-stage encoder and the second-stage encoder to extract key features from medical images; (2) The network fuses the stage features corresponding to the first-level encoder and the second-level encoder and passes them to the skip connection and decoder; (3) In the skip connection, channel attention and spatial attention are used to fuse the corresponding features of the first-level encoder and the second-level encoder; (4) Decoding uses DMW-LA to extract deep semantic information and uses bilinear interpolation to restore the image to the same resolution as the input image to achieve pixel-level segmentation.

2. According to claim 1, a similarity memory prior network segmentation method for medical image segmentation, in the step 1, the first-level encoder is composed of four-level DMW-LA, and DMW-LA and DS-GIM are alternately stacked at the third level, DMW-LA is a dynamic memory weight loss attention module, DS-GIM is a dual similarity global internal enhancement module, DMW-LA extracts key features from medical images, inputs X∈RH×w×c, where H, W and C refer to the length, width and channel dimension of the input image respectively, linearly transforms X with weights W4Wk and W, generates Query, Key and Value for self-attention calculation, and outputs the attention result as X'.

3. According to the similarity memory prior network segmentation method for medical image segmentation in claim 2, in the step 1, the self-attention result X' is subjected to K-means to fill the memory bank through grid training, and the generated semantic clusters are called prototype prior units, and the center point of each cluster is called a semantic core unit. The prototype prior unit can be expressed as {{E11, E12, ..., E1c}, ..., {Ek1, Ek2, ..., Ekc}}, and the semantic core unit can be expressed as nE1, E2, E3, ..., Ek; After the feature memory is completed, the prior knowledge is supplemented for network training. First, the cosine similarity between the query and the semantic core unit is calculated to match the prototype prior unit corresponding to xthe. Assuming that there is Xi∈RN×C,N≤H×WEi=Ei1,Ei2,…,Eic|EiT∈RC×M matching the prototype prior unit in the Query, then Ei will self-focus on Xi. This step simulates the process of "grandmother cells" recognizing specific visual stimuli. Finally, the semantic features of the high-level category are added to the global features calculated by self-attention, and then mapped to the original feature space through the convolution operation ψ(); After the memory bank is initialized, a W-LD update strategy is proposed to dynamically update the memory bank. Assuming that Xi∈RN×c in X∈pH×w×c matches the prototype prior unit E1∈RM×C, the Softmax activation function is used to calculate the weights of Xi and E respectively, and sort them according to the weight of each column. M represents the feature dimension of the prototype prior unit.

4. The similarity memory prior network segmentation method for medical image segmentation according to claim 3 is characterized in that: Dual Similarity Global Internal Enhancement Module (DS-GIM), assuming that the input size is X∈RH×W×c, performs division: o X becomes P×P windows are used to calculate the internal cosine similarity of these windows. Different decay coefficients are assigned to the windows according to the number of layers of the network at the current stage. The purpose is to use the decay coefficient to increase the difference between windows. The calculation formula of the decay coefficient is as follows. In the above formula, 1 represents the total number of layers in the current stage, n = {0, 1, 2, ..., 1-1}, and the attenuation coefficient of each layer in the current stage of the network is obtained from the above equation. The windows are connected in series to form a new feature representation, and self-attention is performed to obtain the attention matrix X1∈RHW×Hw. The columns of X1 are averaged to obtain Xn∈RHW×1, and then X is copied to obtain a new matrix S∈RHW×HW. At this time, the elements in S can be regarded as a set of spatial positioning elements. The Euclidean distance between elements is calculated using the formula Euc=S-ST to measure their proximity in the feature space. According to Euc, the internal elements in X1 are divided into two parts, and the two parts are calculated separately by the Softmax function. Finally, the two parts are spliced ​​together to form a new matrix and multiplied with Value.

5. According to the similarity memory prior network segmentation method for medical image segmentation in claim 1, in step 2, the decoder uses a four-stage DMW-LA to mine and preserve deep semantic information, and in skip connection, channel attention and spatial attention are used to fuse corresponding features from the encoder and decoder, and the fused features are added to the decoder.

6. A similarity memory prior network system for medical image segmentation, characterized in that: It includes a first-level encoder, a second-level encoder, a decoder and a skip connection. The first-level encoder consists of four levels of DMW-LA. DMW-LA and DS-GIM are alternately stacked at the third level. DMW-LA is a dynamic memory weight loss attention module. The first-level encoder has dynamic memory weight loss attention. DS-GIM is a dual similarity global internal enhancement module. The second encoder consists of a four-level visual backbone MaxViT. The decoder consists of DMW-LA. After the encoder completes feature extraction, the network will fuse the stage features corresponding to the two encoders and pass them to the skip connection and the decoder. In the skip connection, channel attention and spatial attention are used to fuse the corresponding features of the encoder and decoder. The decoder also uses DMW-LA to extract deep semantic information and uses bilinear interpolation to restore the image to the same resolution as the input image to achieve pixel-level segmentation.

Images