MRI image segmentation system and method based on hybrid attention supervision U-type network

Through the MRI image segmentation system based on a hybrid attention-supervised U-shaped network, the problem of lesion region segmentation in multimodal MRI images is solved, and the diagnostic efficiency and accuracy are improved.

CN118587442BActive Publication Date: 2024-11-29BEIJING HUIZHONGJUCHENG ENTERPRISE SERVICE CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202411067389.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-29
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

The prior art is difficult to effectively analyze and segment the lesion area in multimodal MRI images, resulting in time-consuming diagnosis and risk of misdiagnosis and misdiagnosis.

Method used

A MRI image segmentation system based on hybrid attention-supervised U-shaped network is proposed. Through image preprocessing, multimodal MRI image segmentation model training and image generation module, combined with convolutional division module, local context fusion Transformer block and global Transformer block, extracted Multi-scale features and generate segmentation masks.

Benefits of technology

The quality of multimodal MRI image segmentation is significantly improved, the clinical diagnosis process is simplified, and the doctor's ability to diagnose diseases is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118587442B_ABST
    Figure CN118587442B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of medical image processing technology, and provides a system and method for MRI image segmentation based on a hybrid attention supervised U-type network, wherein the system comprises: an image preprocessing module, a multimodal MRI image segmentation model training module and an image generation module; the image preprocessing module is used to preprocess multimodal MRI images, and stack the preprocessed MRI images in the channel direction to obtain multimodal MRI images; the multimodal MRI image segmentation model training module is used to train a hybrid attention supervised U-type network based on multimodal MRI images to obtain a multimodal MRI image segmentation model; the image generation module is used to input the actual multimodal MRI image into the multimodal MRI image segmentation model to generate a segmentation mask. The present invention can significantly improve the quality of multimodal MRI image segmentation, simplify the process of clinical diagnosis and improve the ability of doctors to diagnose diseases.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a system and method for MRI image segmentation based on a hybrid attention-supervised U-type network. Background Art

[0002] Determining the lesion area by analyzing medical images is an important technical means for disease diagnosis and treatment. Magnetic resonance imaging (MRI) technology is essential for accurate diagnosis and treatment of patients because of its non-radioactive, harmless to the human body, excellent performance in soft tissue imaging, and ability to provide high-resolution spatial information. In the process of disease diagnosis and treatment, MRI images with multiple modalities are usually used. They each have unique characteristics and can complement each other's information, providing strong support for accurate analysis of lesion areas in clinical practice. However, in actual clinical processes, the analysis of lesion areas in MRI images still mainly relies on the subjective experience of doctors. Due to the uneven distribution of voxels in different areas of MRI images and the unclear boundary between the region of interest and the surrounding tissue, manual analysis of medical images requires a lot of effort and time, and there is still a potential risk of misdiagnosis and missed diagnosis. Therefore, by introducing the idea of ​​image segmentation, it is very important to design a model that can automatically analyze and segment the specified target area based on multimodal MRI images. Deep learning neural networks can automatically learn complex feature representations from raw input data by building multi-layer neural networks to simulate the learning process of the brain, without the need for experts to manually edit feature extractors based on professional knowledge. In medical image processing, deep learning has also shown great potential.

[0003] The current mainstream deep learning network captures the local features of the image through convolution, and uses convolutional attention to assign differential weights to different pixels of the image, helping the model focus on more core features while effectively filtering out insignificant redundant features and noise interference. This is important when processing MRI feature maps that contain rich structural details, because the region of interest that needs to be analyzed usually only occupies a small part of the medical image. However, when the convolutional model performs image processing, it can often only focus on the local area within the size of the convolution kernel, and cannot directly capture the global context information. The self-attention mechanism allows the model to focus on different positions in the sequence when processing sequence data and capture the relationship between them. In the image processing process, self-attention helps the model understand the global structure of the image by decomposing the image into a series of small image patches and then calculating the attention correlation between these small image patches. Although the original Transformer model performs well in capturing global information, it is poor in processing local details.

[0004] In summary, there is an urgent need to provide an MRI image segmentation method that can fully mine the local and global potential representations between MRI images, accurately detect the edges of lesion areas in MRI images, and thus significantly improve the segmentation quality of multimodal MRI images. Summary of the invention

[0005] Based on this, the present invention proposes a system and method for MRI image segmentation based on a hybrid attention supervised U-type network to solve the technical defects of the prior art. The system and method can fully explore the potential representations between multimodal MRI images, significantly improve the quality of multimodal MRI image segmentation, further simplify the clinical diagnosis process and improve the doctor's ability to diagnose diseases.

[0006] In a first aspect, the present invention provides a hybrid attention-supervised U-type network MRI image segmentation system, the system comprising: an image preprocessing module, a multimodal MRI image segmentation model training module and an image generation module, wherein:

[0007] The image preprocessing module is used to preprocess the MRI images of four modes, namely, T1, T1CE, T2 and T2FLAIR, and stack the preprocessed MRI images of the four modes in the channel direction to obtain a multi-modal MRI image;

[0008] The multimodal MRI image segmentation model training module is used to train a hybrid attention supervised U-type network based on the multimodal MRI image to obtain a multimodal MRI image segmentation model;

[0009] The image generation module is used to input the actual multimodal MRI image into the multimodal MRI image segmentation model to generate a segmentation mask.

[0010] In a possible implementation, the multimodal MRI image segmentation model training module includes an encoder, a decoder and a skip connection, wherein:

[0011] The encoder is used to downsample the multimodal MRI image layer by layer, extract multi-scale features at different resolutions, and use the multi-scale features as input to the decoder;

[0012] The decoder is used to fuse the multi-scale features to generate a segmentation mask;

[0013] The jump connection is used to fuse the features between the encoder and the decoder to reduce the semantic difference between shallow features and deep features.

[0014] In a possible implementation, the encoder includes a convolutional partitioning module and four encoding modules connected in sequence, wherein:

[0015] The convolution partitioning module includes a first three-dimensional convolution and a first layer of normalization connected in sequence, and the convolution kernel and stride of the first three-dimensional convolution are both 2;

[0016] The first encoding module includes a first local context fusion Transformer block and a first three-dimensional image fusion module connected in sequence; the second encoding module includes a second local context fusion Transformer block and a second three-dimensional image fusion module connected in sequence; the third encoding module includes a first global Transformer block and a third three-dimensional image fusion module connected in sequence; the fourth encoding module includes a second global Transformer block and a fourth three-dimensional image fusion module connected in sequence;

[0017] The first local context fusion Transformer block and the second local context fusion Transformer block are each composed of a second layer of normalization, a three-dimensional multi-head neighborhood self-attention, a third layer of normalization, a first multi-layer perceptron, a fourth layer of normalization, a three-dimensional multi-head void neighborhood self-attention, a fifth layer of normalization, and a second multi-layer perceptron connected in sequence;

[0018] The output features of the first local context fusion Transformer block and the second local context fusion Transformer block The calculation formula is as follows:

[0019] ,

[0020] ,

[0021] ,

[0022] ,

[0023] in, LN For layer normalization, MNSA is the three-dimensional multi-head neighborhood self-attention, is the input feature, MLP is a multi-layer perceptron, MDNSA is the multi-headed empty neighborhood self-attention, , , All are intermediate features. l The sequence number of the feature.

[0024] In a possible implementation, the three-dimensional multi-head neighborhood self-attention includes a first linear layer and a first softmax weighted layer connected in sequence, and the neighborhood self-attention score of the i-th image patch is The calculation formula is:

[0025] ,

[0026] ,

[0027] in, is the value matrix; is the transposed matrix of the key matrix Key of the dth neighborhood image patch centered on the ith image patch; softmax is the softmax weight; d is the size of the neighborhood range of the i-th image patch; is the attention weight of the i-th image patch, The calculation formula is:

[0028] ,

[0029] Among them, Q is the query vector matrix Query of the i-th image patch, is the transposed matrix of the key matrix Key of the dth neighborhood image patch centered on the ith image patch, is the dth neighborhood image patch centered on the ith image patch, Query the query matrix for the i-th image patch and the positional relationship between the d-th neighborhood image patch.

[0030] In a possible implementation, the three-dimensional multi-headed hole neighborhood self-attention includes a second linear layer and a second softmax weighting connected in sequence, and the neighborhood expansion self-attention score of the i-th image patch is The calculation formula is:

[0031] ,

[0032] ,

[0033] in, To expand the value matrix; is the transposed matrix of the value matrix Value of the dth neighborhood image block that is not filled with holes and is centered on the ith image block; softmax is the softmax weight; d is the size of the neighborhood range of the i-th image patch; is the attention weight of the i-th image patch among the N image patches with r holes between them in the input features, The calculation formula is:

[0034] ,

[0035] Among them, Q is the query vector matrix Query of the i-th image patch, is the transposed matrix of the key matrix Key of the d-th hole neighborhood image patch centered on the i-th image patch, is the dth neighborhood image patch that is not filled with holes and is centered on the ith image patch. The query matrix of the i-th image patch is queried for the positional relationship between the d-th hole neighborhood image patch, and r is the distance between every two hole neighborhood image patches.

[0036] In a possible implementation, the first global Transformer block and the second global Transformer block are each composed of a sixth layer of normalization, a first three-dimensional multi-head self-attention, a seventh layer of normalization, a third multi-layer perceptron, an eighth layer of normalization, a second three-dimensional multi-head self-attention, a ninth layer of normalization, and a fourth multi-layer perceptron, which are connected in sequence;

[0037] Output features of the first global Transformer block and the second global Transformer block The calculation formula is:

[0038] ,

[0039] ,

[0040] ,

[0041] ,

[0042] in, LN For layer normalization, MSA is three-dimensional multi-head self-attention, is the input feature, MLP is a multi-layer perceptron, , and All are intermediate features. l The sequence number of the feature.

[0043] In a possible implementation, the first 3D image fusion module, the second 3D image fusion module, the third 3D image fusion module and the fourth 3D image fusion module all include sequentially connected feature groups and a third linear layer.

[0044] In a possible implementation, the decoder includes four different levels of decoding modules connected in sequence, a segmentation mask output module and a layer-by-layer supervision loss, and each level of the decoding module includes a feature focusing module, a feature restoration module, a first feature refinement block and a size adjustment block connected in sequence;

[0045] The feature focusing module includes three-dimensional channel attention and spatial attention connected in sequence; the channel attention includes a first 3DMaxPooling, a first 3DAvgPooling, a fifth multilayer perceptron and a first Sigmoid weighting, the output ends of the first 3DMaxPooling and the first 3DAvgPooling are both connected to the input end of the fifth multilayer perceptron, and the output end of the fifth multilayer perceptron is connected to the input end of the first Sigmoid weighting; the spatial attention includes a second 3DMaxPooling, a second 3DAvgPooling, a 3D convolution and a second Sigmoid weighting, the output ends of the second 3DMaxPooling and the second 3DAvgPooling are both connected to the input end of the 3D convolution, and the output end of the 3D convolution is connected to the input end of the second Sigmoid weighting;

[0046] The feature restoration module includes a three-dimensional transposed convolution, a tenth layer of normalization, and a first relu activation function connected in sequence;

[0047] The first feature refinement block includes a second three-dimensional convolution, an eleventh layer of standardization, a second relu activation function and a residual connection; the second three-dimensional convolution, the eleventh layer of standardization and the second relu activation function are connected in sequence, and the two ends of the residual connection are respectively connected to the input end of the second three-dimensional convolution and the output end of the second relu activation function, so as to splice the input feature map of the second three-dimensional convolution with the output feature map of the second relu activation function in the channel direction;

[0048] The resizing block includes a 1×1×1 convolutional layer and three linear upsampling layers connected in sequence.

[0049] In one possible implementation, the segmentation mask output module includes a second feature refinement block and a third feature refinement block connected in sequence; the input feature of the second feature refinement block is a multimodal MRI image, the output feature of the second feature refinement block is spliced ​​with the output feature of the fourth-level decoding module, and the spliced ​​features pass through the third feature refinement block to output a segmentation mask.

[0050] In a second aspect, the present invention provides an MRI image segmentation method based on a hybrid attention supervised U-type network MRI image segmentation system according to the first aspect, the method comprising:

[0051] S1: Preprocess the MRI images of four modalities, namely, T1, T1CE, T2 and T2FLAIR, and stack the preprocessed MRI images of the four modalities in the channel direction to obtain a multimodal MRI image;

[0052] S2: training a hybrid attention-supervised U-type network based on the multimodal MRI image to obtain a multimodal MRI image segmentation model;

[0053] S3: Inputting the actual multimodal MRI image into the multimodal MRI image segmentation model to generate a segmentation mask; wherein step S2 is specifically as follows:

[0054] S21: the multimodal MRI image is input into a convolutional partitioning module, the multimodal MRI image is partitioned into image blocks, and each image block is stretched into an image block vector;

[0055] S22: the multimodal MRI image divided into small image blocks is input into a first encoding module, and the first encoding module performs a first downsampling on the multimodal MRI image to extract a first low-resolution feature;

[0056] S23: the first low-resolution feature is input to a second encoding module, and the second encoding module performs a second downsampling on the first low-resolution feature to extract a second low-resolution feature;

[0057] S24: the second low-resolution feature is input to a third encoding module, and the third encoding module performs a third downsampling on the second low-resolution feature to extract a third low-resolution feature;

[0058] S25: the third low-resolution feature is input to a fourth encoding module, and the fourth encoding module performs a fourth downsampling on the third low-resolution feature to extract a fourth low-resolution feature;

[0059] S26: inputting the fourth low-resolution feature into the first decoding module, the first decoding module performing a first upsampling on the fourth low-resolution feature to extract a first high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the first high-resolution feature and the gold standard annotated by the doctor;

[0060] S27: fusing the first high-resolution feature and the third low-resolution feature through a jump connection to obtain a first fused feature; and inputting the first fused feature into a second decoding module, the second decoding module performs a second upsampling on the first fused feature to extract a second high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the second high-resolution feature and the gold standard annotated by the doctor;

[0061] S28: fusing the second high-resolution feature and the second low-resolution feature through a jump connection to obtain a second fused feature; and inputting the second fused feature into a third decoding module, the third decoding module upsampling the second fused feature for a third time to extract a third high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the third high-resolution feature and the gold standard annotated by the doctor;

[0062] S29: fusing the third high-resolution feature and the first low-resolution feature through a jump connection to obtain a third fused feature; and inputting the third fused feature into a fourth decoding module, the fourth decoding module performing a fourth upsampling on the third fused feature to extract a fourth high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the fourth high-resolution feature and the gold standard annotated by the doctor;

[0063] S210: inputting the multimodal MRI image into the second feature refinement block to obtain a refined feature; concatenating the refined feature with the fourth high-resolution feature to obtain a fourth fused feature; inputting the fourth fused feature into the third feature refinement block to output a segmentation mask;

[0064] S211: Evaluate and identify the segmentation mask and the doctor's annotated image. If the evaluation and identification pass, the hybrid attention supervised U-type network at this time is the multimodal MRI image segmentation model; if the evaluation and identification fails, repeat steps S21-S10 to repeatedly train the hybrid attention supervised U-type network until the segmentation mask passes the evaluation and identification, and the hybrid attention supervised U-type network at this time is the multimodal MRI image segmentation model.

[0065] The present invention constructs a hybrid attention supervision U-type network for MRI images. In the encoder part of the network, the present invention designs a pyramid hierarchical encoder based on the proposed three-dimensional local context fusion Transformer block and the three-dimensional global Transformer block, which can more accurately extract the local and global features of multimodal MRI images, accurately detect the edges of the lesion area in the MRI image, and thus significantly improve the MRI image segmentation method of the multimodal MRI image segmentation quality. In the decoder part of the network, the present invention proposes a three-dimensional channel and spatial attention supervision module, which guides the model to more accurately capture and utilize the key information of MRI during training, and suppress redundant information and noise.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] (1) We propose a hybrid attention-supervised U-network that effectively learns the spatial continuity of 3D MRI images and obtains high-precision segmentation masks.

[0068] (2) The present invention first builds three-dimensional neighborhood self-attention and three-dimensional hole neighborhood self-attention, and then designs a context-fused Transformer block that can fully extract local feature information by combining neighborhood self-attention and hole neighborhood self-attention, and successfully applies it to the multimodal MRI image segmentation task.

[0069] (3) In the encoder part, the present invention integrates neighborhood self-attention, void neighborhood self-attention and global Transformer blocks to build a hierarchical hybrid encoder, which enables the model to achieve more refined feature extraction in three-dimensional space. At the same time, it also effectively models long- and short-distance dependencies, significantly improving the encoder's ability to capture MRI image features.

[0070] (4) In the decoder part, the present invention uses a feature focusing module to highlight useful features and suppress redundant information, and combines it with layer-by-layer supervision loss to ensure the accuracy of loss calculation. It can effectively guide the network training process, so that the model can learn the feature representation of 3D MRI images more accurately during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0072] Figure 1 A structural block diagram of a hybrid attention-supervised U-network MRI image segmentation system provided in an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of a flow chart of a hybrid attention-supervised U-type network MRI image segmentation method provided in an embodiment of the present invention;

[0074] Figure 3 A schematic diagram of a workflow for training a hybrid attention supervision U-type network provided in an embodiment of the present invention;

[0075] Figure 4 A schematic diagram of original MRI images of four modalities, namely, T1, T1CE, T2, and T2FLAIR, physician-annotated gold standards, and segmentation masks provided in an embodiment of the present invention;

[0076] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0078] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0080] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0081] See also Figure 1 , is a structural block diagram of a hybrid attention-supervised U-type network MRI image segmentation system provided by an embodiment of the present invention. Figure 1 As shown, the system includes: an image preprocessing module 101, a multimodal MRI image segmentation model training module 102 and an image generation module 103.

[0082] The image preprocessing module 101 is used to preprocess MRI images of four modes, namely, T1 (T1 weighted), T1CE (T1 weighted contrast enhancement), T2 (T2 weighted), and T2FLAIR (T2 fluid attenuated inversion recovery), and stack the preprocessed MRI images of the four modes in the channel direction to obtain a multi-modal MRI image.

[0083] The preprocessing includes: in the MRI images of the four modalities, due to the use of different acquisition devices when acquiring the images, the scanning intensity of the same type of tissue between different images may also be significantly different. In order to solve the two problems of noise effect and intensity inhomogeneity, in the embodiment of the present invention, the highest 1% and the lowest 1% of voxels are first deleted from the MRI image of each case, and then the intensity minimum-maximum standardization is performed on each MRI image.

[0084] The T1 modality MRI images were paired with the corresponding T1CE, T2, and T2FLAIR modality MRI images to obtain image data pairs; all image data pairs were divided into training data sets and test data sets in a ratio of 8:2.

[0085] The training data set is used to train the hybrid attention supervised U-type network to obtain a multimodal MRI image segmentation model; the test data set is used to input MRI images of four modalities into the multimodal MRI image segmentation model to obtain a segmentation mask image.

[0086] It should be noted that, in actual use, there is no need to divide all image data pairs into training data sets and test data sets in a ratio of 8:2. All image data pairs are used to train the hybrid attention supervised U-type network to obtain a multimodal MRI image segmentation model; then the actual MRI image (obtained from the actual image actually collected from the patient through image preprocessing) is input into the multimodal MRI image segmentation model to generate a segmentation mask image.

[0087] The multimodal MRI image segmentation model training module 102 is used to train a hybrid attention supervised U-type network based on the multimodal MRI images in the training data set to obtain a multimodal MRI image segmentation model.

[0088] The multimodal MRI image segmentation model training module 102 includes an encoder, a decoder and a skip connection.

[0089] The encoder is used to downsample the input multimodal MRI image layer by layer, extract multi-scale features at different resolutions, and use the multi-scale features as input to the decoder.

[0090] The encoder includes a convolutional partitioning module and four encoding modules connected in sequence, wherein the first encoding module includes a first local context fusion Transformer block and a first three-dimensional image fusion module connected in sequence, the second encoding module includes a second local context fusion Transformer block and a second three-dimensional image fusion module connected in sequence, the third encoding module includes a first global Transformer block and a third three-dimensional image fusion module connected in sequence, and the fourth encoding module includes a second global Transformer block and a fourth three-dimensional image fusion module connected in sequence.

[0091] The convolution partitioning module is used to divide the input multimodal MRI image into small image blocks, and stretch each small image block into an image block vector for subsequent network training.

[0092] The convolution division module includes a first three-dimensional convolution and a first layer of normalization connected in sequence, and the convolution kernel and stride of the first three-dimensional convolution are both 2.

[0093] The convolutional partitioning module divides the MRI images of four modalities, T1, T2, T1CE and T2FLAIR, into non-overlapping image patches so that the encoder can learn the position information of each image patch. The input channels of the first three-dimensional convolution are 4 (corresponding to the contrast of the MRI images of the four modalities) and the output channels are C The size is After the first three-dimensional convolution, the input image of The output feature map of the input image is converted to an output feature map of size , and the first three dimensions of the output feature map are merged into one dimension, and the fourth dimension remains unchanged. of image patches, where C is the dimension of the image patch after it is stretched into a one-dimensional image patch vector, H is the height, W is the width, D For depth, is the number of image patches; the dimension is C The image patch vector is normalized by the first layer and sent to the first local context fusion Transformer block to learn feature representation.

[0094] The local context fusion Transformer block includes layer normalization (LN), 3D multi-head neighborhood self-attention (MNSA), multi-layer perceptron (MLP), and 3D multi-head void neighborhood self-attention (MDNSA).

[0095] Specifically, the local context fusion Transformer block consists of a second layer of normalization, a three-dimensional multi-head neighborhood self-attention, a third layer of normalization, a first multi-layer perceptron, a fourth layer of normalization, a three-dimensional multi-head void neighborhood self-attention, a fifth layer of normalization and a second multi-layer perceptron connected in sequence.

[0096] Furthermore, the working process of the local context fusion Transformer block is as follows:

[0097] The input features are sequentially subjected to second-layer normalization and three-dimensional multi-head neighborhood self-attention to obtain a first intermediate feature, and the first intermediate feature is concatenated with the input feature to obtain a first concatenated feature;

[0098] The first concatenated feature is sequentially subjected to the third-layer normalization and the first multi-layer perceptron to obtain a second intermediate feature, and the second intermediate feature is concatenated with the first concatenated feature to obtain a second concatenated feature;

[0099] The second splicing feature is sequentially subjected to fourth-layer normalization and three-dimensional multi-head hole neighborhood self-attention to obtain a third intermediate feature, and the third intermediate feature is spliced ​​with the second splicing feature to obtain a third splicing feature;

[0100] The third concatenated feature is sequentially normalized by the fifth layer and subjected to the second multi-layer perceptron to obtain a fourth intermediate feature, and the fourth intermediate feature is concatenated with the third concatenated feature to obtain an output feature.

[0101] In summary, the local context fusion Transformer block output features The calculation formula is as follows:

[0102] ,

[0103] ,

[0104] ,

[0105] ,

[0106] in, LN For layer normalization, MNSA is the three-dimensional multi-head neighborhood self-attention, is the input feature, MLP is a multi-layer perceptron, MDNSA is the multi-headed empty neighborhood self-attention, , , All are intermediate features. l The sequence number of the feature.

[0107] It should be noted that 3D multi-head neighborhood self-attention (MNSA) and 3D multi-head hole neighborhood self-attention (MDNSA) are two advanced self-attention mechanisms that greatly enhance the model's ability to capture and process information. The 3D multi-head neighborhood self-attention (MNSA) is used to perform self-attention calculations on neighboring image patches within a certain range around the selected image patch. This localized processing method cleverly solves the challenges faced in implementing global self-attention calculations on large feature maps, and also greatly improves the efficiency of calculations. The 3D multi-head hole neighborhood self-attention (MDNSA) is used to select other image patches for self-attention calculations at a certain interval with the selected image patch as the center, so as to further enhance the local self-attention interactions between image patches in different ranges. MNSA and MDNSA allow the model to learn information in parallel in different representation subspaces by introducing multi-head designs, which not only greatly improves the computational efficiency, but also enhances the stability of the network model. In addition, by combining layer normalization (LN) and multi-layer perceptron (MLP), the model further improves its expressiveness and generalization capabilities. Layer normalization helps stabilize the training process and reduce internal covariate shift, while multilayer perceptrons provide powerful nonlinear fitting capabilities.

[0108] Furthermore, the three-dimensional multi-head neighborhood self-attention (MNSA) includes a first linear layer and a first softmax weighting connected in sequence.

[0109] For the three-dimensional input feature matrix X input to the three-dimensional multi-head neighborhood self-attention (MNSA), after passing through the first linear layer, the three projections of the query (Query, Q), key (Key, K) and value (Value, V) of the three-dimensional input feature matrix X, as well as the relative position deviation B(i, j) can be obtained.

[0110] It should be noted that the relative position deviation B(i, j) is used in this embodiment of the present invention to replace the absolute position deviation in the original Transformer. B(i, j) is also a learnable variable, which is added to the dot product of the query (Query, Q) and the key (Key, K) in the attention weight calculation. By adding the relative position deviation B(i, j), the trained model can better capture the position information in the MRI image and improve the segmentation performance of the model.

[0111] Assuming that the neighborhood size of the three-dimensional multi-head neighborhood self-attention is d, the size of the neighborhood is similar to the size of the convolution kernel in the convolution, and it will be centered on the query image patch. is the attention weight of the ith image patch among the N image patches of the three-dimensional input feature matrix X, that is, the dot product of the query input of the ith image patch and all adjacent keys K within its neighborhood d.

[0112] ,

[0113] Among them, Q is the query vector matrix Query of the i-th image patch, is the transposed matrix of the key matrix Key of the dth neighborhood image patch centered on the ith image patch, is the dth neighborhood image patch centered on the ith image patch, Query the query matrix for the i-th image patch and the positional relationship between the d-th neighborhood image patch.

[0114] To prevent Too large a value will affect the stability of network training. Divide by Scaling is performed, and the weights are redistributed after the first softmax, and then combined with the value matrix Multiplying them together, we can get the neighborhood self-attention score of the i-th image patch. for:

[0115] ,

[0116] ,

[0117] in, is the neighborhood self-attention score of the i-th image patch, softmax Assign weights to softmax, is the attention weight of the i-th image patch, d is the size of the neighborhood range of the i-th image patch, is the value matrix, is the transposed matrix of the key matrix Key of the dth neighborhood image patch centered on the i-th image patch.

[0118] Furthermore, the three-dimensional multi-headed hole neighborhood self-attention (MDNSA) includes a second linear layer and a second softmax weighting connected in sequence.

[0119] The three-dimensional multi-headed hole neighborhood self-attention can help the model to more fully extract local information on low-level high-resolution feature maps. It introduces a new parameter expansion rate r based on NSA (neighborhood self-attention), which injects holes within the standard self-attention range centered on the selected image patch to compensate for the local details that may be lost in NSA. is the attention weight of the i-th image patch among the N image patches with r holes spaced between them in the input features, and is the dot product of the query input of the i-th image patch and all adjacent keys K within its selected range that are not filled with holes.

[0120] ,

[0121] in, is the dth neighborhood image patch centered on the ith image patch that is not filled with holes, Q is the query vector matrix Query of the ith image patch, is the transposed matrix of the key matrix Key of the d-th hole neighborhood image patch centered on the i-th image patch, The query matrix of the i-th image patch is queried for the positional relationship between the d-th hole neighborhood image patch, and r is the distance between every two hole neighborhood image patches.

[0122] To prevent If the value is too large, it will affect the stability of network training. Divide by Scaling is performed, and the weights are redistributed through the second softmax, and then the expansion value matrix Multiplying them together, we can get the neighborhood expansion self-attention score of the i-th image patch. :

[0123] ,

[0124] ,

[0125] in, is the neighborhood expansion self-attention score of the i-th image patch, softmax Assign weights to softmax, is the attention weight of the i-th image patch among the N image patches with r holes in between each two in the input feature, d is the size of the neighborhood range of the i-th image patch, To expand the value matrix, It is the transposed matrix of the value matrix Value of the dth neighborhood image block that is not filled with holes and is centered on the i-th image block.

[0126] In the three-dimensional multi-head neighborhood self-attention and three-dimensional multi-head hole neighborhood self-attention, the above operation will be performed on each selected query image block of the input feature, that is, each Query will calculate the correlation between the Key and Value within the selected range, thereby efficiently extracting local information of the medical image.

[0127] The global Transformer block includes layer normalization (LN), two 3D multi-head self-attention (MSA), and a multi-layer perceptron (MLP).

[0128] Specifically, the global Transformer block consists of a sixth layer of normalization, a first three-dimensional multi-head self-attention, a seventh layer of normalization, a third multi-layer perceptron, an eighth layer of normalization, a second three-dimensional multi-head self-attention, a ninth layer of normalization and a fourth multi-layer perceptron, which are connected in sequence.

[0129] Furthermore, the working process of the global Transformer block is as follows:

[0130] The input features are sequentially normalized by the sixth layer and subjected to the first three-dimensional multi-head self-attention to obtain a fifth intermediate feature, and the fifth intermediate feature is concatenated with the input feature to obtain a fourth concatenated feature;

[0131] The fourth splicing feature is sequentially subjected to the seventh-layer standardization and the third multi-layer perceptron to obtain a sixth intermediate feature, and the sixth intermediate feature is spliced ​​with the fourth splicing feature to obtain a fifth splicing feature;

[0132] The fifth splicing feature is sequentially subjected to eighth-layer normalization and second-dimensional multi-head self-attention to obtain a seventh intermediate feature, and the seventh intermediate feature is spliced ​​with the fifth splicing feature to obtain a sixth splicing feature;

[0133] The sixth splicing feature is sequentially subjected to the ninth layer standardization and the fourth multi-layer perceptron to obtain an eighth intermediate feature, and the eighth intermediate feature is spliced ​​with the sixth splicing feature to obtain an output feature.

[0134] In summary, the output features of the global Transformer block are The calculation formula is:

[0135] ,

[0136] ,

[0137] ,

[0138] ,

[0139] in, LN For layer normalization, MSA is three-dimensional multi-head self-attention, is the input feature, MLP is a multi-layer perceptron, , and All are intermediate features. l The sequence number of the feature.

[0140] It should be noted that the global Transformer block no longer adopts the neighborhood self-attention strategy, but instead performs a global multi-head self-attention operation on the entire three-dimensional high-level low-resolution feature map output by the local context fusion Transformer block to fully calculate the global correlation between the compressed and fused image patches, capture long-distance dependencies and achieve global modeling.

[0141] The three-dimensional image fusion module is used to fuse and compress feature information extracted from the local context fusion Transformer block or the global Transformer block.

[0142] The three-dimensional image fusion module includes a feature grouping and a third linear layer connected in sequence.

[0143] The specific working process of the 3D image fusion module is as follows: After the image block vector passes through the first local context fusion Transformer block and the second local context fusion Transformer block, the output feature representation size is When the output feature passes through the 3D image fusion module, the feature is first divided into 8 parts by the feature grouping and spliced ​​together in the channel direction. At this time, the size of the feature is reduced by 2 times, while the number of channels (the dimension of the image patch vector) is increased to 8, and the size of the feature is increased from becomes Then, the third linear layer in the channel direction reduces the vector length of the feature by 4 times, so that the size of the feature is reduced from Reduce to .

[0144] In the above manner, the 3D image fusion module performs data fusion and compression on the image block vectors, so that the number of image blocks is reduced by half, and the vector dimension corresponding to each image block is doubled. In the following third and fourth 3D image fusion modules, the 3D image fusion modules perform similar operations on the output features of the first global Transformer block and the second global Transformer block, so that the size of the feature map in the encoder changes hierarchically.

[0145] The decoder is used to fuse the multi-scale features and generate segmentation masks respectively.

[0146] Specifically, the decoder includes four decoding modules of different levels connected in sequence, a segmentation mask output module and layer-by-layer supervision loss.

[0147] The decoding module at each level includes a feature focusing module, a feature restoration module, a first feature refinement block, and a size adjustment block which are connected in sequence.

[0148] The output end of the feature restoration module and the output end of the first feature refinement block are connected together through a residual connection, so that the feature map adjusted by the feature restoration module and the feature map adjusted by the first feature refinement block are spliced ​​in the channel direction and then input into the size adjustment block.

[0149] The feature focusing module includes three-dimensional channel attention and spatial attention connected in sequence. The feature map output by the encoder F After being input into the feature focusing module, the feature map is first inferred in an orderly manner through channel attention and spatial attention F The channel weight and spatial weight , and then the feature map is adjusted according to the learned weights F Weighting is performed in the channel and spatial directions. The channel attention can adaptively adjust the feature weights of different channels, so that the network focuses on the most representative channels and learns the information between MRI images of different modalities to improve the network's segmentation ability; the spatial attention adjusts the feature weights of different positions of the same channel, so that the network focuses on the most representative spatial position of the glioma and improves the network's adaptability to different segmentation areas.

[0150] The process of correcting the feature map by the feature focusing module can be described as follows:

[0151] ,

[0152]

[0153] in, The feature map corrected by the feature focusing module; is the channel weight; is the spatial weight; F is the feature map; Represents the element-by-element multiplication of the feature map and the weight.

[0154] Furthermore, the channel attention includes a first 3DMaxPooling, a first 3DAvgPooling, a fifth multilayer perceptron and a first Sigmoid weighting, the output ends of the first 3DMaxPooling and the first 3DAvgPooling are both connected to the input ends of the fifth multilayer perceptron, and the output end of the fifth multilayer perceptron is connected to the input end of the first Sigmoid weighting.

[0155] The specific working process of the channel attention is as follows: F First, go through the first 3DMaxPooling and the first 3DAvgPooling in the spatial direction to get the spatial features and , spatial characteristics and The shared fifth multi-layer perceptron is used for learning and fusion compression, and finally the first Sigmoid weighting is used to obtain the channel weight. .

[0156] Channel Weight The calculation formula is as follows:

[0157] ,

[0158] in, F is the feature map, Empower Sigmoid. MLP A multi-layer perceptron.

[0159] The spatial attention includes a second 3DMaxPooling, a second 3DAvgPooling, a 3D convolution and a second Sigmoid weighting, the output ends of the second 3DMaxPooling and the second 3DAvgPooling are both connected to the input end of the 3D convolution, and the output end of the 3D convolution is connected to the input end of the second Sigmoid weighting.

[0160] The specific working process of the spatial attention is as follows: The channel attention uses the channel weight For feature maps F Weighted, get the feature map of channel direction correction ; Feature map After entering the spatial attention, it first passes through the second 3DMaxPooling and the second 3DAvgPooling in the channel direction to obtain the feature and ;feature and After learning fusion through the 3D convolution with a shared convolution kernel of (7,7,7), the spatial weight is finally obtained through the second Sigmoid weighting. .

[0161] Spatial weight The calculation formula is as follows:

[0162] ,

[0163] in, is the feature map corrected in the channel direction, Empower Sigmoid. It is a 3D convolution with the convolution kernel (7,7,7).

[0164] The feature restoration module includes a three-dimensional transposed convolution, a tenth layer of normalization, and a first relu activation function connected in sequence. First, it is processed through a 3D transposed convolution block, which doubles the spatial size of the decoded feature map in depth, height, and width, while reducing its number of channels to half. This step is to gradually enlarge the compact feature map output by the encoder to make it close to the spatial size of the original input.

[0165] The first feature refinement block includes a second three-dimensional convolution, an eleventh layer of standardization, a second relu activation function and a residual connection. The second three-dimensional convolution, the eleventh layer of standardization and the second relu activation function are connected in sequence. The two ends of the residual connection are respectively connected to the input end of the second three-dimensional convolution and the output end of the second relu activation function to splice the input feature map of the second three-dimensional convolution with the output feature map of the second relu activation function in the channel direction.

[0166] It should be noted that the present invention introduces the first feature refinement block to solve the defect that the feature restoration module may cause the feature map to lose some details or produce distortion. The first feature refinement block uses residual connections to help the network better retain and restore detail information. In addition, residual connections can also effectively alleviate the problem of gradient disappearance or explosion, thereby accelerating network convergence and capturing more complex and abstract feature representations. Feature map adjusted by the first feature refinement block The feature map adjusted by the feature restoration module After splicing in the channel direction, after the size adjustment block, it is sent to the feature focusing module of the decoding module of the next level for further decoding learning.

[0167] The resizing block includes a 1×1×1 convolutional layer and a cubic linear upsampling layer connected in sequence. The 1×1×1 convolutional layer first performs channel fusion on the input feature map, and then uses the cubic linear upsampling technique to expand the spatial size of the input feature map after channel fusion.

[0168] Furthermore, the shape of the output feature map of the resizing block is the same as the gold standard annotated by the doctor.

[0169] The segmentation mask output module includes a second feature refinement block and a third feature refinement block connected in sequence, and is used to output a segmentation mask.

[0170] The input features of the second feature refinement block are multimodal MRI images, the output features of the second feature refinement block are spliced ​​with the output features of the fourth-level decoding module, and the spliced ​​features pass through the third feature refinement block to output a segmentation mask.

[0171] The layer-by-layer supervised loss is used to perform strong supervised loss calculation on the output feature map of each resized block and the gold standard annotated by the doctor. This layer-by-layer optimization strategy ensures that the decoder can learn more refined and rich feature representations. During the training process, each supervisor will calculate the corresponding loss to guide the network to learn different feature representations.

[0172] The jump connection is used to fuse the features between the encoder and the decoder to reduce the semantic difference between shallow features and deep features.

[0173] Furthermore, the multimodal MRI image segmentation model training module also includes an image evaluation and identification module for identifying whether the segmentation mask of the hybrid attention supervision U-type network is qualified.

[0174] The image evaluation and identification module evaluates and identifies the segmentation mask and the doctor's annotated image. If the evaluation and identification pass, the hybrid attention supervised U-type network at this time is a multimodal MRI image segmentation model; if the evaluation and identification fails, the hybrid attention supervised U-type network continues to be repeatedly trained until the segmentation mask passes the evaluation and identification, and the hybrid attention supervised U-type network at this time is a multimodal MRI image segmentation model.

[0175] The image generation module 103 is used to input the actual multimodal MRI image into the multimodal MRI image segmentation model to generate a segmentation mask.

[0176] Based on the above embodiments, this embodiment provides a MRI image segmentation method based on a hybrid attention-supervised U-type network, see Figure 2 , is a flow chart of a hybrid attention-supervised U-type network MRI image segmentation method provided by an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0177] S1: Preprocess the MRI images of four modalities, namely, T1, T1CE, T2 and T2FLAIR, and stack the preprocessed MRI images of the four modalities in the channel direction to obtain a multimodal MRI image. Specifically:

[0178] S11: The top 1% and bottom 1% of voxels were removed from the MRI images of the four modalities of T1, T1CE, T2, and T2FLAIR for each case, and then the intensity minimum-maximum normalization was performed on each MRI image;

[0179] S12: Pairing the MRI image in T1 mode with the corresponding MRI images in T1CE, T2 and T2FLAIR modes to obtain image data pairs;

[0180] S13: The preprocessed MRI images of the four modalities are stacked in the channel direction to obtain a multi-modal MRI image.

[0181] S2: Based on the multimodal MRI image, a hybrid attention supervised U-type network is trained to obtain a multimodal MRI image segmentation model.

[0182] See also Figure 3 , is a schematic diagram of the workflow of training a hybrid attention supervision U-type network provided by an embodiment of the present invention. Figure 3 As shown, step S2 is specifically as follows:

[0183] S21: the multimodal MRI image is input into a convolutional partitioning module, the multimodal MRI image is partitioned into image blocks, and each image block is stretched into an image block vector;

[0184] S22: the multimodal MRI image divided into small image blocks is input into a first encoding module, and the first encoding module performs a first downsampling on the multimodal MRI image to extract a first low-resolution feature;

[0185] S23: the first low-resolution feature is input to a second encoding module, and the second encoding module performs a second downsampling on the first low-resolution feature to extract a second low-resolution feature;

[0186] S24: the second low-resolution feature is input to a third encoding module, and the third encoding module performs a third downsampling on the second low-resolution feature to extract a third low-resolution feature;

[0187] S25: the third low-resolution feature is input to a fourth encoding module, and the fourth encoding module performs a fourth downsampling on the third low-resolution feature to extract a fourth low-resolution feature;

[0188] S26: inputting the fourth low-resolution feature into the first decoding module, the first decoding module performing a first upsampling on the fourth low-resolution feature to extract a first high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the first high-resolution feature and the gold standard annotated by the doctor;

[0189] S27: fusing the first high-resolution feature and the third low-resolution feature through a jump connection to obtain a first fused feature; and inputting the first fused feature into a second decoding module, the second decoding module performs a second upsampling on the first fused feature to extract a second high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the second high-resolution feature and the gold standard annotated by the doctor;

[0190] S28: fusing the second high-resolution feature and the second low-resolution feature through a jump connection to obtain a second fused feature; and inputting the second fused feature into a third decoding module, the third decoding module upsampling the second fused feature for a third time to extract a third high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the third high-resolution feature and the gold standard annotated by the doctor;

[0191] S29: fusing the third high-resolution feature and the first low-resolution feature through a jump connection to obtain a third fused feature; and inputting the third fused feature into a fourth decoding module, the fourth decoding module performing a fourth upsampling on the third fused feature to extract a fourth high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the fourth high-resolution feature and the gold standard annotated by the doctor;

[0192] S210: inputting the multimodal MRI image into the second feature refinement block to obtain a refined feature; concatenating the refined feature with the fourth high-resolution feature to obtain a fourth fused feature; inputting the fourth fused feature into the third feature refinement block to output a segmentation mask;

[0193] S211: Evaluate and identify the segmentation mask and the doctor's annotated image. If the evaluation and identification pass, the hybrid attention supervised U-type network at this time is the multimodal MRI image segmentation model; if the evaluation and identification fails, repeat steps S21-S10 to repeatedly train the hybrid attention supervised U-type network until the segmentation mask passes the evaluation and identification, and the hybrid attention supervised U-type network at this time is the multimodal MRI image segmentation model.

[0194] S3: Inputting the actual multimodal MRI image into the multimodal MRI image segmentation model to generate a segmentation mask.

[0195] See also Figure 4 , which is a schematic diagram of the original T1, T1CE, T2, and T2FLAIR MRI images, the doctor-annotated gold standard, and the segmentation mask provided in an embodiment of the present invention. Figure 4 As shown, it can be clearly seen that the images generated by the system and method provided by the embodiments of the present disclosure are very similar to the images manually annotated by the doctor, which proves the effectiveness of the system and method provided by the embodiments of the present disclosure.

[0196] Corresponding to the above embodiment, an embodiment of the present invention further provides an electronic device.

[0197] See also Figure 5 , is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 As shown, the electronic device 500 may include: a processor 501, a memory 502 and a communication unit 503. These components communicate via one or more buses. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0198] The communication unit 503 is used to establish a communication channel so that the electronic device can communicate with other devices.

[0199] The processor 501 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 502, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 501 can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0200] The memory 502 is used to store the execution instructions of the processor 501. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0201] When the execution instructions in the memory 502 are executed by the processor 501 , the electronic device 500 is enabled to execute part or all of the steps in the above method embodiment.

[0202] Corresponding to the above embodiment, the embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, wherein when the program is running, the device where the computer-readable storage medium is located may be controlled to execute some or all of the steps in the above method embodiment. In a specific implementation, the computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0203] Corresponding to the above embodiment, an embodiment of the present invention further provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes part or all of the steps in the above method embodiment.

[0204] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. A and B may be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c may be single or multiple.

[0205] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0206] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0207] In several embodiments provided by the present invention, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0208] The above description is only a specific implementation of the present invention. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention.

Claims

1. A hybrid attention-supervised U-network MRI image segmentation system, characterized in that: The system comprises: an image preprocessing module, a multimodal MRI image segmentation model training module and an image generation module, wherein: The image preprocessing module is used to preprocess the MRI images of four modes, namely, T1, T1CE, T2 and T2FLAIR, and stack the preprocessed MRI images of the four modes in the channel direction to obtain a multi-modal MRI image; The multimodal MRI image segmentation model training module is used to train a hybrid attention supervision U-type network based on the multimodal MRI image to obtain a multimodal MRI image segmentation model; The image generation module is used to input the actual multimodal MRI image into the multimodal MRI image segmentation model to generate a segmentation mask; The multimodal MRI image segmentation model training module includes an encoder, a decoder and a skip connection, wherein: The encoder is used to downsample the multimodal MRI image layer by layer, extract multi-scale features at different resolutions, and use the multi-scale features as input to the decoder; The decoder is used to fuse the multi-scale features to generate a segmentation mask; The jump connection is used to fuse the features between the encoder and the decoder to reduce the semantic difference between the shallow features and the deep features; The encoder includes a convolutional partitioning module and four encoding modules connected in sequence, wherein: The convolution partitioning module includes a first three-dimensional convolution and a first layer of normalization connected in sequence, and the convolution kernel and stride of the first three-dimensional convolution are both 2; The first encoding module includes a first local context fusion Transformer block and a first three-dimensional image fusion module connected in sequence; the second encoding module includes a second local context fusion Transformer block and a second three-dimensional image fusion module connected in sequence; the third encoding module includes a first global Transformer block and a third three-dimensional image fusion module connected in sequence; the fourth encoding module includes a second global Transformer block and a fourth three-dimensional image fusion module connected in sequence; The first local context fusion Transformer block and the second local context fusion Transformer block are each composed of a second layer of normalization, a three-dimensional multi-head neighborhood self-attention, a third layer of normalization, a first multi-layer perceptron, a fourth layer of normalization, a three-dimensional multi-head void neighborhood self-attention, a fifth layer of normalization, and a second multi-layer perceptron connected in sequence; The output features of the first local context fusion Transformer block and the second local context fusion Transformer block The calculation formula is as follows: , , , , in, LN For layer normalization, MNSA is the three-dimensional multi-head neighborhood self-attention, is the input feature, MLP is a multi-layer perceptron, MDNSA is the multi-headed empty neighborhood self-attention, , , All are intermediate features. l The sequence number of the feature.

2. According to claim 1, a hybrid attention-supervised U-network MRI image segmentation system is characterized in that: The three-dimensional multi-head neighborhood self-attention includes the first linear layer and the first softmax weighting connected in sequence, and the neighborhood self-attention score of the i-th image patch The calculation formula is: , , in, is the value matrix; is the transposed matrix of the key matrix Key of the dth neighborhood image patch centered on the ith image patch; softmax is the softmax weight; d is the size of the neighborhood range of the i-th image patch; is the attention weight of the i-th image patch, The calculation formula is: , Among them, Q is the query vector matrix Query of the i-th image patch, is the transposed matrix of the key matrix Key of the dth neighborhood image patch centered on the ith image patch, is the dth neighborhood image patch centered on the ith image patch, Query the query matrix for the i-th image patch and the positional relationship between the d-th neighborhood image patch.

3. According to claim 1, a hybrid attention-supervised U-network MRI image segmentation system is characterized in that: The three-dimensional multi-headed hole neighborhood self-attention includes a second linear layer and a second softmax weighting connected in sequence, and the neighborhood expansion self-attention score of the i-th image patch The calculation formula is: , , in, To expand the value matrix; is the transposed matrix of the value matrix Value of the dth neighborhood image block that is not filled with holes and is centered on the i-th image block; softmax is the softmax weight; d is the size of the neighborhood range of the i-th image patch; is the attention weight of the i-th image patch among the N image patches with r holes between them in the input features, The calculation formula is: , Among them, Q is the query vector matrix Query of the i-th image patch, is the transposed matrix of the key matrix Key of the d-th hole neighborhood image patch centered on the i-th image patch, is the dth neighborhood image patch that is not filled with holes and is centered on the ith image patch. The query matrix of the i-th image patch is queried for the positional relationship between the d-th hole neighborhood image patch, and r is the distance between every two hole neighborhood image patches.

4. According to claim 1, a hybrid attention-supervised U-network MRI image segmentation system is characterized in that: The first global Transformer block and the second global Transformer block are both composed of the sixth layer of normalization, the first three-dimensional multi-head self-attention, the seventh layer of normalization, the third multi-layer perceptron, the eighth layer of normalization, the second three-dimensional multi-head self-attention, the ninth layer of normalization, and the fourth multi-layer perceptron connected in sequence; Output features of the first global Transformer block and the second global Transformer block The calculation formula is: , , , , in, LN For layer normalization, MSA is three-dimensional multi-head self-attention, is the input feature, MLP is a multi-layer perceptron, , and All are intermediate features. l The sequence number of the feature.

5. The MRI image segmentation system based on hybrid attention supervision U-type network according to claim 1 is characterized in that: The first three-dimensional image fusion module, the second three-dimensional image fusion module, the third three-dimensional image fusion module and the fourth three-dimensional image fusion module all include feature groups and a third linear layer connected in sequence.

6. The MRI image segmentation system based on hybrid attention supervision U-type network according to claim 1 is characterized in that: The decoder includes four decoding modules of different levels connected in sequence, a segmentation mask output module and a layer-by-layer supervision loss, and each decoding module of the layer includes a feature focusing module, a feature restoration module, a first feature refinement block and a size adjustment block connected in sequence; The feature focusing module includes three-dimensional channel attention and spatial attention connected in sequence; the channel attention includes a first 3DMaxPooling, a first 3DAvgPooling, a fifth multilayer perceptron and a first Sigmoid weighting, the output ends of the first 3DMaxPooling and the first 3DAvgPooling are both connected to the input end of the fifth multilayer perceptron, and the output end of the fifth multilayer perceptron is connected to the input end of the first Sigmoid weighting; the spatial attention includes a second 3DMaxPooling, a second 3DAvgPooling, a 3D convolution and a second Sigmoid weighting, the output ends of the second 3DMaxPooling and the second 3DAvgPooling are both connected to the input end of the 3D convolution, and the output end of the 3D convolution is connected to the input end of the second Sigmoid weighting; The feature restoration module includes a three-dimensional transposed convolution, a tenth layer of normalization, and a first relu activation function connected in sequence; The first feature refinement block includes a second three-dimensional convolution, an eleventh layer of standardization, a second relu activation function and a residual connection; the second three-dimensional convolution, the eleventh layer of standardization and the second relu activation function are connected in sequence, and the two ends of the residual connection are respectively connected to the input end of the second three-dimensional convolution and the output end of the second relu activation function, so as to splice the input feature map of the second three-dimensional convolution with the output feature map of the second relu activation function in the channel direction; The resizing block includes a 1×1×1 convolutional layer and three linear upsampling layers connected in sequence.

7. The MRI image segmentation system based on hybrid attention supervision U-type network according to claim 6 is characterized in that: The segmentation mask output module includes a second feature refinement block and a third feature refinement block connected in sequence; the input feature of the second feature refinement block is a multimodal MRI image, the output feature of the second feature refinement block is spliced ​​with the output feature of the fourth-level decoding module, and the spliced ​​features pass through the third feature refinement block to output a segmentation mask.

8. The MRI image segmentation method based on the hybrid attention supervised U-type network MRI image segmentation system according to claim 7, characterized in that: The method comprises: S1: Preprocess the MRI images of four modalities, namely, T1, T1CE, T2 and T2FLAIR, and stack the preprocessed MRI images of the four modalities in the channel direction to obtain a multimodal MRI image; S2: training a hybrid attention-supervised U-type network based on the multimodal MRI image to obtain a multimodal MRI image segmentation model; S3: Inputting the actual multimodal MRI image into the multimodal MRI image segmentation model to generate a segmentation mask; wherein step S2 is specifically as follows: S21: the multimodal MRI image is input into a convolutional partitioning module, the multimodal MRI image is partitioned into image blocks, and each image block is stretched into an image block vector; S22: the multimodal MRI image divided into small image blocks is input into a first encoding module, and the first encoding module performs a first downsampling on the multimodal MRI image to extract a first low-resolution feature; S23: the first low-resolution feature is input to a second encoding module, and the second encoding module performs a second downsampling on the first low-resolution feature to extract a second low-resolution feature; S24: the second low-resolution feature is input to a third encoding module, and the third encoding module performs a third downsampling on the second low-resolution feature to extract a third low-resolution feature; S25: the third low-resolution feature is input to a fourth encoding module, and the fourth encoding module performs a fourth downsampling on the third low-resolution feature to extract a fourth low-resolution feature; S26: inputting the fourth low-resolution feature into the first decoding module, the first decoding module performing a first upsampling on the fourth low-resolution feature to extract a first high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the first high-resolution feature and the gold standard annotated by the doctor; S27: fusing the first high-resolution feature and the third low-resolution feature through a jump connection to obtain a first fused feature; and inputting the first fused feature into a second decoding module, the second decoding module performs a second upsampling on the first fused feature to extract a second high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the second high-resolution feature and the gold standard annotated by the doctor; S28: fusing the second high-resolution feature and the second low-resolution feature through a jump connection to obtain a second fused feature; and inputting the second fused feature into a third decoding module, the third decoding module upsampling the second fused feature for a third time to extract a third high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the third high-resolution feature and the gold standard annotated by the doctor; S29: fusing the third high-resolution feature and the first low-resolution feature through a jump connection to obtain a third fused feature; and inputting the third fused feature into a fourth decoding module, the fourth decoding module performing a fourth upsampling on the third fused feature to extract a fourth high-resolution feature; the layer-by-layer supervision loss performs a strong supervision loss calculation on the fourth high-resolution feature and the gold standard annotated by the doctor; S210: inputting the multimodal MRI image into the second feature refinement block to obtain a refined feature; concatenating the refined feature with the fourth high-resolution feature to obtain a fourth fused feature; inputting the fourth fused feature into the third feature refinement block to output a segmentation mask; S211: Evaluate and identify the segmentation mask and the doctor's annotated image. If the evaluation and identification pass, the hybrid attention supervised U-type network at this time is the multimodal MRI image segmentation model; if the evaluation and identification fails, repeat steps S21-S10 to repeatedly train the hybrid attention supervised U-type network until the segmentation mask passes the evaluation and identification, and the hybrid attention supervised U-type network at this time is the multimodal MRI image segmentation model.

Citation Information

Patent Citations

  • A T1WI-fMRI image tumor collaborative segmentation method based on 3D-Unet and graph theory segmentation

    CN109741343A

  • Brain tumor segmentation method based on multi-level structure relation learning network

    CN111402259A

  • Multi-modal brain tumor image segmentation method based on self-supervised learning

    CN115841464A

  • Knowledge-enhanced cross-modal object visual accessibility segmentation method

    CN117392383A

  • Self-supervised multi-modal medical image segmentation method based on artificial intelligence

    CN117409198A