Medical image segmentation model establishment method based on harmonic attention and medical image segmentation method

By introducing the adjustment attention layer and Transformer module into the three-dimensional medical image segmentation model, the feature distribution between different planes is harmonized, and the problem of difficulty in extracting information related to depth dimensions and other dimensions is solved, and the segmentation accuracy and effect are improved.

CN118967714AActive Publication Date: 2024-11-15HUAZHONG UNIV OF SCI & TECH

Patent Information

Application Number
CN202411024514.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

The existing three-dimensional medical image segmentation method cannot effectively extract the correlation information between depth dimensions and other dimensions, resulting in poor quality of segmentation results.

Method used

Using a medical image segmentation model based on harmonic attention, the coordination attention layer is introduced into the upsampling module, the feature distribution between different planes is harmonized, and the position encoding is combined with the Transformer module to extract the correlation relationship between the depth dimension and other dimensions. Improve segmentation accuracy.

Benefits of technology

It effectively reduces the difference in feature distribution between different dimensions and improves the accuracy of three-dimensional medical image segmentation, especially the segmentation effect of lesion areas with discontinuous and unclear contours.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967714A_ABST
    Figure CN118967714A_ABST
Patent Text Reader

Abstract

The invention discloses a medical image segmentation model establishment method based on harmonic attention and a medical image segmentation method, and belongs to the field of three-dimensional medical image segmentation, and the model comprises a modal fusion module which employs a three-dimensional CNN to extract space and depth feature information of different modals at the same time, and carries out modal fusion based on an input-level fusion strategy; the down-sampling module is used for performing down-sampling on the image by using convolution operation of the U-Net network, compressing space and depth information and extracting semantic information at the same time; the Transform module is used for acquiring a global and remote information dependency relationship by utilizing alternating position codes and providing position codes for corresponding semantic information so as to reinforce space information; and the up-sampling module is used for carrying out up-sampling on the image by utilizing a deconvolution operation, acquiring global features by utilizing a harmonic attention layer and stabilizing the distribution condition of the features. According to the invention, the feature distribution condition among different planes of the three-dimensional medical image can be reconciled, and the segmentation precision of the three-dimensional medical image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional medical image segmentation, and more specifically, relates to a method for establishing a medical image segmentation model based on coordinated attention and a medical image segmentation method. Background Art

[0002] Medical image segmentation refers to segmenting the lesion area from a medical image. Early traditional medical image segmentation methods mainly focused on mathematical methods such as edge detection, template matching technology, region growing, graph cutting, active contours, and machine learning. Traditional segmentation methods have problems such as low segmentation efficiency and low accuracy. In recent years, deep learning technology has gradually become the research focus in fields such as computer vision and image processing. Compared with the high sensitivity of traditional segmentation algorithms to images, convolutional neural networks (CNNs) can extract hierarchical features of images, and CNNs are insensitive to image noise, blur, contrast and other attributes, and they provide good segmentation results for medical images.

[0003] In medical image segmentation, many deep learning-based methods are widely used, such as residual networks (ResNet), fully convolutional networks (FCN), and U-Net based on convolutional neural networks (CNN). Among them, U-Net adopts a symmetric encoder-decoder structure with jump connections, which can effectively preserve image details. Although many improved network structures have emerged subsequently, it is difficult for convolution operations to obtain global semantic information. Therefore, inspired by the attention mechanism in natural language processing, existing studies have overcome this limitation by combining the attention mechanism with the CNN model. Recently, the Transformer-based network framework has achieved remarkable performance on various computer vision tasks, represented by the Visual Transformer (ViT), which divides the image into blocks and models the correlation between these blocks with the Transformer sequence. Although this method can improve the segmentation accuracy, it has a large amount of computation, resulting in low image segmentation efficiency.

[0004] Compared with two-dimensional medical images, three-dimensional medical images can provide depth information at the same time, and multimodal medical images can provide information and features of the lesion area and its surrounding areas from multiple levels and multiple angles, which greatly helps the diagnosis of patient lesions. However, the existing three-dimensional medical image segmentation methods cannot effectively extract the correlation information between the depth dimension and other dimensions. In actual three-dimensional medical images, the feature distribution of lesions between planes in different directions is different, which makes the existing medical image segmentation methods have poor segmentation results when applied to three-dimensional medical image segmentation. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a method for establishing a medical image segmentation model based on harmonious attention and a medical image segmentation method, the purpose of which is to harmonize the feature distribution between different planes of a three-dimensional medical image to reduce the difference in feature distribution in different planes and improve the segmentation accuracy of the three-dimensional medical image.

[0006] To achieve the above object, according to one aspect of the present invention, a method for establishing a medical image segmentation model based on harmonious attention is provided, comprising: building a medical image segmentation model and training it using a medical image data set with annotated segmentation results to complete the establishment of the medical image segmentation model; the medical image is a three-dimensional image;

[0007] The medical image segmentation model includes: a downsampling module, which is an encoder structure in a U-Net network, used to downsample the medical image N times, compress the spatial and depth information, and extract the semantic information to obtain a multi-scale primary semantic feature map; a Transformer module, including a Transformer model, used to obtain the global and long-range semantic information of the minimum-scale primary semantic feature map, and perform position encoding to obtain an enhanced semantic feature map; and an upsampling module, used to upsample the enhanced semantic feature map N times to obtain a segmentation map;

[0008] Among them, N is a positive integer greater than 1; the upsampling module includes N upsampling units connected in sequence; the upsampling unit includes an upsampling layer, a skip connection layer and a harmonic attention layer connected in sequence; the upsampling layer is used to upsample the input feature map; the skip connection layer is used to splice the feature map output by the upsampling layer with the primary semantic feature map of the corresponding scale; the harmonic attention layer is used to extract the feature information in different planes of the feature map output by the skip connection layer, and adjust the feature information of each plane to obey the Gaussian distribution, and then fuse the adjusted feature information in each plane into the attention feature map output by the corresponding upsampling unit.

[0009] Furthermore, the reconciled attention layer includes a hard attention branch;

[0010] The hard attention branch includes: three sub-attention branches and the first fusion layer;

[0011] The three sub-attention branches are used to extract feature information in different planes of the input feature map of the harmonic attention layer, and adjust the feature information in the corresponding plane to obey the Gaussian distribution; the sub-attention branches include: an adaptive average pooling layer and a layer normalization layer connected in sequence;

[0012] The first fusion layer uses the matrix broadcast mechanism to perform dot multiplication on the feature information output by the three sub-attention branches to obtain a hard attention map, and uses its output as the attention feature map output by the corresponding upsampling unit.

[0013] Furthermore, the reconciled attention layer also includes a soft attention branch and a second fusion layer;

[0014] The soft attention branch includes: a spatial attention sub-branch, a channel attention sub-branch and a third fusion layer; the spatial attention sub-branch extracts the spatial attention map of the input feature map of the harmonic attention layer based on the spatial attention mechanism, and the channel attention sub-branch extracts the channel attention map of the input feature map of the harmonic attention layer. The third fusion layer uses the matrix broadcast mechanism to perform point multiplication on the spatial attention map and the channel attention map to obtain the soft attention map;

[0015] The second fusion layer is used to perform dot multiplication on the soft attention map and the hard attention map using the matrix broadcasting mechanism before outputting the hard attention map, and output the resulting feature map as the attention feature map output by the corresponding upsampling unit.

[0016] Furthermore, the Transformer module includes a plurality of Transformer models cascaded in sequence, and the plurality of Transformer models alternately use an absolute position encoding method and a relative position encoding method to implement position encoding;

[0017] The absolute position encoding method includes: at the input layer of the Transformer model, a learnable position vector is set for each image block in the image block sequence input to the Transformer model, so as to identify the position information of the corresponding image block in the entire image block sequence; the position vector of each image block is concatenated with the vectorized representation of the corresponding image block as the input of the layer after the input layer;

[0018] The relative position encoding method includes: in the multi-head attention layer of the Transformer model, the attention matrix output by the multi-head attention layer is added to the non-learnable relative distance matrix as the input of the layer after the multi-head attention layer;

[0019] Among them, the image block sequence input to the Transformer model is obtained by dividing the minimum-scale primary semantic feature map or the feature map output by the previous level Transformer model; the relative distance matrix records the distance information between any two image blocks in the image block sequence input to the Transformer model.

[0020] Furthermore, the relative distance matrix is ​​calculated by:

[0021] Calculate the Euclidean distance between any two image blocks in the image block sequence to obtain a Euclidean distance matrix;

[0022] Calculate the cosine distance between any two image blocks in the image block sequence to obtain a cosine distance matrix;

[0023] Divide the corresponding elements in the Euclidean distance matrix and the pre-distance matrix to obtain the relative distance matrix.

[0024] Furthermore, the medical image segmentation model further includes: a modality fusion module provided before the downsampling module; the modality fusion module includes an input layer, an output layer and C convolutional layers; C is the modality number of the medical image; wherein:

[0025] The input layer is used to extract C modal data from medical images;

[0026] The C convolutional layers are used to perform convolution operations on the data of C modalities with a convolution kernel size of 1, an expansion value of 0, and a step size of 1 to obtain C modal features;

[0027] The output layer is used to merge C modal features to obtain a medical image after modal fusion;

[0028] Furthermore, the input image of the downsampling module is a medical image after modality fusion.

[0029] Furthermore, the batch normalization layers in the downsampling modules are replaced by group normalization layers.

[0030] According to another aspect of the present invention, a medical image segmentation method based on harmonic attention is provided, comprising: inputting the medical image to be segmented into the medical image segmentation model established by the above-mentioned medical image segmentation model establishment method based on harmonic attention provided by the present invention, to obtain a segmentation map.

[0031] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when processed qualitatively, implements the above-mentioned method for establishing a medical image segmentation model based on harmonic attention provided by the present invention, and / or the above-mentioned medical image segmentation method based on harmonic attention provided by the present invention.

[0032] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising a stored computer program; when the computer program is processed qualitatively, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for establishing a medical image segmentation model based on harmonic attention provided by the present invention, and / or the above-mentioned medical image segmentation method based on harmonic attention provided by the present invention.

[0033] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0034] (1) The medical image segmentation model based on harmonic attention established by the present invention introduces a harmonic attention layer in the upsampling module to harmonize the feature distribution between planes in different directions of the three-dimensional medical image, which plays a transitional role after the jump connection. Specifically, after respectively extracting the feature information of the planes in three different directions, the feature information of the three planes is made to obey the Gaussian distribution to reduce the difference in feature distribution between different dimensions and stabilize the feature distribution. Then, the feature information of the three planes is fused to make the feature information between the planes complementary, thereby effectively extracting the correlation between the depth dimension and other dimensions, realizing accurate modeling of the overall segmentation area, and facilitating the segmentation of discontinuous lesion areas with unclear contours, thereby improving the segmentation accuracy.

[0035] (2) In the preferred embodiment of the present invention, in the medical image segmentation model based on harmonic attention, the introduced harmonic attention layer includes, in addition to the hard attention branch for harmonizing the feature distribution between different planes, a soft attention branch for extracting the spatial attention map and the channel attention map and fusing the two. The soft attention branch can obtain relatively fine-grained pixel features in advance, thereby achieving modeling of each pixel in the segmented area. Combined with the hard attention branch, it can more accurately model the lesion area and further improve the segmentation accuracy.

[0036] (3) In the preferred embodiment of the present invention, in the medical image segmentation model based on coordinated attention, the Transformer module specifically includes multiple Transforer models, and these multiple Transformer models adopt alternating position coding, that is, alternately using absolute position coding and relative position coding to achieve position coding. Absolute position coding reflects the globality of semantic position, and relative position coding reflects the spatial relationship between semantics. The combination of the two can reflect the spatial changes of semantics both globally and locally, and at the same time, use the explicit long-distance dependency established by global features and the details contained in local features to clearly segment boundaries and lesions, thereby further improving segmentation accuracy.

[0037] (4) In the preferred embodiment of the present invention, in the relative position coding, the relative distance information between two image blocks is specifically determined by the Euclidean distance and the cosine distance between the image blocks, which includes both the spatial distance information and the direction information between the image blocks. This is adapted to the characteristics that the internal elements of the lesion area have strong spatial connections and the three-dimensional spatial elements in the segmentation task have directionality, and is conducive to further improving the segmentation accuracy.

[0038] (5) In the preferred embodiment of the present invention, a modality fusion module is introduced before the downsampling module, and features of each modality are extracted and then spliced ​​as the input of the downsampling module, thereby realizing input-level fusion and being able to fully extract and fuse multimodal information in medical images.

[0039] (6) In the preferred embodiment of the present invention, based on the selection of the encoder structure in the U-Net network as the downsampling module, the batch normalization layers therein are modified into group normalization layers, which can reduce the dependence of data on batch size and weaken the distribution relationship between features, avoid the influence of diverse data distribution on the training process, and further improve the segmentation performance of the established model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the structure of a medical image segmentation model provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a modality fusion module provided in an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a downsampling module provided in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a Transformer module provided in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of image segmentation provided by an embodiment of the present invention;

[0045] Figure 6 This is a diagram of the structure of the multi-head attention layer in the existing Transformer model;

[0046] Figure 7 Schematic diagram of the multi-layer perceptron (MLP) layer in the existing Transformer model;

[0047] Figure 8 A schematic diagram of an upsampling module provided in an embodiment of the present invention;

[0048] Fig. 9 A schematic diagram of the coordination attention layer provided in an embodiment of the present invention;

[0049] Fig.10 A schematic diagram showing different regions in a three-dimensional MRI image of the brain provided by an embodiment of the present invention;

[0050] Fig.11 A schematic diagram comparing segmentation results of different segmentation methods provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0052] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0053] In order to solve the technical problem that the existing three-dimensional medical image segmentation method cannot accurately extract the correlation information between the depth dimension and other dimensions and cannot accurately model the lesion area, resulting in poor segmentation result quality, the present invention has fully analyzed and studied the cause of the problem. The present invention finds that the traditional segmentation method is mostly based on the model of the encoder-decoder structure to achieve segmentation. The encoder part mainly uses convolution operation to achieve downsampling, and the decoder part mainly uses deconvolution to achieve upsampling, and the feature maps of the corresponding scales in the encoder and decoder are fused through jump connections. In clinical practice, the feature distribution of medical images of different patients is different, and the feature distribution of the lesion area in different planes in the medical image of the same patient is also different, which makes the conventional encoder-decoder structure model unable to effectively extract the correlation information between the depth dimension and other dimensions, so it is impossible to accurately model the lesion area, and it is impossible to accurately segment the lesion area. When the outline of the lesion area is not clear, the quality of the segmentation result is worse. Based on this discovery, the present invention accordingly proposes to reconcile the feature distribution of the lesion area in different planes in the medical image to reduce the difference in feature distribution between different dimensions, thereby solving the above-mentioned technical problems.

[0054] Based on the above concept, the present invention proposes a new segmentation model structure, which introduces a reconciliation attention layer for reconciling the feature distribution of the lesion area in different planes in the medical image after the jump connection in the encoder-decoder structure, so that the model can accurately model the lesion area during the upsampling process, thereby obtaining an accurate segmentation result. The reconciliation attention layer specifically includes a hard branch for extracting global features under the intersection of two axes, that is, the features in the plane formed by any two coordinate axes in the three-dimensional coordinate axis, and further utilizes the distribution of Gaussian distribution to stabilize the features, reduce the influence of feature distribution differences on the segmentation results, and achieve modeling of the segmentation area as a whole. The reconciliation attention layer may further include a soft attention branch for extracting relatively fine-grained pixel features in space and channels, mainly modeling each pixel in the segmentation area. In order to fully extract the semantic information of the medical image, the present invention introduces a Transformer module based on the Transformer model between the encoder and the decoder. Therefore, in some optional embodiments, the segmentation model mainly includes: a downsampling module, a Transformer module and an upsampling module, and a reconciliation attention layer is introduced in the upsampling module.

[0055] Existing three-dimensional medical images are usually composed of multiple modalities. Making full use of modal information and using the diversity and complexity of modalities for modeling is conducive to improving segmentation accuracy. In order to fully integrate multimodal information, the present invention further proposes to introduce a modality fusion module before the encoder-decoder structure to achieve input-level fusion. Therefore, in some optional embodiments of the present invention, the segmentation model mainly includes: a modality fusion module, a downsampling module, a Transformer module and an upsampling module. Similarly, a reconciliation attention layer is introduced in the upsampling module.

[0056] In both model architectures, the structures and functions of the downsampling module, Transformer module, and upsampling module are the same.

[0057] The medical image segmentation model established by the present invention can be applied to the segmentation of any three-dimensional, multimodal medical image. Without loss of generality, the following embodiments are described by taking the segmentation of brain tumor areas in three-dimensional MRI images of the brain as an example.

[0058] The following are examples.

[0059] Embodiment 1:

[0060] A method for establishing a medical image segmentation model based on harmonious attention, comprising: building a medical image segmentation model and training it using a medical image data set with annotated segmentation results to complete the establishment of the medical image segmentation model; the medical image is a three-dimensional image;

[0061] In this embodiment, the medical image segmentation model is used to segment the brain tumor region in the three-dimensional MRI image of the brain, and accordingly, the medical image dataset is a set of three-dimensional MRI images of the brain with the brain tumor region annotated. It is easy to understand that if the trained model is used for other types of medical image segmentation, the corresponding type of annotated image dataset can be used for model training.

[0062] like Figure 1 As shown in the figure, the medical image segmentation model as a whole consists of four modules: modality fusion module, downsampling module, Transformer module and upsampling module.

[0063] The modality fusion module uses a three-dimensional convolutional neural network (CNN) to simultaneously extract spatial and depth feature information of different modalities, and performs modality fusion to achieve input-level fusion; the downsampling module uses the encoder structure in the U-Net network to downsample the image through convolution operations, compress the spatial and depth information, and extract semantic information at the same time; the Transformer module is implemented based on the Transformer model, using the multi-head attention mechanism in it to obtain global and long-range semantic information, and using position encoding to strengthen spatial information; the upsampling module uses deconvolution operations to upsample the image, and uses the harmonized attention layer to obtain global features and stabilize the distribution of features to obtain high-quality segmentation maps. The specific implementation methods of each module are described in detail below.

[0064] Regarding the modality fusion module based on the input-level fusion strategy, this module, as the input end of the model, performs modality fusion on the input brain 3D MRI image data. C×H×W×D , H×W is its spatial resolution, D is its depth dimension, and C is its modal dimension and feature dimension.

[0065] like Figure 2 As shown, in this module, different modal data are extracted for the input data. A convolution operation with kernel size (kernel_size) of 1, padding value (padding) of 0, and stride (stride) of 1 is performed on each modality. Based on this convolution operation, the image dimension is not changed while the features of each modality are extracted. Finally, the output result is z ai (i=1, 2, ... C) and After that, the output results of each mode are combined (Concat) to obtain the output z of the module b and The convolution operation is uniformly referred to as Conv(kernel_size,padding,stride). The operation process of the modal fusion module is as follows:

[0066] z b =Concat(Conv(z a1 ,z a2 ,...,z ac ))

[0067] There are two main ways to fuse multimodal information: (1) One is through dimension merging and fusion, which will obviously have the problem of low network learning efficiency when the amount of data increases; (2) through addition fusion, which is just a simple superposition and cannot fully extract the correlation information between modalities. In contrast, in this embodiment, the modal fusion module can more fully extract the features of each modality and effectively fuse them.

[0068] Regarding the U-Net-based downsampling module, the input of this module is the output z of the modal fusion module b ,The downsampling module mainly utilizes the multi-layer convolution structure in the U-Net network to extract the ,spatial and depth information of the image, and uses the convolution operation to downsample the image, ,compress the spatial and depth information while extracting the semantic ,information.

[0069] The U-Net network has achieved great success in the field of medical image segmentation with its high accuracy and high performance. The network is a typical encoder-decoder structure. In order to utilize its superior performance in medical image segmentation, this embodiment implements a downsampling module based on the encoder structure in the U-Net network, which consists of a convolutional layer, an activation function, a normalization layer, and a downsampling layer.

[0070] Taking into account the differences in brain tumor images of different patients, in order to avoid the influence of diverse data distribution on the subsequent model training process, as a preferred implementation, this embodiment makes further improvements on the traditional U-Net network encoder structure. Specifically, the batch normalization layer is modified into a group normalization layer to improve network performance, reduce the dependence of data on batch size, and weaken the distribution relationship between features.

[0071] like Figure 3 The figure shows the structure of the downsampling module based on U-Net, which is composed of a four-layer convolution stacking structure and three downsampling layers. In the convolution stacking layer, the core convolution block is UNetConvBlock (hereinafter referred to as UConv), which consists of a series of convolution layers (Conv(3,1,0)), activation functions, and group normalization layers. In this embodiment, the ReLU activation function is referred to as Re, the Sigmoid activation function is referred to as σ, the batch normalization is referred to as BN, the layer normalization is referred to as LN, the group normalization is referred to as GN, and the linear layer is referred to as L. The specific expression of the convolution block UNetConvBlock is as follows:

[0072] UConv(x)=GN(Re(Conv(GN(Re(Conv(x)))))))+x

[0073] The downsampling layer (DownSample) uses the Conv(3,2,1) operation to reduce the image width, height, and depth by half, compressing the spatial and depth information while obtaining more semantic information. The relevant expressions are as follows:

[0074] DownSample(x)=Conv(x)

[0075] For the input z of the downsampling module b , first converted to z through Conv(3,1,1) operation c1 , After that, the output result of the module is obtained through four layers of convolution stacking structure and three downsampling layers. The specific operation process is as follows:

[0076]

[0077] The final output of the downsampling module is z d , The image will continue to be input into the Transformer module for operation. It should be noted that in practical applications, the number of downsampling can be determined according to the characteristics of the actual medical image to be segmented.

[0078] In this embodiment, the Transformer module is implemented based on the Transformer model, using the multi-head attention mechanism in the Transformer module to obtain global and long-range semantic information, and using position encoding to strengthen spatial information. In this embodiment, the overall structure of the Transformer module is as follows: Figure 4 As shown, in order to simultaneously utilize the explicit long-distance dependencies (areas far away from the tumor) established in the global feature space and the details of the local features (areas close to the tumor) to clearly segment boundaries and lesions, in this embodiment, the Transformer module includes multiple Transformer models, and unlike the traditional Transformer module, in this embodiment, the position of the Transformer module adopts alternating position encoding, that is, multiple Transformer models alternately use absolute position encoding and relative position encoding. The absolute position encoding reflects the globality of the semantic position, and the relative position encoding reflects the spatial relationship between semantics. The combination of the two can simultaneously reflect the spatial changes of semantics from both global and local perspectives, thereby improving the accuracy of segmentation.

[0079] The input of this module is the output z of the module in the previous section.d , borrowing ViT's ideas, d The image is subdivided into multiple patches, and the patches are used as indexes to form a linear sequence that is input into the Transformer module. Figure x is the overall structure of the Transformer module based on alternating position coding. Next, the module composition and input data operation process will be explained from three parts: image patch processing, alternating position coding, and Transformer model.

[0080] Before inputting the data into the Transformer block, it needs to be processed into blocks and converted into a linear sequence of image blocks. Convert it into a 2D image block Where (P, P, P) is the spatial resolution of the image block, and N = HWD / P 3 is the number of image blocks. The same hidden layer size H is used in all Transformer blocks. d Therefore, after the data is processed in blocks, a linear layer is needed to transform the feature dimension into H d . Take the input z of this module d Take as an example, the block operation process is as follows Figure 5 The corresponding expressions are as follows:

[0081]

[0082] Regarding alternating position coding, in natural language processing tasks, Transformer adds position coding according to the order of the sequence to build a sequence model, considering that the attention mechanism cannot use the position information of the sequence. In computer vision tasks, position coding can also be used to add position information to the image block sequence to obtain the semantic relationship of the image block context as much as possible.

[0083] Position encoding can be divided into two main categories, one is absolute position encoding and the other is relative position encoding. Absolute position encoding encodes the input sequence from 1 to the maximum sequence length, and each position has a separate position encoding vector. Then, the position vector is combined with the input sequence so that the model can obtain the position information. Relative position encoding encodes the relative distance between input elements so that the model can obtain the position information between elements.

[0084] The Transformer module of this embodiment alternately uses learnable absolute position encoding (hereinafter referred to as APE) and non-learnable relative position encoding (hereinafter referred to as RPE). The former is applied to the first and third Transformer models in the model, and the latter is applied to the second and fourth Transformer models.

[0085] APE has the same dimension as the input data, and each element can be learned through the back propagation mechanism of the model. At the input layer of the Transformer model, APE sets a learnable position vector for each image block in the image block sequence input to the Transformer model, which is used to identify the position information of the corresponding image block in the entire image block sequence; APE adds position encoding in an additive manner. Specifically, the position vector of each image block is concatenated with the vectorized representation of the corresponding image block as the input of the layer after the input layer; the operation process of absolute position encoding is as follows:

[0086]

[0087] RPE also adds position encoding in an additive way. Specifically, in the multi-head attention layer of the Transformer model, the attention matrix output by the multi-head attention layer is added to the non-learnable relative distance matrix as the input of the layer after the multi-head attention layer; considering the strong spatial connection between elements within the brain tumor area, each element does not need to be learned through the back propagation mechanism, and the Euclidean distance is used to obtain its relative spatial distance; considering the directionality between the three-dimensional space elements in the segmentation task, the cosine distance is used to obtain the directional information between the elements. In this embodiment, RPE is determined by the Euclidean distance and cosine distance between image blocks. The specific calculation process is as follows:

[0088] Assume the three-dimensional absolute position of any two image blocks is p i (x pi ,y pi ,z pi ), p j (x pj ,y pj ,z pj ), first calculate the Euclidean distance between the two:

[0089]

[0090] Then calculate the cosine distance between the two:

[0091]

[0092] By calculating the Euclidean distance and cosine distance between any two image blocks, the Euclidean distance matrix can be obtained: and cosine distance matrix Finally, the two matrices are divided element by element to obtain the final relative distance matrix (The relative distance between an element and an element itself is 0). The timing of adding relative position encoding is different from that of absolute position encoding. Since each element has a relative position encoding with respect to other elements, the position encoding is added to the attention matrix and becomes part of the attention. Let x be the input. The specific calculation process is as follows:

[0093]

[0094] Regarding the Transformer model topic, this module uses the Transformer encoder structure, which consists of two normalization layers, a multi-head attention layer and an MLP layer. The components of this module will be introduced in turn below.

[0095] The multi-head attention layer consists of multiple self-attention blocks, such as Figure 6 As shown. The attention function can be described as mapping a query and a set of key-value pairs to an output result, where the query, key, value, and output result are all vectors. The output result is the weighted sum of the values, where the weight assigned to each value is calculated by the compatibility function of the query and the corresponding key. In the multi-head attention layer, the compatibility function uses a scaled point attention function, and the input consists of a dimension of d k Query and Key, and dimension d v Calculate the dot product of all Keys and Query, and divide each Key by And apply a Softmax function to get the weights of these Values.

[0096] In actual calculation, the attention function on a group of queries is calculated at the same time, and all queries are put into the matrix Q. All keys and values ​​are put into the matrices K and V, and the calculation of the output result matrix is ​​as follows:

[0097]

[0098] The attention function used by the multi-head attention layer is different from that of the single-head layer. First, it uses a linear layer to expand the feature dimension of the input data by three times, and then takes Query, Key and Value in turn, and then divides the feature dimension according to the number of attention heads. After performing single-head attention operations, different feature dimensions are merged, and finally a linear layer is used to output the result. The specific calculation method of the multi-head attention layer is shown in the following formula:

[0099] head i =Attention(QWi i Q ,KW i K,VW i V )

[0100] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0101] in and are all learnable parameters of the linear layer, d model is the feature dimension of the input data, h is the number of attention heads, d q =d k =d v =d model / h. All attention functions in this module use self-attention, so Q, K, and V are all themselves. In the following, MultiHead(Q,K,V) is referred to as MSA(x).

[0102] The MLP layer consists of two linear layers, such as Figure 7 As shown in the figure, the first linear layer uses the GeLU activation function and the Dropout mechanism, and the second linear layer only uses the Dropout mechanism. Therefore, the operation flow of the MLP layer can be expressed by the following expression:

[0103] MLP(x)=L(ReLU(L(x)))

[0104] Based on the above operation mechanism, input z d1 As an example, the operation and change process of the data are introduced in detail. Since the operation process of the Transformer model using absolute position encoding and relative position encoding is different, they are described separately here. First, the Transformer model using absolute position encoding (hereinafter referred to as APET) is introduced. The operation process is as follows:

[0105] x′=MSA(LN(APE(x)))+x

[0106] APET(x)=MLP(LN(x′))+x′

[0107] In the Transformer model using relative position encoding (hereinafter referred to as RPET), the position encoding is added during the attention calculation of multi-head attention, as shown in the following formula. The remaining steps are similar to APET, and the calculation process is as follows:

[0108]

[0109] x′=MSA(LN(x))+x

[0110] RPET(x)=MLP(LN(x′))+x′

[0111] The number of Transformer models in the Transformer module of this embodiment is 4. d1 In the Transformer module with alternating position encoding, the data operation and transformation process is shown as follows, where z e This is the final output of the module.

[0112]

[0113] In this embodiment, the upsampling module uses a skip connection and a harmonized attention layer. The upsampling module is similar to the downsampling module based on U-Net, and both are composed of a four-layer network structure, such as Figure 8 As shown in the figure, the input of each layer is jump-connected to the output of the corresponding layer of the downsampling module, that is, a Concat operation is performed. Then, the harmonized attention block is used to extract the spatial attention and channel attention of the output result after the Concat operation, and the Gaussian distribution is used to stabilize the distribution of the attention weights. Finally, deconvolution is used for upsampling to gradually restore the image to its original size.

[0114] In this embodiment, the harmonized attention layer (HA) consists of two branches, such as Fig. 9 As shown in the figure, it is divided into soft attention and hard attention. The soft attention branch (A) is used to extract fine-grained pixel features in space and channels, and mainly models each pixel in the segmented area. The hard attention branch (H) is used to extract global features under the intersection of two axes, and uses the Gaussian distribution to stabilize the distribution of features, mainly modeling the entire segmented area. Finally, the soft attention and hard attention matrices are point-multiplied using the matrix broadcast mechanism to obtain the final attention matrix.

[0115] The reconciled attention layer plays a transitional role after the jump connection: it globally utilizes the spatial and channel information of low-level and high-level features of the image while stabilizing the distribution of both features, which helps to extract higher-level features. The soft attention branch is divided into spatial attention (SA) and channel attention (CA). The purpose of soft attention learning is to generate a significant weight map of the same size as the input. Considering the independence between spatial (i.e., pixel-to-pixel) and channel (i.e., scale-to-scale) attention, the attention map A output by the soft attention branch can be expressed as follows:

[0116] A=SA⊙CA

[0117] in, The two codes represent the spatial attention weight map and the channel attention weight map respectively, and ⊙ represents the dot product of the matrix-based broadcast mechanism.

[0118] In the soft attention branch, the spatial attention weight map and the channel attention weight map are extracted by the spatial attention sub-branch and the channel attention sub-branch respectively, where:

[0119] The spatial attention sub-branch uses a four-layer sub-network for modeling: a global cross-channel average pooling layer (GCCAP), an HConv(3,1,1) convolution layer, a trilinear interpolation upsampling layer, and an HConv(1,1,0) convolution layer. The GCCAP layer realizes the sharing of spatial attention of each channel, and then a convolution layer is used to adaptively learn the spatial attention. The operation process can be expressed as follows:

[0120]

[0121] In the spatial attention sub-branch, the data operation and flow process can be expressed as follows:

[0122] HConv(x)=BN(Re(Conv(x)))

[0123]

[0124] The channel attention sub-branch uses a three-layer sub-network for modeling, a global pooling layer (GAP) and two HConv (1,1,0) convolutional layers. The global pooling layer aggregates the feature information distributed in the three-dimensional space into channel information. This operation provides relatively complete information for subsequent modeling of inter-channel dependencies. Its operation process can be expressed as follows:

[0125]

[0126] In the second HConv layer, the number of channels is reduced in dimension to aggregate channel information while reducing the amount of computation.

[0127] In the channel attention sub-branch, the operation and flow of data can be expressed as follows:

[0128]

[0129] Finally, the spatial attention map and the channel attention map are multiplied by the matrix broadcast mechanism to form a soft attention map A(x), which is expressed as follows:

[0130] A(x)=SA(x)⊙CA(x)

[0131] The hard attention branch consists of three non-learnable adaptive average pooling layers (AAP) and three normalization layers. The adaptive pooling layer is used to extract the feature information of the biaxial cross plane, and the normalization layer is used to make the feature information of the biaxial cross plane (i.e., each directional plane in the three-dimensional medical image) obey the Gaussian distribution and reduce the difference in feature distribution between different dimensions. Considering the correlation and dependence between the three dimensions in the three-dimensional image, three AAP layers are used here to extract features from the WD, HD, and HW planes. Taking the HW plane as an example, the specific calculation method is as follows:

[0132]

[0133] After calculating the cross features of the three planes respectively, the matrix broadcast mechanism is used to perform point multiplication, the coaxial information of the three planes is shared, and the feature information between the planes is complementary. In the hard attention branch, the data operation and flow process is as follows:

[0134]

[0135] H(x)=x′

[0136] Finally, the soft attention map and the hard attention map are dot-multiplied using the matrix broadcast mechanism, and the dot-multiplication result is normalized using the Sigmoid function to obtain the final attention weight matrix. The overall calculation process is as follows:

[0137]

[0138] The model established in this embodiment is an encoder-decoder structure as a whole. The encoder part uses convolutional layers for step-by-step downsampling, and the decoder part uses deconvolution TConv(2,0,2) for step-by-step upsampling. The high-frequency information (i.e., image details) and low-frequency information (i.e., image contours) in the original input image will be gradually restored. Therefore, the low-resolution feature map will eventually be mapped to a pixel-level segmentation map.

[0139] In order to further compensate for the information lost in the downsampling in the encoding stage, the Concat operation is used between the encoder and decoder of the network to fuse the feature maps at the corresponding positions in the two processes in the channel dimension, so that the decoder can retain more high-resolution detail information contained in the high-level feature maps when upsampling, thereby more perfectly restoring the detail information in the original image and improving the segmentation accuracy. The operation process of the upsampling block (UpSample) is as follows:

[0140] UpSample(x)=TConv(Conv(x))

[0141] During the downsampling process of the encoder, the output results of each layer are: Combined with the output results of the Transformer module based on alternating position encoding in the previous section Compared with the structure of the upsampling module, the data operation and transformation process in the upsampling module based on skip connection and reconciled attention is expressed as follows:

[0142]

[0143] Perform an upsampling, a skip connection and calculate the harmonic attention at each layer, and finally get the output of the upsampling module based on skip connection and harmonic attention as z g , The operation process is as follows:

[0144]

[0145] Finally, the output result z of the upsampling module g , through a Conv(1,0,0) operation and Softmax function, the final segmentation result z is obtained h , the operation process is as follows:

[0146] z h =Softmax(Conv(z g ))

[0147] In general, this embodiment uses an input-level fusion strategy in the modal fusion module to integrate multi-modal space and depth information; uses the translation invariance and local receptive field of convolution in the downsampling module to compress spatial and depth information while extracting local semantic information; uses a multi-head self-attention mechanism in the Transformer module to obtain global features and long-range information dependencies, and adds alternating relative position encoding and absolute position encoding to simultaneously strengthen semantics and spatial information between semantics; uses jump connections and harmonic attention in the upsampling module to obtain global low-level and high-level features and stabilize the distribution of both features, ultimately effectively improving the accuracy of medical image segmentation.

[0148] As a preferred implementation, the model training process of this embodiment adopts additive cross entropy and Dice loss function to solve the problem of imbalance between positive and negative sample classes in medical images; the preheating-cosine annealing learning rate is adopted to accelerate the convergence speed of the model and prevent overfitting. Figure 1 shows a detailed schematic diagram of the medical image segmentation model after the training, including module name, structure name, structure details, output results and output size. In the table, all blocks with * use residual structure; in all UEnBlocks, Conv is Conv(3,1,1) operation, in all DeBlock blocks, all Conv is Conv(3,1,1) operation, in all UpSample blocks, Conv is Conv(1,0,1) operation, all TConv is TConv(2,0,2) operation, and in the Segmentation Head block, Conv is Conv(1,0,1) operation.

[0149] Table 1. Details of medical image segmentation model

[0150]

[0151]

[0152] In order to verify the segmentation performance of the medical image segmentation model (abbreviated as HATransUNet) established in this embodiment, 1600 training sample data of the BraTS2021 training set were further used for training, 400 test set samples were used for model testing, and different evaluation indicators of the model were calculated to evaluate the model performance. The same training set and test set were used to train and test the 3D U-Net, 3D V-Net, and Attention U-Net models, and the test results were compared and analyzed with those of this embodiment.

[0153] The BRATS2021 dataset consists of images from multiple MRI scans of 2,000 patients, divided into 1,600 training sets and 400 test sets in a ratio of 8:2. The sub-regions considered for evaluation in BraTS2021 are tumor enhancement (ET), tumor core (TC), and whole tumor (WT). ET includes tumor enhancement, and TC includes tumor enhancement, tumor necrosis, and tumor non-enhancement. WT includes all areas except the background area, describing the complete range of the tumor.

[0154] The patient's scan images contain co-registered MRI images of four modalities: T1-weighted (T1), T1-weighted and contrast-enhanced (T1ce), T2-weighted (T2) and FLAIR, with an image size of 240×240×155. All data have been skull-stripped, linearly aligned, and spatially resampled to 1mm×1mm×1mm. The segmentation labels of the training set are divided into four categories, namely label 1 (necrotic area and non-enhanced area), label 2 (edema area), label 3 (enhanced area) and label 0 (background). The four modalities share segmentation labels. Fig.10 shown.

[0155] Two evaluation indicators are mainly used to evaluate the performance of the model, namely Dice coefficient and Hausdorff distance. Table 2 shows the comparison results of the model method of this application and the other three model methods on the test set.

[0156] Table 2 Segmentation performance indicators of different models on the BraTS2021 test set

[0157]

[0158]

[0159] Three random patients from the BraTS2021 test set were selected for demonstration, such as Fig.11 As shown. The first row is the 32nd layer of the coronal plane of BraTS2021_00014 patient, the second row is the 82nd layer of the coronal plane of BraTS2021_00324 patient, and the third row is the 111th layer of the coronal plane of BraTS2021_00768 patient. The first column is the segmentation result of 3D U-Net, the second column is the segmentation result of 3D V-Net, the third column is the segmentation result of Attention U-Net, the fourth column is the segmentation result of HATransUNet, and the fifth column is the distribution of the true label.

[0160] In order to more clearly show the segmentation performance of each model inside the tumor region and at the edge of the region, Fig.11 The interior and edges of some areas of each patient were partially enlarged and displayed, with different areas represented in the same way. Fig.10. The real labels reflect the problems of similar features, blurred boundaries and discontinuous targets between tumor sub-regions. In the local enlarged images of the segmentation results of the 3D U-Net, 3D V-Net and AttentionU-Net models, there are problems such as discontinuous segmentation areas and unclear segmentation boundaries, especially between the edema area and the background, and between the enhanced area and the non-enhanced area. The above evaluation index results and segmentation result visualization results show that the HATransUNet model proposed in this embodiment can better solve the above problems.

[0161] Embodiment 2:

[0162] A method for establishing a medical image segmentation model based on harmonious attention. This embodiment is similar to the above-mentioned embodiment 1, except that, in this embodiment, the medical image segmentation model does not include a modality fusion module, but only includes a downsampling module, a Transformer module and an upsampling module.

[0163] In this embodiment, the specific implementation of each module in the model and the specific implementation of the remaining steps can refer to the description in the above embodiment 1.

[0164] Embodiment 3:

[0165] A medical image segmentation method based on harmonic attention comprises: inputting the medical image to be segmented into the medical image segmentation model established by the medical image segmentation model establishment method based on harmonic attention provided by the above-mentioned embodiment 1 or 2 to obtain a segmentation map.

[0166] Embodiment 4:

[0167] A computer program product includes a computer program, which, when processed qualitatively, implements the method for establishing a medical image segmentation model based on harmonious attention provided in the above-mentioned embodiment 1 or 2, and / or the method for establishing a medical image segmentation model based on harmonious attention provided in the above-mentioned embodiment 3.

[0168] Embodiment 5:

[0169] A computer-readable storage medium includes a stored computer program; when the computer program is processed qualitatively, the device where the computer-readable storage medium is located is controlled to execute the method for establishing a medical image segmentation model based on harmonious attention provided in the above-mentioned embodiment 1 or 2, and / or the medical image segmentation method based on harmonious attention provided in the above-mentioned embodiment 3.

[0170] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for establishing a medical image segmentation model based on harmonious attention, characterized in that: include: Building the medical image segmentation model and training it using a medical image dataset with annotated segmentation results to complete the establishment of the medical image segmentation model; The medical image is a three-dimensional image; The medical image segmentation model includes: a downsampling module, which is an encoder structure in a U-Net network, used to downsample the medical image N times, compress spatial and depth information, and extract semantic information to obtain a multi-scale primary semantic feature map; a Transformer module, including a Transformer model, used to obtain global and long-range semantic information of the minimum-scale primary semantic feature map, and perform position encoding to obtain an enhanced semantic feature map; and an upsampling module, used to upsample the enhanced semantic feature map N times to obtain a segmentation map; Among them, N is a positive integer greater than 1; the upsampling module includes N upsampling units connected in sequence; the upsampling unit includes an upsampling layer, a jump connection layer and a harmonic attention layer connected in sequence; the upsampling layer is used to upsample the input feature map; the jump connection layer is used to splice the feature map output by the upsampling layer with the primary semantic feature map of the corresponding scale; the harmonic attention layer is used to respectively extract the feature information in different planes of the feature map output by the jump connection layer, and adjust the feature information of each plane to obey the Gaussian distribution, and then fuse the adjusted feature information in each plane into the attention feature map output by the corresponding upsampling unit.

2. The method for establishing a medical image segmentation model based on harmonious attention according to claim 1, characterized in that: The reconciled attention layer includes a hard attention branch; The hard attention branch includes: three sub-attention branches and a first fusion layer; The three sub-attention branches are respectively used to extract feature information in different planes of the input feature map of the harmonized attention layer, and adjust the feature information in the corresponding plane to obey the Gaussian distribution; the sub-attention branches include: an adaptive average pooling layer and a layer normalization layer connected in sequence; The first fusion layer uses the matrix broadcast mechanism to perform point multiplication on the feature information output by the three sub-attention branches to obtain a hard attention map, and outputs it as the attention feature map output by the corresponding upsampling unit.

3. The method for establishing a medical image segmentation model based on harmonious attention according to claim 2, characterized in that: The reconciled attention layer also includes a soft attention branch and a second fusion layer; The soft attention branch includes: a spatial attention sub-branch, a channel attention sub-branch and a third fusion layer; the spatial attention sub-branch extracts the spatial attention map of the input feature map of the harmonic attention layer based on the spatial attention mechanism, the channel attention sub-branch extracts the channel attention map of the input feature map of the harmonic attention layer, and the third fusion layer uses the matrix broadcast mechanism to perform a dot multiplication on the spatial attention map and the channel attention map to obtain a soft attention map; The second fusion layer is used to perform a dot product of the soft attention map and the hard attention map using a matrix broadcasting mechanism before outputting the hard attention map, and output the obtained feature map as the attention feature map output by the corresponding upsampling unit.

4. The method for establishing a medical image segmentation model based on harmonious attention according to any one of claims 1 to 3, characterized in that: The Transformer module includes a plurality of Transformer models cascaded in sequence, and the plurality of Transformer models alternately use an absolute position encoding method and a relative position encoding method to implement position encoding; The absolute position encoding method includes: at the input layer of the Transformer model, setting a learnable position vector for each image block in the image block sequence input to the Transformer model, so as to identify the position information of the corresponding image block in the entire image block sequence; splicing the position vector of each image block with the vectorized representation of the corresponding image block as the input of the layer after the input layer; The relative position encoding method includes: in the multi-head attention layer of the Transformer model, adding the attention matrix output by the multi-head attention layer to the non-learnable relative distance matrix as the input of the layer after the multi-head attention layer; Among them, the image block sequence input to the Transformer model is obtained by dividing the minimum-scale primary semantic feature map or the feature map output by the previous level Transformer model; the relative distance matrix records the distance information between any two image blocks in the image block sequence input to the Transformer model.

5. The method for establishing a medical image segmentation model based on harmonious attention according to claim 4, characterized in that: The relative distance matrix is ​​calculated as follows: Calculate the Euclidean distance between any two image blocks in the image block sequence to obtain a Euclidean distance matrix; Calculate the cosine distance between any two image blocks in the image block sequence to obtain a cosine distance matrix; The Euclidean distance matrix is ​​divided by corresponding elements in the pre-distance matrix to obtain the relative distance matrix.

6. The method for establishing a medical image segmentation model based on harmonious attention according to any one of claims 1 to 3, characterized in that: The medical image segmentation model further includes: a modality fusion module disposed before the downsampling module; the modality fusion module includes an input layer, an output layer and C convolutional layers; C is the modality number of the medical image; wherein: The input layer is used to extract C modal data from medical images; The C convolutional layers are used to perform convolution operations on the data of C modalities with a convolution kernel size of 1, an expansion value of 0, and a step size of 1 to obtain C modal features; The output layer is used to merge C modal features to obtain a medical image after modal fusion; Furthermore, the input image of the downsampling module is the medical image after the modality fusion.

7. The method for establishing a medical image segmentation model based on harmonious attention according to any one of claims 1 to 3, characterized in that: The batch normalization layers in the downsampling modules are replaced by group normalization layers.

8. A medical image segmentation method based on harmonic attention, characterized in that: include: The medical image to be segmented is input into the medical image segmentation model established by the method for establishing a medical image segmentation model based on harmonic attention as described in any one of claims 1 to 7 to obtain a segmentation map.

9. A computer program product, characterized in that It comprises a computer program, which, when processed qualitatively, implements the method for establishing a medical image segmentation model based on harmonious attention as described in any one of claims 1 to 7, and / or the method for medical image segmentation based on harmonious attention as described in claim 8.

10. A computer-readable storage medium, characterized in that: Comprising a stored computer program; when the computer program is processed materially, it controls the device where the computer-readable storage medium is located to execute the method for establishing a medical image segmentation model based on harmonious attention as described in any one of claims 1 to 7, and / or the method for medical image segmentation based on harmonious attention as described in claim 8.

Citation Information

Patent Citations

  • Multi-scale bimodal text image generation method based on generative adversarial network

    CN113421314A

  • Medical image segmentation method based on CNN and Transform fusion network

    CN117173412A

  • Ultrasonic image processing method and system based on cloud computing

    CN117974738A

  • Systems and Methods for Mitigating Vehicle Pose Error Across an Aggregated Feature Map

    US20220032970A1

Cited By

  • Improved image segmentation method and system

    CN119580261A

  • Microscopic hyperspectral image segmentation method based on multi-scale attention fusion

    CN119810116A

  • Medical image analysis method and device of end side model, equipment and storage medium

    CN119904408A

  • Infrared small target detection method

    CN120070849A

  • Infrared small target detection method

    CN120070849B