Bidirectional Attention Parallel Network Segmentation Method for 3D Magnetic Resonance Images

Abstract

The present invention relates to the technical fields of deep learning, computer-aided medical treatment, and medical image processing. Specifically, it is a bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images, which includes: S1. An improved channel attention mechanism module processes the three-dimensional magnetic resonance image to obtain a three-dimensional magnetic resonance image feature map; S2. A parallel spatial attention mechanism module is constructed to process the three-dimensional magnetic resonance image feature map to obtain a three-dimensional magnetic resonance image input block; S3. A double-line parallel mode is constructed to perform feature splicing on the three-dimensional magnetic resonance image input block; S4. Triple consistency training is performed on the three-dimensional magnetic resonance image to complete the segmentation operation of the three-dimensional magnetic resonance image. By constructing a channel attention mechanism and a parallel spatial attention mechanism, the present invention extracts spatial and channel features, combines consistency training and entropy regularization, enables the sub-models to learn from each other, and achieves more accurate judgment; through the evaluation of the bidirectional attention parallel network, it is proved that the present method has generality and robustness.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning, computer-aided medicine, and medical image processing, and particularly relates to a bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images. Background Art

[0002] Medical image segmentation with high accuracy is desired for many downstream clinical applications. Accurate segmentation of medical images can provide rich visible information for doctors' diagnosis. With the booming development of computer computing power and neural networks, a large number of supervised deep learning networks have emerged. Moreover, fully supervised deep learning methods have achieved state-of-the-art segmentation levels in many segmentation tasks. However, fully supervised networks require a large amount of labeled data sets, which is very difficult for medical images. Because medical images can only be understood by professional doctors, and labels can only be added to images in the medical fields familiar to doctors. However, doctors do not need to add labels to the images during medical treatment because they can understand them. This means that it is necessary to specifically seek cooperation with doctors to label specific medical images. This is very costly and time-consuming. And very often, it is not possible to obtain enough data sets with annotations of the required structures.

[0003] Unsupervised domain adaptation can be used to solve this problem. Self-training strategies are adopted for learning. Such methods mainly segment each region by the fact that regions with the same semantic label have a large number of similar features and are dissimilar to regions with different semantic labels. Among them, correctly defining similarity and dissimilarity to obtain better recognition and distinction is crucial. However, although such methods can be trained without labeled data, it is often difficult to match the obtained alternative labels with the target semantic classes. Coupled with the fact that the same structure on different images often has poor similarity or dissimilarity due to various external conditions. These problems always limit the quality of unsupervised learning image segmentation, and the achieved segmentation accuracy is often unqualified. Even though the recently emerging representation learning has slightly improved the segmentation performance, it is still far from being acceptable and applicable.

[0004] The image segmentation method based on semi-supervised learning is a very feasible direction to break through this difficult problem. It only requires a small amount of labeled data sets and a large amount of unlabeled data sets to train a network model with a slightly lower accuracy than fully supervised learning. This method first conducts full-supervised training through a small amount of labeled data sets to generate a model, and uses this model to generate pseudo-labels for a large amount of unlabeled data sets. Then, these pseudo-labels and real labels are used for cyclic training. During the iteration process, better pseudo-labels and model data will be continuously updated. This method is increasingly favored because of the small number of labels it requires and its ability to effectively utilize a large amount of unlabeled data sets. In recent years, various semi-supervised image segmentation networks have emerged in many application scenarios, whether in natural scene images, biomedical image analysis, or even in other industries. It is gradually becoming the trend of future development due to its own advantages.

[0005] Currently, as the network becomes deeper, the feature information that can be extracted becomes more and more abundant. However, most existing semi-supervised segmentation methods have difficulty clearly knowing the features they need to learn, and can only perform extremely inefficient iterative learning again and again, while ignoring a large amount of feature information in terms of space and channels. Thus, the performance is limited in places that are not noticed. Although many people have previously studied and applied the information in terms of space and channels and achieved good results. But not all the information in terms of space and channels has been fully applied. There is still rich information in terms of space and channels waiting to be mined and utilized. For the three-dimensional medical data sets processed currently, the information in terms of space and channels is particularly rich, and the connection relationships between the information are also closer. The present invention can extract more information in the spatial and channel directions to better improve the segmentation performance of the model.

[0006] Although semi-supervised learning has achieved great success in many fields in recent years, it does not mean that the problem of medical image segmentation has been perfectly solved. Although the accuracy of current semi-supervised medical image segmentation can already reach 80%-90%, there is still a gap from being able to be applied clinically. Learning discriminative features from a small amount of annotated data and achieving accurate image segmentation still has a long way to go. In addition, many current network models are for two-dimensional images, and there are relatively few networks specifically for segmenting three-dimensional magnetic resonance images, and the effects are also much worse than two-dimensional network models. To make the network training more efficient, many people add various attentions to the network model. However, these networks do not well extract all useful information in space or channels. Especially for three-dimensional medical images, the information in space and channels is particularly rich, and the connection relationships between information are also closer. There is still rich information in terms of space and channels to be mined and utilized. It is hoped to more comprehensively extract information in space and channels and construct a network with better performance for three-dimensional medical image segmentation. This prompts the construction of an end-to-end joint training framework suitable for three-dimensional medical image segmentation tasks. Summary of the Invention

[0007] Aiming at the deficiencies of the prior art, the present invention provides a bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images. The present invention extracts more spatial and channel feature information by constructing a channel attention mechanism module and a parallel spatial attention mechanism module, and combines the ideas of consistency training and entropy regularization to make the sub-models learn from each other and achieve more accurate judgment; through the evaluation of the bidirectional attention parallel network, it is proved that the method has good use effects and has good versatility and robustness.

[0008] To achieve the above object, the present invention discloses the following technical solutions:

[0009] A bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images, which includes:

[0010] S1: Improve the channel attention mechanism module to process three-dimensional magnetic resonance images and obtain a three-dimensional magnetic resonance image feature map, which specifically includes the following sub-steps:

[0011] S11: Perform global average pooling operation on the input three-dimensional magnetic resonance image X to obtain an aggregated feature; perform fast one-dimensional convolution on an adjacent channel to obtain a corresponding channel weight;

[0012] S12: After activating the channel weight in step S11 through the sigmoid (σ) function, obtain a normalized weight, multiply it with the input three-dimensional magnetic resonance image X, and weight it to the features of each channel of the channel attention mechanism module; determine the channel weight of the features of each channel of the channel attention mechanism module;

[0013] S13: Stack the channel attention mechanism module obtained in step S12 with the 3D convolutional neural network to obtain an improved channel attention mechanism module, and use the improved channel attention mechanism module to process the 3D magnetic resonance image to obtain a 3D magnetic resonance image feature map;

[0014] S2: Construct a parallel spatial attention mechanism module to process the 3D magnetic resonance image feature map to obtain a 3D magnetic resonance image input block, which specifically includes the following sub-steps:

[0015] S21: Generate a spatial attention feature map through the spatial internal relationship of the 3D magnetic resonance image feature map in step S1; perform average pooling and max pooling operations on the spatial attention feature map along the channel axis respectively; and splice the two obtained spatial attention feature maps in the channel dimension;

[0016] S22: Use a convolutional kernel to perform a convolution operation to fuse the 3D magnetic resonance image information, normalize the spatial weight of the spatial attention feature map through the sigmoid function for the result after convolution, and then multiply it with the input spatial attention feature map to obtain a 3D magnetic resonance image input block; the specific process is:

[0017]

[0018] where, M s (F) is the 3D magnetic resonance image input block; F is the 3D magnetic resonance image information; σ is the sigmoid activation function; AvgPool(F) is the average pooling process for the feature block; MaxPool(F) is the max pooling process for the feature block; f 7×7 is to use a convolutional kernel of size 7×7 to perform a convolution operation to fuse the two kinds of information; is the weight obtained by average pooling; is the weight obtained by max pooling;

[0019] S3: Construct a dual-line parallel mode to perform feature splicing on the 3D magnetic resonance image input block;

[0020] Obtain the 3D magnetic resonance image input block obtained in step S2, and obtain a 3D magnetic resonance image first feature block through the first operation mode; obtain a 3D magnetic resonance image second feature block through the second operation mode; then splice the 3D magnetic resonance image first feature block and the 3D magnetic resonance image second feature block in the channel dimension, and then perform a 3D convolution operation once; process to obtain a feature block that contains the channel attention mechanism and the spatial attention mechanism and has the same shape as the 3D magnetic resonance image;

[0021] S4: Perform triple consistency training on the 3D magnetic resonance image input block to complete the segmentation operation of the 3D magnetic resonance image, which specifically includes the following sub-steps:

[0022] S41: Use the input block of the 3D magnetic resonance image obtained by training the model of the bidirectional attention parallel network composed of one encoder and three decoders in training step S3, and use the consistency loss to constrain the bidirectional attention parallel network training model; the three decoders receive the same depth feature F e , and generate three depth features F A , F B and F C ; then, use the sigmoid activation function to obtain the predicted probability outputs P A , F B and F C respectively from the depth features F A , P B and P C ;

[0023] S42: Design a cyclic pseudo-labeling scheme, and convert the prediction deviation into an auxiliary supervision signal through the predicted probability outputs of the three decoders in step S41 to promote model training; first, use the sharpening function to convert the predicted probability outputs P A , P B , P C into soft pseudo-labels sPL A , sPL B and sPL C ; then, make the output results of the three decoders of sPL A , sPL B and sPL C supervise each other for training to achieve mutual consistency for the same input, so that the predicted probability outputs P A , P B , P C are consistent and have low entropy; finally, use the cross-entropy of the three decoders and the consistency loss to construct the total training loss function Loss of the bidirectional attention parallel network training model to complete the segmentation of the 3D magnetic resonance image.

[0024] Preferably, the channel attention mechanism module in step S12 is processed in parallel by two parallel sub-modules. The first sub-module is an adaptive average pooling module, and the second sub-module is an adaptive maximum pooling processing module; after continuous fast convolution, the results obtained by the two sub-modules are concatenated in the channel dimension and then a three-dimensional convolution operation is performed; the information features of the two sub-modules are fused; then it is activated through the sigmoid function, and the activated weight is multiplied by the input 3D magnetic resonance image X.

[0025] Preferably, the first feature block of the three-dimensional magnetic resonance image obtained by processing in the first operation mode in step S3 is specifically as follows: the first operation mode is that the input block of the three-dimensional magnetic resonance image is subjected to a convolution operation with a channel attention mechanism module added, and then a downsampling operation is performed to obtain the first feature block of the three-dimensional magnetic resonance image including channel aspect attention.

[0026] Preferably, the second feature block of the three-dimensional magnetic resonance image obtained by processing in the second operation mode in step S3 is specifically as follows: the second operation mode is that the input block of the three-dimensional magnetic resonance image is subjected to ordinary convolution and downsampling operations, and then processed by a parallel spatial attention mechanism module to obtain the second feature block of the three-dimensional magnetic resonance image including spatial aspect attention.

[0027] Preferably, the bidirectional attention parallel network in step S4 has four layers. The operation process of the first-layer parallel network is jointly realized based on step S1, step S2, and step S3; the operation processes of the second, third, and fourth-layer parallel networks only adopt the parallel spatial attention mechanism in step S2; in the first-layer parallel network, the feature block obtained by three-dimensional convolution is used for skip connection.

[0028] Preferably, the three decoders in step S41 are specifically as follows:

[0029] The first decoder predicts the output by performing an upsampling operation in the transposed convolution;

[0030] The second decoder performs trilinear interpolation operation and then performs an upsampling operation in the transposed convolution to predict the output;

[0031] The third decoder uses nearest neighbor interpolation to expand the previously transmitted feature image, performs three-dimensional convolution operation and performs an upsampling operation to predict the output.

[0032] Preferably, the sharpening function in step S42 is specifically as follows:

[0033]

[0034] Among them, sPL is the soft pseudo-label; P is the probability output; T is a constant used to control the sharpening temperature.

[0035] Preferably, the total training loss function Loss of the bidirectional attention parallel network training model in step S42 is obtained by weighted addition of the segmentation loss function L seg and the consistency loss function L c , specifically as follows:

[0036] The segmentation loss function L seg is:

[0037] L seg = Dice(P A , Y)+ Dice(PB , Y) + Dice(P C , Y);

[0038] Among them, L seg is the segmentation loss function; P A is the first predicted probability output; P B is the second predicted probability output; P C is the third predicted probability output; Y is the ground truth; Dice is the segmentation loss;

[0039] The consistency loss function L c is:

[0040] L c = L2(P A , sPL B ) + L2(P A , sPL C ) + L2(P B , sPL A ) + L2(P B , sPL C ) + L2(P C , sPL A ) + L2(P C , sPL B );

[0041] Among them, L c is the consistency loss function; sPL A is the first soft pseudo-label; sPL B is the second soft pseudo-label; sPL C is the third soft pseudo-label; L2 is the mean squared error loss function;

[0042] The total training loss function Loss of the training model is:

[0043] Loss = L seg + λ × L c ;

[0044] Among them, Loss is the total training loss function of the training model; λ is the consistency loss weight.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The improved channel attention mechanism designed by the present invention enables the training model to pay more attention to the channel information that is more useful for the current task by learning the channel weights after adaptive average pooling and adaptive max pooling; and only by adding a small amount of parameter information, a significant performance improvement can be brought.

[0047] (2) The present invention synergistically operates through the designed parallel spatial attention mechanism, splicing together the results after the parallel operation of max pooling and average pooling, and adding the previous input block to the feature block, so as to more detailedly understand the spatial information of three-dimensional magnetic resonance images.

[0048] (3) The present invention is composed of an encoder and three different decoders, and uses consistency loss to constrain a result with consistent input and output; designs a cyclic pseudo-label scheme, converts the prediction deviation into an auxiliary supervision signal through the prediction outputs of three different decoders to promote model training, and learns more information by learning the differences between each other, thereby improving the performance of the training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of the bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images of the present invention;

[0050] Figure 2 is a structural diagram of the bidirectional attention parallel network for three-dimensional magnetic resonance image segmentation of the present invention;

[0051] Figure 3 is a schematic diagram of the model of the efficient channel attention mechanism network of the present invention;

[0052] Figure 4 is a schematic diagram of the model of the improved channel attention mechanism network of the present invention;

[0053] Figure 5 is a convolutional model diagram of the improved channel attention mechanism of the present invention;

[0054] Figure 6 is a schematic diagram of the model of the parallel spatial attention mechanism network of the present invention;

[0055] Figure 7 is a comparison diagram of the image segmentation results of various methods of the present invention under two different supervision settings on the LA database. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] Hereinafter, exemplary embodiments, features, and aspects of the present invention will be described in detail with reference to the drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0057] The embodiments of the present invention analyze three-dimensional left atrial magnetic resonance images and provide a bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images, as Figure 1As shown, the improved channel attention mechanism module processes the three-dimensional left atrial magnetic resonance image to obtain the three-dimensional left atrial magnetic resonance image feature map; constructs a parallel spatial attention mechanism module to process the three-dimensional left atrial magnetic resonance image feature map to obtain the three-dimensional left atrial magnetic resonance image input block; constructs a dual-line parallel mode to perform feature splicing on the three-dimensional left atrial magnetic resonance image input block; performs triple consistency training on the three-dimensional left atrial magnetic resonance image to complete the segmentation operation of the three-dimensional left atrial magnetic resonance image; the specific steps include:

[0058] Step S1: The improved channel attention mechanism module processes the three-dimensional left atrial magnetic resonance image to obtain the three-dimensional left atrial magnetic resonance image feature map.

[0059] Step S11: Perform global average pooling operation on the input three-dimensional left atrial magnetic resonance image X to obtain the aggregated feature; as Figure 3 is the model schematic diagram of the efficient channel attention mechanism network of the present invention; the corresponding channel weight is obtained by performing fast one-dimensional convolution on an adjacent channel through the efficient channel attention mechanism network.

[0060] The embodiment of the present invention processes the three-dimensional left atrial magnetic resonance image. First, perform preprocessing operations, enlarge the edges of the three-dimensional left atrial magnetic resonance image, and then crop it to the target size according to requirements. After that, further normalize it to zero mean and unit variance. Randomly crop the obtained image, and the size of each image is cropped to H×W×D = 112×112×80. That is, each three-dimensional left atrial magnetic resonance image is composed of 80 two-dimensional image slices, and these 80 slices are not completely adjacent; for example, the three-dimensional left atrial magnetic resonance image dataset of the left atrium is obtained by cutting a heart into 80 equidistant images; the distance between every two adjacent slices is set to about 3 millimeters, that is, there is a part of the gap between each slice and each slice. And this module is the channel attention mechanism, the convolution kernel size of k is 3, and the three channels it covers are not necessarily on the same slice. It is precisely because of this part of the spatial gap that errors are caused. Finally, perform data augmentation by using two-dimensional random rotation and flipping operations, input the obtained three-dimensional left atrial magnetic resonance image data to the preset position, and run the network model proposed by the embodiment of the present invention to prepare for the segmentation prediction task.

[0061] As Figure 3The efficient channel attention mechanism network shown. In an embodiment of the present invention, the size of the convolutional kernel k is fixed to 1. By processing slice by slice, there is no such gap in this part, thus avoiding the error it brings, and the subsequent effect is significantly improved. Although many studies have proven that appropriate cross-channel interaction learning is beneficial for training, this is okay for two-dimensional images, but not so good for three-dimensional left atrial magnetic resonance images. High-dimensional channels have longer-distance interactions, while low-dimensional channels have shorter-distance interactions. For three-dimensional left atrial magnetic resonance images, they cannot be directly processed. The last two dimensions of 1 after the average pooling operation are squeezed out; these dimensions are temporarily ignored to facilitate subsequent processing, and these dimensions will be restored later.

[0062] Step S12: After activating the channel weights in step S11 through the sigmoid (σ) function, the normalized weights are obtained, and then multiplied by the input three-dimensional left atrial magnetic resonance image X; the previously obtained normalized weights are weighted to the features of each channel of the channel attention mechanism module; the channel weights of the features of each channel of the channel attention mechanism module are determined; the channel attention mechanism module is processed in parallel through two parallel sub-modules. The first sub-module is an adaptive average pooling module, and the second sub-module is an adaptive max pooling processing module; after consecutive fast convolutions, the results obtained by the two sub-modules are concatenated in the channel dimension, and then a three-dimensional convolution operation is performed; the information features of the two sub-modules are fused; then, after activation through the sigmoid function, the activated weights are multiplied by the input three-dimensional left atrial magnetic resonance image X.

[0063] As Figure 4 Shown is a schematic diagram of the model of the improved channel attention mechanism network of the present invention; in the embodiment, global average pooling is no longer used to obtain the weights. Experiments have proven that: adaptive average pooling often has better effects. Moreover, average pooling can obtain a part of the information features, while max pooling can obtain another part of the useful feature information. Therefore, a method of parallel processing through two routes is adopted. One side uses adaptive average pooling, and the other side uses adaptive max pooling processing. After consecutive fast convolutions with k = 1, the results obtained on both sides are concatenated in the channel dimension, and then a three-dimensional convolution operation is performed. In this way, the information features extracted on both sides are fused together; then, after activation through the sigmoid function, the activated weights are multiplied by the original three-dimensional left atrial magnetic resonance image X. In this way, the obtained training module will pay more attention to the feature channels that are useful for the current task and suppress the feature channels that are not useful for the current task.

[0064] Step S13: Stack the above channel attention mechanism module with the three-dimensional convolutional neural network to obtain an improved channel attention mechanism module, as Figure 5 shown in the improved channel attention mechanism convolution model diagram of the present invention; obtain the three-dimensional left atrial magnetic resonance image feature map through the improved channel attention mechanism module. In the embodiment, this module is used to replace the traditional convolution block, and this network is processed in parallel with the running route of the ordinary three-dimensional convolution network, so that the network can extract more channel information and adaptively weight each channel to emphasize or suppress different feature channels according to the needs of the task.

[0065] Step S2: Construct a parallel spatial attention mechanism module to process the three-dimensional left atrial magnetic resonance image feature map to obtain a three-dimensional left atrial magnetic resonance image input block.

[0066] Step S21: Generate a spatial attention feature map through the spatial internal relationship of the three-dimensional left atrial magnetic resonance image feature map in Step S1; perform average pooling and max pooling operations on the input spatial attention feature map along the channel axis respectively; then splice the two obtained feature maps in the channel dimension.

[0067] Since there are certain limitations in the channel attention mechanism in terms of spatial relationships; therefore, the embodiment of the present invention designs a parallel spatial attention mechanism module to make up for this defect; generate a spatial attention feature map through the relationship inside the feature map space; different from the channel attention mechanism, the spatial attention mechanism focuses on "where" the effective information on the feature map is. The previous spatial attention mechanism was to choose one of average pooling or max pooling to obtain the weight. However, average pooling is to extract the average value within each execution range to obtain an information feature of the average type within the spatial region, while max pooling is to extract the maximum value within each execution range to obtain an information feature of the maximum type within the spatial region, and the two are not completely decoupled. Therefore, there is no need to choose one and discard the other; since the max pooling feature is as meaningful as the average pooling feature, and the two feature informations are complementary in some aspects. Obtaining the feature informations of both average pooling and max pooling can enable the network to more clearly understand the spatial information of each region. This has also been successfully confirmed through ablation studies. The schematic diagram of the parallel spatial attention mechanism module is as Figure 6As shown in the figure; first, perform average pooling and max pooling operations on the input feature block along the channel axis respectively; it is proved that pooling along the channel axis is helpful for obtaining special information of the region; then, splice the two pooled feature maps in the channel dimension. After that, use a convolutional kernel of size 7×7 to perform convolution operations to fuse information, and the shape of the feature map changes from [b, 2, h, w, d] to [b, 1, h, w, d]. At this time, pass the result of the convolution through the sigmoid function to normalize the spatial weight of the feature map, and then multiply the newly obtained weight by the input feature block. Finally, a new feature block containing special attention to the space is obtained.

[0068] Step S22: Use a convolutional kernel to perform convolution operations to fuse the three-dimensional left atrial magnetic resonance image information, pass the result of the convolution through the sigmoid function to normalize the spatial weight of the spatial attention feature map, and then multiply it by the input spatial attention feature map; the specific process is as follows:

[0069]

[0070] Among them, M s (F) is the input block of the three-dimensional left atrial magnetic resonance image; F is the three-dimensional left atrial magnetic resonance image information; σ is the sigmoid activation function; AvgPool(F) is the average pooling process for the feature block; MaxPool(F) is the max pooling process for the feature block; f 7×7 is to use a convolutional kernel of size 7×7 to perform convolution operations to fuse the two kinds of information; is the weight obtained by average pooling; is the weight obtained by max pooling.

[0071] However, through multiple experiments, it is found that using such feature blocks not only does not have any performance gain, but the effect of the network becomes worse instead; originally, although the feature block that fuses the two weights does have greater attention to some more useful parts, it has deviated greatly from the original input feature block. This is not adding attention, but adding a constraint, resulting in the loss of a lot of original information features. Therefore, the original input three-dimensional left atrial magnetic resonance image X is added to the finally obtained block containing special attention. In this way, the finally obtained block not only contains two aspects of attention in the space, but also retains some original feature information, and experiments also prove that such an effect is indeed better.

[0072] Step S3: Construct a double-line parallel mode to perform feature splicing on the input block of the three-dimensional left atrial magnetic resonance image.

[0073] It has been found through experiments in the present invention that the results obtained after the feature blocks are processed by these two modules successively are not very good. Because the feature information noted by the two modules is both the same and different, after being put together, the information is disordered, which ultimately leads to such a result. Therefore, a processing method of parallel operation of two lines is set up.

[0074] The three-dimensional left atrial magnetic resonance image input block obtained in step S2 is acquired and processed by the first operation mode to obtain the first feature block of the three-dimensional left atrial magnetic resonance image. Specifically, the first operation mode is that the three-dimensional left atrial magnetic resonance image input block undergoes a convolution operation with a channel attention mechanism module added, and then a downsampling operation to obtain the first feature block of the three-dimensional left atrial magnetic resonance image with attention in the channel aspect.

[0075] It is processed by the second operation mode to obtain the second feature block of the three-dimensional left atrial magnetic resonance image. Specifically, the second operation mode is that the three-dimensional left atrial magnetic resonance image input block undergoes ordinary convolution and downsampling operations, and then is processed by a parallel spatial attention mechanism module to obtain the second feature block of the three-dimensional left atrial magnetic resonance image with attention in the spatial aspect.

[0076] Then, the first feature block of the three-dimensional left atrial magnetic resonance image and the second feature block of the three-dimensional left atrial magnetic resonance image are concatenated in the channel dimension, and then a three-dimensional convolution operation is performed again; a feature block is processed that includes a channel attention mechanism and a spatial attention mechanism, and the shape of the three-dimensional left atrial magnetic resonance image remains unchanged.

[0077] Step S4: Perform triple consistency training on the three-dimensional left atrial magnetic resonance image to complete the segmentation operation of the three-dimensional left atrial magnetic resonance image.

[0078] Step S41: Use a bidirectional attention parallel network training model composed of one encoder and three decoders to train the three-dimensional left atrial magnetic resonance image input block obtained in step S3, and use consistency loss to constrain the bidirectional attention parallel network training model to obtain a result with consistent input and output; the bidirectional attention parallel network has four layers. The operation process of the first layer of the parallel network is jointly realized based on step S1, step S2, and step S3; the operation processes of the second, third, and fourth layers of the parallel network only adopt the route of the parallel spatial attention mechanism in step S2; in the first layer of the parallel network, the feature block obtained through ordinary three-dimensional convolution is used for skip connection, rather than the feature block obtained after being processed by the improved channel attention module.

[0079] The three decoders receive the same depth feature F e , and generate three features F A , F B and F C ; then, the sigmoid activation function is used to separately obtain from the depth feature FA , F B and F C to obtain the probability output P A , P B and P C ; The three decoders are specifically: The first decoder performs upsampling through transposed convolution for predictive output; The second decoder replaces the upsampling by transposed convolution with trilinear interpolation operation and then performs convolution for upsampling for 3D left atrial magnetic resonance image classification; The third decoder uses nearest neighbor interpolation to expand the previously transmitted feature image and then performs 3D convolution operation for upsampling for 3D left atrial magnetic resonance image classification.

[0080] Step S42: Design a cyclic pseudo-labeling scheme. Through the predictive outputs of the three decoders in step S41, convert the prediction deviation into an auxiliary supervision signal to promote model training; First, use the sharpening function to convert the probability outputs P A , P B , P C into soft pseudo-labels sPL A , sPL B and sPL C ; The sharpening function is specifically:

[0081]

[0082] where sPL is the soft pseudo-label; P is the probability output; T is a constant used to control the sharpening temperature.

[0083] Then, make sPL A , sPL B and sPL C mutually supervise the output results of the three decoders for training to achieve mutual consistency for the same input, making the predictive probability outputs P A , P B , P C be consistent and of low entropy; Finally, use the cross-entropy and consistency losses of the three decoders to construct the total loss function Loss of the training model of the bidirectional attention parallel network, which is obtained by weighted summation of the segmentation loss function L seg and the consistency loss function L c , specifically:

[0084] The segmentation loss function L seg is:

[0085] L seg = Dice(P A , Y)+ Dice(P B , Y)+ Dice(P C , Y);

[0086] Among them, L seg is the segmentation loss function; P A is the first probability output; P B is the second probability output; P C is the third probability output; Y is the ground truth; Dice is the segmentation loss.

[0087] The consistency loss function L c is as follows:

[0088] L c = L2(P A , sPL B ) + L2(P A , sPL C ) + L2(P B , sPL A ) + L2(P B , sPL C ) + L2(P C , sPL A ) + L2(P C , sPL B );

[0089] Among them, L c is the consistency loss function; sPL A is the first soft pseudo-label; sPL B is the second soft pseudo-label; sPL C is the third soft pseudo-label; L2 is the mean squared error loss function.

[0090] The total training loss function Loss of the training model is:

[0091] Loss = L seg + λ × L c ;

[0092] Among them, Loss is the total training loss function of the training model; λ is the consistency loss weight.

[0093] As Figure 2 shown is the structural diagram of the bidirectional attention parallel network for three-dimensional left atrial magnetic resonance image segmentation of the present invention; the performance of the bidirectional attention parallel network training model is improved through the total training loss function Loss, the segmentation of three-dimensional left atrial magnetic resonance images is completed, the accuracy of medical three-dimensional left atrial magnetic resonance image segmentation is significantly improved, and a new idea is provided for computer-aided diagnosis technology.

[0094] The embodiments of the present invention are evaluated using the LA database of the atrial segmentation challenge. This dataset consists of 100 labeled three-dimensional gadolinium-enhanced left atrial magnetic resonance images. 80 of these samples are used for training, and the remaining 20 samples are used for validation. In preprocessing, first, three-dimensional left atrial magnetic resonance images with enlarged edges are obtained, and then they are cropped to the target size according to requirements. After that, they are further normalized to zero mean and unit variance. During training, the previously obtained images are randomly cropped into three-dimensional patches with a size of 112×112×80. Two-dimensional random rotation and flipping operations are used for data augmentation. The batch size is set to 4, and each batch contains two labeled images and two unlabeled images. The temperature constant T is set to 0.1, and the weight λ is set as a Gaussian warm-up function that changes over time. The present invention is trained by the SGD optimizer for 15K iterations, with an initial learning rate of 0.01, and it decays by 10% every 2.5K iterations. During testing, a sliding window with a fixed span of 18×18×4 is used to extract features. After that, the predictions of all patches are recombined into a complete result. Finally, during testing, the average value of P A and P B is used as the final output. All the experiments in this paper are conducted with a fixed random seed under the following same device environment. The computer configuration is: 13th Gen Intel Core i7-13700HX CPU, NVIDIA GeForce RTX 4080 GPU; software: Pytorch 1.13.1+cu117, Python 3.9.16.

[0095] The embodiments of the present invention are compared on the LA database: UA-MT, SASSNet, DTC, DAP, LG-ER-MT, BCP, MC-Net and DUWM. For fairness, the experimental settings are the same as those of the recent MC-Net method. 10% and 20% of the labeled data are used to train the model respectively, and the rest are unlabeled data. As Figure 7 is the comparison chart of the image segmentation results of multiple methods of the present invention under two different supervision settings on the LA database. Figure 7 The first column of Figure 7 is the original image, and the 2nd to 5th columns from left to right show the image segmentation results of SASSNet, MC-Net, BCP and the present method under two different supervision settings on the LA database respectively. The results using 10% of the labels and using 20% of the labels are shown in the upper and lower parts of the figure respectively, and the corresponding ground truth on the LA database is placed in the last column for comparison. As Figure 7 shown, whether in the 3D view or the 2D view, the model of the present method generates a more complete left atrium than all existing SOTA methods. Note that the embodiments of the present invention do not use any morphological algorithm post-processing module to process the result images.

[0096] Embodiments of the present invention use four different metrics to evaluate model performance, namely Dice Similarity Score (DSC), Jaccard, Hausdorff Distance 95 (HD95), and Average Symmetric Distance (ASD). At the same time, the results of V-Net under various different supervision settings (10%, 20%, all labeled data) are given as a reference. This method achieves the best performance on all four evaluation metrics, far exceeding eight other recent SOTA methods. Moreover, this method achieves an excellent Dice score of 91.52% by only using 20% of the labeled data, exceeding the second place by 0.96, achieving a new state-of-the-art segmentation performance on the LA database. For fair comparison, these results have not undergone any post-processing. Experiments show that the embodiments of the present invention can extract more spatial and channel information, thereby achieving more accurate segmentation. And it can be seen from Figure 7 that this method can more accurately and finely segment the target organ, especially in some regions that are easily misidentified. The present invention can be used to assist doctors in disease diagnosis, accurately segment abnormal regions in magnetic resonance images, such as the accurate positioning and analysis of tumors, lesions, organ abnormalities, etc.

[0097] The beneficial effects of the present invention are as follows: The present invention provides a bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images. Through the designed improved channel attention mechanism, the training model pays more attention to the channel information that is more useful for the current task and requires less parameter information; it cooperates with the designed spatial attention mechanism, and splices the results after the parallel operation of max pooling and average pooling to obtain spatial information; finally, it is composed of an encoder and three different decoders to train the model, thereby improving the performance of the training model; the present invention is evaluated on datasets in different fields and compared with several current state-of-the-art semi-supervised methods, indicating that this method produces the most efficient performance and has a certain degree of generality and robustness; the network model designed by the present invention can be used for other three-dimensional medical image datasets, bringing significant performance gains to three-dimensional medical image segmentation, and the actual use effect is very good.

[0098] The above-described embodiments are only descriptions of the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A bidirectional attention parallel network segmentation method for three-dimensional magnetic resonance images, characterized in that: It includes: S1: Improve the channel attention mechanism module to process the three-dimensional magnetic resonance image and obtain the three-dimensional magnetic resonance image feature map, which specifically includes the following sub-steps: S11: Perform a global average pooling operation on the input three-dimensional magnetic resonance image X to obtain aggregate features; perform a fast one-dimensional convolution on an adjacent channel to obtain the corresponding channel weight; S12: After activating the channel weights in step S11 by the sigmoid (σ) function, a normalized weight is obtained, which is multiplied by the input three-dimensional magnetic resonance image X; the normalized weight obtained above is weighted to the features of each channel of the channel attention mechanism module; The channel attention mechanism module in step S12 is processed in parallel by two parallel submodules, the first submodule is an adaptive average pooling module, and the second submodule is an adaptive maximum pooling processing module; After continuous fast convolution, the results of the two submodules are concatenated in the channel dimension and then subjected to a three-dimensional convolution operation to fuse the information features of the two submodules. S13: stacking the channel attention mechanism module obtained in step S12 with the three-dimensional convolutional neural network to obtain an improved channel attention mechanism module, and using the improved channel attention mechanism module to process the three-dimensional magnetic resonance image to obtain a three-dimensional magnetic resonance image feature map; S2: Construct a parallel spatial attention mechanism module to process the 3D MRI feature map and obtain the 3D MRI input block, which specifically includes the following sub-steps: S21: Generate a spatial attention feature map through the spatial internal relationship of the three-dimensional magnetic resonance image feature map in step S1; perform average pooling and maximum pooling operations on the spatial attention feature map along the channel axis; and splice the two obtained spatial attention feature maps in the channel dimension; S22: Use the convolution kernel to perform convolution operation to fuse the three-dimensional magnetic resonance image information, normalize the spatial weight of the spatial attention feature map through the sigmoid function, and then multiply it with the input spatial attention feature map to obtain the three-dimensional magnetic resonance image input block; the specific process is: Among them, M s (F) is the 3D MRI input block; F is the spatial attention feature map; σ is the sigmoid activation function; AvgPool(F) is the average pooling process for the spatial attention feature map; MaxPool(F) is the maximum pooling process for the spatial attention feature map; f 7×7 A convolution operation is performed using a 7×7 convolution kernel to fuse the two types of information; The weights obtained for average pooling; The weights obtained for max pooling; S3: construct a two-line parallel mode to perform feature stitching on the 3D magnetic resonance image input block; The three-dimensional magnetic resonance image input block obtained in step S2 is obtained, and processed by the first operation mode to obtain a first feature block of the three-dimensional magnetic resonance image; processed by the second operation mode to obtain a second feature block of the three-dimensional magnetic resonance image; then the first feature block of the three-dimensional magnetic resonance image and the second feature block of the three-dimensional magnetic resonance image are spliced ​​in the channel dimension, and then a three-dimensional convolution operation is performed again; the processing is performed to obtain a feature block that includes a channel attention mechanism and a spatial attention mechanism and has a consistent shape of the three-dimensional magnetic resonance image; In step S3, the first feature block of the three-dimensional magnetic resonance image is obtained by processing in the first operation mode, specifically: the first operation mode is that the three-dimensional magnetic resonance image input block is subjected to a convolution operation of adding a channel attention mechanism module, and then a downsampling operation is performed to obtain the first feature block of the three-dimensional magnetic resonance image including channel attention; The second operating mode is used to process the three-dimensional magnetic resonance image second feature block, specifically: the second operating mode is that the three-dimensional magnetic resonance image input block is processed by the parallel spatial attention mechanism module after the ordinary convolution and downsampling operations are performed, so as to obtain the three-dimensional magnetic resonance image second feature block including spatial attention; S4: Perform triple consistency training on the 3D magnetic resonance image input block to complete the segmentation operation of the 3D magnetic resonance image, which specifically includes the following sub-steps: S41: Use one encoder and three decoders to form a bidirectional attention parallel network training model to train the feature block obtained in step S3, and use consistency loss to constrain the bidirectional attention parallel network training model; the three decoders receive the same deep feature F from the encoder e , generate three deep features F A , F B and F C ; Then, the sigmoid activation function is used to extract the deep features F A , F B and F C The predicted probability output P is obtained A , P B and P C ; S42: Design a cyclic pseudo-labeling scheme, through the three decoder prediction probability outputs in step S41, to convert the prediction bias into an auxiliary supervision signal to facilitate model training; first, use a sharpening function to convert the predicted probability output P A , P B , P C Convert to soft pseudo label sPL A 、sPL B and sPL C ; Then, make sPL A 、sPL B and sPL C The output results of the three decoders supervise each other for training, so as to maintain consistency with the same input and make the predicted probability output P A , P B , P C is consistent and low entropy; finally, the three decoder cross entropy and consistency loss are used to construct the total training loss function Loss of the bidirectional attention parallel network training model to complete the segmentation of the three-dimensional magnetic resonance image.

2. The method for segmenting three-dimensional magnetic resonance images using a bidirectional attention parallel network according to claim 1, characterized in that: The bidirectional attention parallel network in step S4 has four layers, and the operation process of the first layer of the parallel network is implemented based on steps S1, S2 and S3; the operation process of the second, third and fourth layers of the parallel network only uses the parallel spatial attention mechanism in step S2; In the first layer of parallel networks, feature blocks obtained by 3D convolution are used for skip connections.

3. The method for segmenting three-dimensional magnetic resonance images using a bidirectional attention parallel network according to claim 1, characterized in that: The three decoders in step S41 are specifically: The first decoder performs an upsampling operation on the transposed convolution to predict the output; The second decoder performs a trilinear interpolation operation and then a transposed convolution upsampling operation to predict the output; The third decoder uses nearest neighbor interpolation to expand the previously transmitted feature image, performs a three-dimensional convolution operation, and performs an upsampling operation to predict the output.

4. The method for segmenting three-dimensional magnetic resonance images using a bidirectional attention parallel network according to claim 1, characterized in that: The sharpening function in step S42 is specifically: Among them, sPL is the soft false label; P is the probability output; T is the constant used to control the sharpening temperature.

5. The method for segmenting three-dimensional magnetic resonance images using a bidirectional attention parallel network according to claim 1, characterized in that: In step S42, the total training loss function Loss of the bidirectional attention parallel network training model is to subtract the segmentation loss function L seg And the consistency loss function L c The weighted addition of is obtained, specifically: The segmentation loss function L seg for: L seg =Says(P A ,Y)+Dice(P B ,Y)+Dice(P C ,AND); Among them, L seg is the segmentation loss function; P A is the first predicted probability output; P B is the second predicted probability output; P C is the third predicted probability output; Y is the ground truth; Dice is the segmentation loss; The consistency loss function L c for: L c =L2(P A ,sPL B )+L2(P A ,sPL C )+L2(P B ,sPL A )+L2(P B ,sPL C )+L2(P C ,sPL A )+L2(P C ,sPL B ); Among them, L c is the consistency loss function; sPL A It is the first soft pseudo label; sPL B The second soft pseudo label; sPL C is the third soft false label; L2 is the mean square error loss function; The total training loss function Loss of the training model is: Loss=L seg +λ×L c ; Among them, Loss is the total training loss function of the training model; λ is the consistency loss weight.

Images