A fetal heart segmentation method based on self-supervised transfer and global attention
By adopting the self-supervised migration and global attention module methods in fetal cardiac ultrasound image segmentation task, the problems of insufficient feature fusion and overfitting are solved, and high-quality cardiac segmentation results and good generalization capabilities are achieved.
Patent Information
- Application Number
- CN202210210353.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-04
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2042-03-04
AI Technical Summary
The prior art has problems such as insufficient feature fusion, inconsistent overfitting and segmentation labels in fetal cardiac ultrasound image segmentation task, resulting in insufficient segmentation performance and generalization ability.
Using a method based on self-supervised migration and whole-domain attention, the characteristic information of the channel, space and location domains is fused through the whole-domain attention module, and self-supervised training is used to optimize network parameters to improve the performance of the segmentation model.
It achieves more efficient feature fusion, reduces overfitting, improves segmentation performance and generalization capabilities, and achieves the best results in Dice, HD, IoU, and Sensitivity indicators.
Smart Images
Figure CN114663444B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical image segmentation in artificial intelligence, and relates to a fetal heart segmentation method based on self-supervised migration and global attention. The feature information in unlabeled data is learned through a self-supervised method, and the global attention mechanism is used to combine the feature information in the channel, space, and position domains to obtain high-quality heart segmentation results. Background Art
[0002] Congenital heart disease is the disease with the highest mortality and morbidity rate in newborns. According to the "China Birth Defect Prevention Report", the incidence of congenital heart disease in Chinese fetuses is estimated to be about 0.74%. In prenatal ultrasound examinations of the fetus, doctors use ultrasound images to observe the status of the fetal ventricles, atria, and cusps in order to screen for congenital heart disease. Accurately segmenting the fetal heart area of ultrasound images can help doctors quickly locate, identify, and even measure the heart area. Due to the presence of speckle noise, strong artifacts, and differences in the shooting angle of the heart area in two-dimensional ultrasound images, segmenting the fetal heart is a challenging task.
[0003] Deep learning technology has been successfully applied in medical image segmentation tasks of various modalities (CT, MRI, X-ray, ultrasound), and the neural network model based on the encoder-decoder structure of skip connection is one of the effective methods. The U-Net model has been applied to ultrasound segmentation tasks. The U-Net network has excellent characteristics, but its features at the skip connection cannot be efficiently fused. The Attention U-Net model attempts to introduce attention to solve this problem, but it cannot fuse features from the channel and spatial scale at the same time, and it also lacks long-distance attention.
[0004] In the process of training the medical image segmentation model, the optimization of the neural network requires continuous fitting of image labels. After multiple training cycles, the model will overfit on the training set, which will affect the generalization ability of the model on the test set. Ultrasound images are characterized by high noise, low contrast, and many artifacts. The imaged tissues and organs are highly similar, and the boundaries of tissues and organs are unclear. In the ultrasound image segmentation task, the production of segmentation labels has different tendencies among different physicians due to the fuzziness of organ boundaries, which will produce inconsistent segmentation labels. Summary of the invention
[0005] In order to more efficiently fuse the encoding layer features and decoding layer information at the jump connection, the present invention develops a global attention block. This module first fuses the shallow semantics of the encoding layer with the deep semantics of the decoding layer, then fuses the features at a spatial scale, and finally uses an attention module with long-distance dependencies. In response to the above-mentioned problems existing in overfitting and segmentation labels, a fetal heart segmentation method based on self-supervised transfer and global attention is used to compare the differences between ultrasound images from unlabeled ultrasound image data so that the network learns network parameters that are beneficial to the ultrasound image segmentation task. These parameters are beneficial to the subsequent fine-tuning of the segmentation performance. At the same time, the method of the present invention, combined with the global attention module, can fuse the encoding layer features and decoding layer information from multiple angles, and can pay attention to the mutual information between channels, spaces, and positions, thereby improving the segmentation performance.
[0006] The present invention aims to solve the above problems of the prior art and proposes a fetal heart segmentation method based on self-supervised migration and global attention, which includes the following steps:
[0007] 1) Preprocess the images in the fetal heart ultrasound image dataset.
[0008] 2) Build a global attention module.
[0009] 3) Use the global attention module obtained in step 2) to build a fetal heart segmentation model.
[0010] 4) constructing a network identical to the encoder of the fetal heart segmentation model described in step 3), and training and optimizing the network using a self-supervised method on unlabeled fetal heart ultrasound image data to obtain network parameters of the network.
[0011] 5) Using the network parameters obtained in step 4), the network parameters of the encoder of the fetal heart segmentation model are initialized, and the labeled fetal heart ultrasound image data is used to train and optimize the entire fetal heart segmentation model.
[0012] 6) The ultrasound image is transmitted to the fetal heart segmentation model obtained in step 5), and after calculation by the model, a heart segmentation result is obtained.
[0013] In summary, the present invention has the following beneficial technical effects:
[0014] The present invention constructs a fetal heart segmentation method based on self-supervised transfer and global attention, which can not only enable the model to focus on more important feature information from the channel domain, spatial domain, and position domain, but also maintain certain characteristics of the ultrasound image itself, and improve the model's heart segmentation ability and generalization ability. Compared with the other four methods, the present invention achieves the best scores in Dice, HD, IoU, and Sensitivity indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a framework diagram of a fetal heart segmentation method based on self-supervised transfer and global attention proposed by the present invention;
[0016] Figure 2 It is a self-supervised training framework diagram of the present invention;
[0017] Figure 3 is a structural diagram of the global attention module of the present invention;
[0018] Figure 4 It is a prototype system of fetal heart segmentation constructed by the present invention. DETAILED DESCRIPTION
[0019] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and in detail describe the technical solutions in the embodiments of the present invention. The described implementation is only a part of the embodiments of the present invention.
[0020] like Figure 1 As shown, a fetal heart segmentation method based on self-supervised transfer and global attention includes:
[0021] 1) Preprocess the images in the fetal heart ultrasound image dataset. The preprocessing operation includes cropping the ultrasound images. All ultrasound images are cropped to a size of 224*224 under uniform conditions.
[0022] 2) Construct a global attention module. Figure 2 As shown in the figure, the global attention module consists of a channel attention network, a spatial attention network, and a position attention network. The channel attention network, the spatial attention network, and the position attention network are constructed from different dimensions by using global average pooling operations, global maximum pooling operations, and activation function operations. These three networks form the global attention module. The process of the global attention module processing the input feature map X is as follows:
[0023] 1. The feature map X is processed in the channel domain. The feature map X is processed into two c×1×1 features by the average pooling operation based on width and height and the maximum pooling operation based on width and height, respectively, to obtain the channel direction feature. Then the two are processed by the shared fully connected MLP layer to obtain two c×1×1 features, where c represents the number of channels of the feature map. That is, the relevant information between different channels is obtained. Then the two are added and processed by the Sigmoid function to obtain the channel weight M c (X), so that the attention of different channels can be obtained. The whole process is shown in formula (1):
[0024] M c (X) = σ(MLP(AvgPool (w,h) (X))+MLP(MaxPool(w,h) (X))) (1)
[0025] Where σ is the Sigmoid activation function. MLP is a multi-layer shared fully connected layer, AvgPool (w,h) MaxPool is a pooling operation based on width and height for the input feature map. (w,h) To perform the maximum pooling operation based on width and height on the feature map. c (X) is multiplied with the input feature map X to obtain the channel domain fused feature map X′. As shown in formula (2):
[0026] X′=X×M c (X) (2)
[0027] 2. The feature map X′ is processed in the spatial domain. First, the feature map X′ undergoes the channel-based maximum pooling operation and the average pooling operation, and then the two feature results are spliced in the channel direction. Then, a convolution operation is performed to reduce the channel dimension to 1 channel. Finally, the spatial domain weight M is generated through Sigmoid. s (X′), thus obtaining the attention in the spatial domain. The whole process is shown in formula (3):
[0028] M s (X′)=σ(f 7×7 ([AvgPool c (X′); MaxPool c (X′)])) (3)
[0029] Among them, f 7×7 is a 7×7 convolution operation. [;] is a concatenation operation in the channel direction. AvgPool (c) MaxPool is a channel-based pooling operation for feature maps. (c) To perform a channel-based maximum pooling operation on the feature map. Then, the spatial domain weight is multiplied by the input feature map X′ to obtain the spatial domain fused feature map. After that, it is processed by a 3×3 convolution kernel to obtain the feature map X″, as shown in formula (4):
[0030] X″=f 3×3 (X′×M s (X′)) (4)
[0031] 3. The feature map X″ is processed in the position domain. The feature map X″ undergoes the maximum pooling operation and the average pooling operation based on the height, and the maximum pooling operation and the average pooling operation based on the width. That is, the input feature map X″ is encoded using the pool kernel (H, 1) in the horizontal coordinate, where H is the set kernel size. The input feature map X″ is encoded using the pool kernel (1, W) in the vertical coordinate, where W is the set kernel size. Then these four results are concatenated in the height dimension. After the combined result is subjected to a convolution operation, it is split into four position information features in the vertical and horizontal directions. The four features are subjected to Sigmoid to generate the attention weights of the position domain. The whole process is shown in formula (5):
[0032] M a (X″)=σ(f 1×1 (Split(f 1×1 ([AvgPool h (X″); MaxPool h (X″); AvgPool w (X″); MaxPool w (X″)])))) (5)
[0033] Among them, Split is a split operation. Then the four position domain weights M a (X″) is multiplied with the input feature map X″ to obtain the fusion feature of the position direction. That is, as shown in formula (6):
[0034] X″′=X″×M a (X″) (6)
[0035] 3) Use the global attention module obtained in step 2) to build a fetal heart segmentation model. Use the global attention module and skip connection operation to build an encoder-decoder based fetal heart segmentation model. The fetal heart segmentation model consists of an encoder with a downsampling process and a decoder with an upsampling process, where the symmetrical network layers between the encoder and the decoder are connected by skip connections, and the global attention module is added after the splicing operation of the skip connection.
[0036] 4) Construct the same network as the encoder of the fetal heart segmentation model in step 3), and use the self-supervised method to train and optimize the network on the unlabeled fetal heart ultrasound image dataset to obtain the network parameters of the network. Use the momentum contrast self-supervised method to optimize the parameters of the encoder network. Figure 3As shown in the figure, the momentum contrast method mainly provides a large number of negative sample encodings for the contrast loss through queues. Encoder Q has the same network structure as encoder K. Encoder Q generates sample encoding q; encoder K generates sample encoding k. In order to provide a large number of negative sample encodings, a large sample encoding queue is constructed. The ultrasound sample encodings of the current batch are queued, and the ultrasound image sample encodings of the earliest batch in the queue are dequeued. The large number of negative samples is decoupled from the small batch through dynamic changes, so that a smaller batch can be used to train the encoder, and the queue can provide a large number of negative sample encodings k. - The current batch is considered as positive sample k + , and added to the team as negative sample k - The sample codes in the queue come from multiple batches. In order to ensure the consistency of the ultrasound sample codes in the queue, the moving average update method is used to make the encoder K change slowly, and only the gradient descent update is used for the encoder Q.
[0037] 5) Use the network parameters obtained in step 4) to initialize the network parameters of the encoder of the fetal heart segmentation model, and use the labeled fetal heart dataset to train and optimize the entire fetal heart segmentation model. The loss function used is the contrast segmentation loss L CS , as shown in formula (7). The loss function consists of two parts. The first part is used to measure the difference between feature vectors of different categories, and the second part calculates the difference between the segmentation prediction result and the segmentation label of each pixel.
[0038]
[0039] Among them, L CS is the contrast segmentation loss. q is the image sample code. k + Encode the positive image sample. is the negative image sample encoding. M is the size of the queue. τ is the temperature hyperparameter, which is used to control the concentration of distance and make the distance distribution smoother. w and h are the width and height of the image respectively. y ij is the true label at position i, j, y i ' j is the category prediction value at position i and j. The two ultrasound image views generated by using different data augmentation of the same ultrasound image are regarded as positive sample pairs, and the different ultrasound image pairs are regarded as negative sample pairs. Sample encoding k + is the positive sample code corresponding to the sample code q, that is, the compression code generated by the two ultrasound views of the same ultrasound image through the basic network. Sample code The M negative sample codes corresponding to the sample code q are the compressed codes generated by different ultrasound images after passing through the basic network. The dot product operation means using cosine similarity to measure the distance between two sample codes, that is, to measure the difference between ultrasound images.+ The more similar, q·k + The closer the calculation result is to 1, the smaller the difference between the corresponding ultrasound image pairs is. Similarly, q and The more different, The calculation result will be close to 0, the greater the difference between the corresponding ultrasound image pairs. + The more similar The more different the function is, the lower the loss value will be.
[0040] 6) The ultrasound image is passed into the fetal heart segmentation model obtained in step 5), and a probability map is obtained after encoder calculation, skip connection, decoder calculation, global attention calculation and activation function calculation of the model, that is, the final heart segmentation result is obtained.
[0041] like Figure 4 , is a fetal heart segmentation prototype system constructed using the method of the present invention, which shows the results of fetal heart segmentation.
[0042] The above examples should be understood to be used only to explain the present invention and not to limit the protection scope of the present invention. Various changes or modifications made according to the present invention, such equivalent changes and modifications also fall within the scope defined by the claims of the present invention.
Claims
1. A fetal heart segmentation method based on self-supervised transfer and global attention, characterized in that: The following steps are involved: 1) Preprocessing images in a fetal heart ultrasound image dataset; 2) Constructing the global attention module includes using global average pooling operation, global maximum pooling operation, and activation function operation to construct a channel attention network, a spatial attention network, and a position attention network from different dimensions, and using these three networks to form a global attention module; 3) Using the global attention module obtained in step 2) to build a fetal heart segmentation model; 4) constructing a network identical to the encoder of the fetal heart segmentation model in step 3), and training and optimizing the network using a self-supervised method on an unlabeled fetal heart ultrasound image dataset to obtain network parameters of the network; 5) Initializing the network parameters of the encoder of the fetal heart segmentation model using the network parameters obtained in step 4), and using the labeled fetal heart ultrasound image data to train and optimize the entire fetal heart segmentation model, specifically including: migrating the network parameters obtained in step 4) to the encoder of the fetal heart segmentation model, and using the labeled fetal heart data set to train and optimize the entire fetal heart segmentation model, and using the loss function of the contrast segmentation loss L CS , as follows: The loss function consists of two parts. The first part is used to measure the difference between feature vectors of different categories, and the second part calculates the difference between the segmentation prediction result and the segmentation label of each pixel. CS is the contrast segmentation loss, q is the image sample code, k + Encode the positive image sample, is the negative image sample encoding, M is the size of the queue, τ is the temperature hyperparameter, w and h are the width and height of the image respectively, and y ij is the true label at position i, j, y′ ij is the category prediction value at position i, j; 6) The ultrasound image is transmitted to the fetal heart segmentation model obtained in step 5), and after calculation by the model, a heart segmentation result is obtained.
2. A fetal heart segmentation method based on self-supervised transfer and global attention according to claim 1, characterized in that: The preprocessing in step 1) includes cropping the ultrasound image to a uniform size.
3. The fetal heart segmentation method based on self-supervised transfer and global attention according to claim 1, characterized in that: The process of the global attention module processing the input feature map X is as follows: (1) Processing the feature map X in the channel domain: The feature map X is processed into two c×1×1 features by the average pooling operation based on width and height and the maximum pooling operation based on width and height, respectively, to obtain the channel-direction features; Then the two are processed through a shared fully connected MLP layer to obtain two c×1×1 features; then the two are added and processed with the Sigmoid function to obtain the channel weight M c (X), M c (X) is multiplied with the input feature map X to obtain the channel domain fused feature map X′; (2) The feature map X′ is processed in the spatial domain: First, the feature map X′ undergoes the channel-based maximum pooling operation and the average pooling operation, and then the two feature results are spliced in the channel direction. Then, a convolution operation is performed to reduce the channel dimension to one channel, and finally, a sigmoid operation is performed to generate the spatial domain weight M. s (X′), and then the spatial domain weight M s (X′) is multiplied with the input feature map X′ to obtain the spatial domain fused feature map X″; (3) The feature map X″ is processed in the position domain: the feature map X″ undergoes the maximum pooling operation and the average pooling operation based on the height, and the maximum pooling operation and the average pooling operation based on the width, respectively. That is, the input feature map X″ is encoded using the pooling kernel (H, 1) in the horizontal coordinate, and the input feature map X″ is encoded using the pooling kernel (1, W) in the vertical coordinate; then these four results are concatenated in the height dimension, and the combined result is subjected to a convolution operation and then split into four position information features in the vertical and horizontal directions. The four features are subjected to Sigmoid to generate the attention weights of the position domain.
4. The fetal heart segmentation method based on self-supervised transfer and global attention according to claim 1, characterized in that: Step 3) The fetal heart segmentation model consists of an encoder with a downsampling process and a decoder with an upsampling process, wherein the symmetrical network layers between the encoder and the decoder are connected by skip connections, and a global attention module is added after the splicing operation of the skip connections.
5. The fetal heart segmentation method based on self-supervised transfer and global attention according to claim 1, characterized in that: Step 4) The momentum contrast self-supervision method is used in the training optimization network to optimize the parameters of the encoder network. Encoder Q and encoder K have the same network structure. Encoder Q generates sample code q, and encoder K generates sample code k. A sample code queue is constructed, and the ultrasound sample codes of the current batch are queued, while the ultrasound image sample codes of the earliest batch in the queue are dequeued. Encoder K is updated using a moving average, and encoder Q is updated using a gradient descent.
6. A computer-readable storage medium storing a computer program, characterized in that: Run the program to implement the fetal heart segmentation method based on self-supervised transfer and global attention as described in any one of claims 1-5.