Consistent semi-supervised pericardial fat segmentation method based on difference guidance

Abstract

The invention belongs to the technical field of medical image segmentation, and particularly relates to a consistency semi-supervised pericardial fat segmentation method based on difference guidance. Firstly, a data set is acquired and processed; then, replacing VSS blocks of the Mamb-Unet network one by one by using MCM blocks, and further obtaining a teacher model and a student model; in an MCM block, an input tensor is firstly equally divided into two tensors # imgabs0 # and # imgabs1 # in a channel dimension; after the tensor # imgabs2 # is sequentially subjected to convolution, activation and convolution and activation operation, the tensor # imgabs2 # and the tensor # imgabs2 # are subjected to jump connection, and a tensor # imgabs3 # is obtained; processing the tensor # imgabs4 # through two information flow branches to obtain a tensor # imgabs5 #; the tensor # imgabs6 # and the tensor # imgabs7 # are spliced and subjected to convolution, and then jump connection is carried out on the tensor # imgabs6 # and the tensor # imgabs7 # and the input tensor in the MCM block to obtain the output tensor of the MCM block; and finally, performing semi-supervised training on the collaborative mean teacher model based on difference fusion, and using the trained teacher model as a segmentation model for segmentation of pericardial fat. The model pays more attention to fuzzy regions and boundary information of pericardial fat, and the segmentation precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cardiac medical image segmentation, and in particular is a difference-guided consistent semi-supervised pericardial fat segmentation method. Background Art

[0002] The pericardial fat tissue surrounding the heart and coronary arteries is composed of paracardial fat tissue and epicardial fat tissue. Many studies have confirmed that pericardial fat tissue is significantly associated with a variety of cardiovascular diseases. It can promote the occurrence and development of cardiovascular diseases by releasing proinflammatory factors, oxidative stress and endothelial damage, and may also cause local inflammation, left atrial fibrosis and fat infiltration, thereby causing atrial fibrillation. Therefore, the segmentation of pericardial fat tissue plays an important role in the prevention and diagnosis of cardiovascular diseases such as coronary artery disease. However, pericardial fat tissue has various shapes, sizes, locations and discontinuous distribution in cardiac MR images, and its accurate segmentation faces certain challenges.

[0003] In recent years, semantic segmentation technology based on deep learning has made significant progress in the field of medical image analysis, which has greatly improved the accuracy and speed of medical image segmentation. However, these methods still face challenges in dealing with the task of pericardial fat segmentation in cardiac MR images. First, the uniqueness of pericardial fat in cardiac MR images increases the difficulty of segmentation. The boundaries of pericardial fat are usually fuzzy, with low contrast with other cardiac tissues, and are easily confused with adjacent tissues. In addition, its morphology and position vary significantly between individuals, and it is irregularly distributed, making it difficult for the model to accurately extract edge features, prone to missed detection or false detection, resulting in unsatisfactory segmentation results. Second, as one of the semi-supervised methods, the consistency regularization method, especially the model based on the mean teacher framework, emphasizes enhancing the stability of the model through the prediction consistency between the teacher model and the student model. Although this method can alleviate the lack of labeled data to a certain extent, the model often has large prediction fluctuations when dealing with information-intensive and complex feature areas. In addition, the characteristics of pericardial fat in cardiac MR images make the model susceptible to interference from small noise during segmentation, which in turn causes unstable predictions. This volatility reflects the uncertainty of the model in a specific area, making it difficult to further improve the segmentation accuracy. Third, clinical data of cardiac MR images are scarce, especially high-quality annotated data sets. Pixel-level accurate annotation of pericardial fat not only takes a lot of time and manpower, but also requires the annotators to have professional medical knowledge to ensure the accuracy and consistency of the annotation. Since pericardial fat is distributed around the heart and has a complex anatomical position, and it is difficult to distinguish between small areas and boundaries during manual annotation, there are differences in the annotation results between different annotators, which directly limits the effectiveness of model training and makes it difficult to generalize the model to diverse data. Based on the aforementioned problems, the present invention proposes a consistent semi-supervised pericardial fat segmentation method based on difference guidance. Summary of the invention

[0004] In view of the deficiencies in the prior art, the technical problem that the present invention intends to solve is to propose a difference-guided consistent semi-supervised pericardial fat segmentation method.

[0005] The present invention solves the technical problem by adopting the following technical solution: A difference-guided consistent semi-supervised pericardial fat segmentation method, characterized in that the method comprises the following steps: S1, obtaining a plurality of cardiac MR images containing pericardial fat to form a data set, and preprocessing the data set; S2, use MCM blocks to replace the VSS blocks of the Mamba-Unet network one by one, and then obtain the teacher model and the student model; In the MCM block, the input tensor is first split equally into two tensors along the channel dimension and ; Tensor After convolution, activation, convolution and activation operations, a jump connection is made with itself to obtain the tensor ; Tensor Enter two information flow branches, one of which includes a linear layer, a depth-separable convolutional layer, a SS2D block, and a normalization layer connected in sequence, and the other includes a linear layer and an activation layer connected in sequence; multiply the output tensors of the two information flow branches to obtain the tensor ; The tensor and After concatenation, a 1×1 convolution is performed, and then a jump connection is performed with the input tensor in the MCM block to obtain the output tensor of the MCM block; S3, semi-supervised training of the collaborative mean teacher model is performed based on difference fusion, and the trained teacher model is used as a segmentation model for the segmentation of pericardial fat; The collaborative mean teacher model includes a teacher model and two student models. For unsupervised training, the unlabeled image is input into the teacher model for segmentation. After adding different noises, the unlabeled image is input into the two student models for segmentation. The segmentation map generated by the teacher model is XORed pixel by pixel with the segmentation map generated by the two student models to obtain the first difference map and the second difference map. The process is expressed as: (7) (8) In the formula, represents the difference map, represents the argmax function, Represents the student model The generated segmentation map, are the learnable parameters of the student model, represents the unlabeled image after adding noise, Represents the teacher model The generated segmentation map, are the learnable parameters of the teacher model, represents an unlabeled image, Represents the exclusive OR operation; Unsupervised loss includes consistency loss and cross pseudo-supervision loss ; The calculation formula of consistency loss is as follows: (9) In the formula, represents the consistency loss, represents the consistency weight; The calculation formula of cross pseudo-supervision loss is as follows: (11) In the formula, represents the cross pseudo-supervision loss, Indicates dice loss; Total training loss The calculation formula is as follows: (13) In the formula, is the monitoring loss, is the unsupervised loss, is the weight factor.

[0006] Furthermore, for supervised training, different noises are added to the labeled images and then input into two student models for segmentation. The supervised loss is calculated based on the segmentation map generated by the student model and the corresponding labels. The calculation formula is as follows: (5) (6) In the formula, represents the cross entropy loss, represents the segmentation map generated by the student model, represents the labeled image after adding noise, Represents the label of the labeled image.

[0007] Furthermore, for tensors The convolution and activation operations use 3×3 convolution and LeakyReLU activation function respectively; the convolution kernel size of the depth-wise separable convolution layer is 3×3.

[0008] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention proposes a Mamba-CNN hybrid block, which not only expands the model's ability to perceive global information by integrating the convolutional neural network (CNN) with the Mamba structure, but also effectively captures details and local information, giving full play to the advantages of both in feature extraction and information integration. CNN is good at capturing details and local features in images, especially in processing edge information and small-scale targets, but its ability to perceive global information is relatively limited. The Mamba structure enhances the model's ability to capture long-distance dependencies and global background information through its global perception design, thereby making up for the shortcomings of traditional CNN in global vision. The combination of the two enables the model to process global context and detail features at the same time, significantly improving the ability to understand complex scenes. The Mamba-CNN hybrid block is used as the main module of the segmentation model to organically integrate multi-scale information with local features, effectively enhancing the image feature capture capability. This organic integration is particularly significant in the segmentation of the fuzzy pericardial fat area. The model can not only enhance the overall understanding of the fuzzy area based on the global context, but also refine the complex boundaries in combination with local features, thereby showing higher accuracy and robustness in low-contrast and complex background scenes. In the pericardial fat segmentation task, the model can pay more attention to key areas with rich information and complex areas that are difficult to segment, and weight image features of different scales to improve segmentation effect and accuracy. This adaptability enables the model to show stronger robustness and generalization ability in complex image environments.

[0009] 2. The present invention uses a collaborative consistent mean teacher model for training and introduces a difference fusion strategy. It not only improves the overall performance of the model through the collaborative training of the teacher model and the two student models, but also provides feedback to the student model to correct its deviation. This strategy enables each model to learn from the learning results of other models while retaining its own feature extraction advantages to obtain a more reliable feature representation. By fusing the feature representations of different models, the focus on the fuzzy area of ​​pericardial fat is strengthened. Difference fusion aims to make full use of the information-rich but difficult-to-segment areas in unlabeled data, especially for uncertain areas, so that the model can reach a consensus in these areas and significantly improve the model's ability to segment pericardial fat. Difference fusion guides the model to deeply optimize the inconsistent information parts by focusing on the feature differences of the model in uncertain areas. Through multiple iterative training, the differences between the teacher model and the student model are gradually eliminated, and the performance of the model in the difficult-to-segment areas in unlabeled data is strengthened, thereby achieving more accurate prediction consistency. Through this method, the present invention can make full use of the potential information of the data, improve the accuracy and robustness of the segmentation, and provide a more efficient and accurate solution for pericardial fat segmentation.

[0010] 3. The present invention adopts a semi-supervised learning method. With only a small amount of labeled data and a large amount of unlabeled data, it fully mines the potential information of unlabeled data, thereby effectively reducing the model's dependence on labeled data and greatly improving the performance of pericardial fat segmentation. This method can accurately identify and segment pericardial fat areas with limited annotation resources, providing a more efficient solution for medical image analysis. This method not only improves the model's recognition accuracy of pericardial fat, but also greatly shortens the time required for diagnosis, thereby effectively assisting doctors in quantitative assessment of pericardial fat. It is of great significance in clinical practice and is conducive to the application of medical automatic diagnosis technology from laboratory to clinical. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a structural diagram of the teacher model and the student model of the present invention; Figure 2 It is a structural diagram of the Mamba and CNN hybrid module of the present invention; Figure 3 It is a framework diagram of the collaborative mean teacher model of the present invention; Figure 4 It is a flow chart of the segmentation graph difference fusion of the present invention; Figure 5 Comparison of pericardial fat segmentation results using different methods. DETAILED DESCRIPTION

[0012] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present application is not limited thereto.

[0013] The present invention provides a difference-guided consistent semi-supervised pericardial fat segmentation method (hereinafter referred to as the method, see Figures 1 to 5 ), including the following steps: S1, obtaining a plurality of cardiac MR images containing pericardial fat to form a data set, and preprocessing the data set; A dataset of cardiac MR images of 150 subjects was obtained, including 50 subjects with hypertrophic cardiomyopathy, 50 subjects with acute myocardial infarction, and 50 normal subjects. For each subject, 20, 30, or 40 sets of image sequences were acquired at different stages of a cardiac cycle, and each sequence contained 8 to 18 images. The image resolution was adjusted to 256×256 pixels for preprocessing.

[0014] The preprocessed data set is divided into a training set and a test set; the training set is represented as , where there are label sets , Represents a labeled image, express Tags, Indicates the size of the label set; , represents an unlabeled image, Represents the size of the unlabeled set.

[0015] During model training, labeled images and unlabeled images account for 20% and 80% respectively. During the training phase, a combination of methods is used for data augmentation, including random rotation, contrast enhancement, and random cropping.

[0016] S2, based on the Mamba-Unet network, introduce the Mamba-CNN Mixture Block (MCM block for short) to build a teacher model and a student model with the same structure; Figure 1 As shown, the teacher model and the student model are obtained by replacing the VSS blocks of the Mamba-Unet network one by one using the MCM blocks.

[0017] like Figure 2 As shown, in the MCM block, the input tensor The channel dimension is evenly split into two tensors and , expressed as: (1) In the formula, Represents a split operation; Tensor After 3×3 convolution, LeakyReLU activation, 3×3 convolution and LeakyReLU activation, it is skipped and connected to itself to get the tensor ; Tensor Entering two information flow branches, one information flow branch includes a linear layer, a depth-separable convolutional layer, a 2D state space (SS2D) block, and a normalization layer connected in sequence, and the other information flow branch includes a linear layer and a SiLU activation layer connected in sequence. After multiplying the output tensors of the two information flow branches, the tensor is obtained. ; The specific process is expressed as:

[0018] In the formula, represents the LeakyReLU activation function, represents a 3×3 convolution operation, represents the normalization operation, represents a 2D state-space module, represents a 3×3 depthwise separable convolution operation, represents the SiLU activation function, Represents a linear operation; The tensor and After concatenation, it passes through a 1×1 convolution layer and is combined with the input tensor in the MCM block. Perform skip connection to obtain the output tensor of the MCM block , the process is expressed by formula (4): (4) In the formula, represents a 1×1 convolution operation, Represents a concatenation operation.

[0019] S3. Introduce a consistent collaborative mean teacher model based on difference fusion, train the consistent collaborative mean teacher model in a semi-supervised manner, and use the trained teacher model as a segmentation model for pericardial fat segmentation; like Figure 3 As shown in the figure, the collaborative mean teacher model includes a teacher model and two student models. The teacher model is collaboratively trained by the two student models. The training is divided into two parts: supervised and unsupervised. For supervised training, the labeled images are respectively added with noise and input into the two student models for segmentation. The supervision loss is calculated based on the segmentation map generated by the student model and the corresponding label. The calculation formula is as follows: (5) (6) In the formula, represents the supervision loss, represents the cross entropy loss, Represents the student model The generated segmentation map, represents the labeled image after adding noise, Represents the label of the labeled image.

[0020] For unsupervised training, such as Figure 4 As shown in the figure, the unlabeled image is first input into the teacher model for segmentation. After adding different noises, the unlabeled image is input into two student models for segmentation. The segmentation map generated by the teacher model is XORed with the segmentation maps generated by the two student models pixel by pixel to achieve the difference fusion of the segmentation maps, and the first difference map and the second difference map are obtained. The process is expressed as: (7) (8) In the formula, represents the difference map, represents the first difference map, represents the second difference map, Represents the student model The generated segmentation map, are the learnable parameters of the student model, represents the unlabeled image after adding noise, Represents the teacher model The generated segmentation map, represents an unlabeled image, are the learnable parameters of the teacher model, represents the XOR operation, Represents the argmax function.

[0021] The difference fusion of segmentation maps aims to adaptively strengthen the model's attention to key areas, making it more accurate and robust when segmenting pericardial fat. The XOR operation can highlight areas with significant differences in the segmentation results of the teacher model and the student model, which are usually located at the complex edges of pericardial fat or areas that are difficult to accurately segment; areas with large differences in the segmentation results of the teacher model and the student model are regarded as areas with high uncertainty. Through this uncertainty identification, the model can focus on these areas, which can effectively improve the model's prediction stability and segmentation accuracy.

[0022] Unsupervised training introduces consistency loss and cross pseudo-supervision loss The training loss is calculated jointly. The consistency loss is used to guide the model to focus on learning uncertain areas, learn effective information from them, and make the model produce more consistent predictions in these areas. The calculation formula is as follows: (9) In the formula, represents the consistency weight, which changes with the iteration rounds. The calculation formula is as follows: (10) In the formula, is the maximum value of the weight, Indicates the current iteration number. Indicates the total number of iterations; Cross-pseudo-supervision loss It is used to guide the two student models to collaborate and learn from each other and enhance the consistency between them. The calculation formula is as follows: (11) In the formula, Indicates dice loss; Unsupervised loss It is expressed as follows: (12) Total training loss of the collaborative mean teacher model The calculation formula is: (13) In the formula, is a weight factor used to balance supervised loss and unsupervised loss, and the Gaussian warm-up function is used to control changes, thereby optimizing the training process; (14) In the formula, is the maximum value of the weight; The teacher model is updated by the exponential moving average of the student model, and the process is expressed as: (15) In the formula, are the parameters of the teacher model, and are the parameters of the two student models, represents the decay coefficient of the exponential moving average, It is the coefficient that controls the fusion ratio of the two student model parameters.

[0023] The trained teacher model is used as a segmentation model, and the segmentation model is used to segment the pericardial fat in the cardiac MR image to obtain a pericardial fat segmentation map.

[0024] Example: This example conducts experiments on the MRPEAT dataset and uses the method of the present invention and the existing method to perform pericardial fat segmentation. The present invention uses the PyTorch framework to implement on the NVIDIA GeForce RTX 3090 Ti GPU, uses the AdamW optimizer for training, and has a fixed learning rate of 0.001 and a maximum consistency weight. and the maximum unsupervised weight All are set to 0.1, the decay rate of the exponential moving average The training is set to 0.99, the batch size is set to 4, and the training is performed for 100 epochs.

[0025] Four indicators are used to evaluate the segmentation performance: Dice coefficient, Jaccard index, 95% Hausdorff distance (95HD) and accuracy. The experimental results and segmentation results of different methods are shown in Tables 1 and Figure 5 .

[0026]

[0027] As can be seen from Table 1, the pericardial fat segmentation results obtained by the method of the present invention are significantly better than the existing semi-supervised methods in multiple indicators. Specifically, the method of the present invention has achieved significant improvements in both the Dice coefficient and the Jaccard index, which shows that the method of the present invention can more accurately capture the boundaries and morphology of pericardial fat and improve the overall accuracy of segmentation. In addition, the Precision is also higher than other methods, which further verifies that the method of the present invention has a strong ability to identify the correct area during the segmentation process. Most importantly, in terms of the 95HD indicator, the performance of the method of the present invention is particularly outstanding, which is only 5.02mm, which is significantly lower than that of methods such as MT (13.01mm) and CCT (12.57mm). This shows that the present invention can provide more accurate segmentation results when dealing with complex areas with blurred boundaries and non-fixed positions, especially for the edge areas of pericardial fat, reducing the distance between the predicted results and the true labels.

[0028] from Figure 5 It can be seen that although most methods can segment the main areas of pericardial fat well, they still face challenges when dealing with complex structures. Convolutional neural networks often lose details during the segmentation process, especially for the segmentation of pericardial fat edges and local areas, which can easily lead to overfitting; and networks using the Mamba architecture are prone to inaccuracies in the segmentation of some detail areas. In contrast, the method of the present invention effectively combines convolution and Mamba blocks and introduces difference fusion in semi-supervised training. By identifying areas where there are significant differences in the segmentation results of the teacher model and the student model, it pays special attention to high uncertainty areas, optimizes the parts with blurred boundaries and difficult to segment, and especially in the processing of complex edges and local details, it shows stronger accuracy and robustness, and achieves more accurate pericardial fat segmentation.

[0029] The sources of the above methods are as follows: [1] A. Tarvainen and H. Valpola, Mean teachers are better rolemodels: Weight-averaged consistency targets improve semi-supervised deep learning results, Advances in neural information processing systems 30, 2017. [2] Y. Ouali, C. Hudelot, and M. Tami, Semi-supervised semanticsegmentation with cross consistency training, in Proceedings of the IEEE / CVFconference on computer vision and pattern recognition, pages 12674–12684,2020. [3] V. Verma, K. Kawaguchi, A. Lamb, J. Kannala, A. Solin, Y. Bengio,and D. Lopez-Paz, Interpolation consistency training for semi-supervisedlearning, Neural Networks 145, 90–106 ,2022. [4] L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng, Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation,in Medical image computing and computer assisted intervention–MICCAI 2019:22nd international conference, Shenzhen, China, October 13–17, 2019,proceedings, part II 22, pages 605–613, Springer, 2019. [5] X. Chen, Y. Yuan, G. Zeng, and J. Wang, Semi-supervised semanticsegmentation with cross pseudo supervision, in Proceedings of the IEEE / CVFconference on computer vision and pattern recognition, pages 2613–2622, 2021 [6] X. Luo, W. Liao, J. Chen, T. Song, Y. Chen, S. Zhang, N. Chen, G. Wang, and S. Zhang, Efficient semi-supervised gross target volume ofnasopharyngeal carcinoma segmentation via uncertainty rectified pyramidconsistency 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24, pages 318–329, Springer, [7] Z. Xu, Y. Wang, D. Lu, X. Luo, J. Yan, Y. Zheng, and RK-y.Tong, Ambiguity selective consistency regularization for mean-teacher semi-supervised medical image segmentation, Medical Image Analysis 88, 102880,2023. Thanks for reading the book.

Claims

1. A difference-guided consistent semi-supervised pericardial fat segmentation method, characterized in that: The method comprises the following steps: S1, obtaining a plurality of cardiac MR images containing pericardial fat to form a data set, and preprocessing the data set; S2, use MCM blocks to replace the VSS blocks of the Mamba-Unet network one by one, and then obtain the teacher model and the student model; In the MCM block, the input tensor is first split equally into two tensors along the channel dimension and ; Tensor After convolution, activation, convolution and activation operations, a jump connection is made with itself to obtain the tensor ; Tensor Enter two information flow branches, one of which includes a linear layer, a depth-separable convolutional layer, a SS2D block, and a normalization layer connected in sequence, and the other includes a linear layer and an activation layer connected in sequence; multiply the output tensors of the two information flow branches to obtain the tensor ; The tensor and After concatenation, a 1×1 convolution is performed, and then a jump connection is performed with the input tensor in the MCM block to obtain the output tensor of the MCM block; S3, semi-supervised training of the collaborative mean teacher model is performed based on difference fusion, and the trained teacher model is used as a segmentation model for the segmentation of pericardial fat; The collaborative mean teacher model includes a teacher model and two student models. For unsupervised training, the unlabeled image is input into the teacher model for segmentation. After adding different noises, the unlabeled image is input into the two student models for segmentation. The segmentation map generated by the teacher model is XORed pixel by pixel with the segmentation map generated by the two student models to obtain the first difference map and the second difference map. The process is expressed as: (7) (8) In the formula, represents the difference map, represents the argmax function, Represents the student model The generated segmentation map, are the learnable parameters of the student model, represents the unlabeled image after adding noise, Represents the teacher model The generated segmentation map, are the learnable parameters of the teacher model, represents an unlabeled image, Represents XOR operation; Unsupervised losses include consistency loss and cross-pseudo-supervision loss ; The calculation formula of consistency loss is as follows: (9) In the formula, represents the consistency loss, represents the consistency weight; The calculation formula of cross pseudo-supervision loss is as follows: (11) In the formula, represents the cross pseudo-supervision loss, Indicates dice loss; Total training loss The calculation formula is as follows: (13) In the formula, is the monitoring loss, is the unsupervised loss, is the weight factor.

2. The difference-guided consistent semi-supervised pericardial fat segmentation method according to claim 1, characterized in that: For supervised training, different noises are added to the labeled images and then input into two student models for segmentation. The supervised loss is calculated based on the segmentation map generated by the student model and the corresponding labels. The calculation formula is as follows: (5) (6) In the formula, represents the cross entropy loss, represents the segmentation map generated by the student model, represents the labeled image after adding noise, Represents the label of the labeled image.

3. The difference-guided consistent semi-supervised pericardial fat segmentation method according to claim 1 or 2, characterized in that: For tensors The convolution and activation operations use 3×3 convolution and LeakyReLU activation function respectively; the convolution kernel size of the depth-wise separable convolution layer is 3×3.

Images