Difference-guided consistent semi-supervised pericardial fat segmentation method

Abstract

The present invention belongs to the field of medical image segmentation technology, and specifically is a method for consistent semi-supervised pericardial fat segmentation based on difference guidance. First, a data set is acquired and processed; then, the VSS blocks of the Mamba‑Unet network are replaced one by one using the MCM block to obtain a teacher model and a student model; in the MCM block, the input tensor is first divided into two tensors, y and z, in the channel dimension; the tensor undergoes convolution, activation, convolution and activation operations in sequence, and then jump-connects with itself to obtain a tensor; the tensor is processed by two information flow branches to obtain a tensor; the tensor z and z are concatenated and then convolved, and then jump-connected with the input tensor in the MCM block to obtain the output tensor of the MCM block; finally, the collaborative mean teacher model is semi-supervisedly trained based on difference fusion, and the trained teacher model is used as a segmentation model for pericardial fat segmentation. The model pays more attention to the fuzzy areas and boundary information of pericardial fat, thereby improving segmentation accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of cardiac medical image segmentation, and in particular is a difference-guided consistency semi-supervised pericardial fat segmentation method. Background Art

[0002] The pericardial adipose tissue surrounding the heart and coronary arteries is composed of paracardial adipose tissue and epicardial adipose tissue. Numerous studies have confirmed that pericardial adipose tissue is significantly associated with various cardiovascular diseases. It can promote the occurrence and progression of cardiovascular diseases by releasing proinflammatory factors, oxidative stress, and endothelial damage. It may also cause local inflammation, left atrial fibrosis, and fatty infiltration, thereby causing atrial fibrillation. Therefore, the segmentation of pericardial adipose tissue plays an important role in the prevention and diagnosis of cardiovascular diseases such as coronary artery disease. However, pericardial adipose tissue in cardiac MR images has diverse morphologies, sizes, locations, and discontinuous distribution, making its accurate segmentation challenging.

[0003] In recent years, deep learning-based semantic segmentation techniques have made significant progress in medical image analysis, significantly improving the accuracy and speed of medical image segmentation. However, these methods still face challenges in addressing pericardial fat segmentation in cardiac MR images. First, the unique nature of pericardial fat in cardiac MR images increases the difficulty of segmentation. Pericardial fat boundaries are often fuzzy, with low contrast compared to other cardiac tissues, making it easily confused with neighboring tissues. Furthermore, its morphology and location vary significantly between individuals, and its irregular distribution makes it difficult for the model to accurately extract edge features, leading to missed or false detections and suboptimal segmentation results. Second, consistency regularization, a semi-supervised approach, particularly models based on the mean teacher framework, emphasizes enhancing model stability through consistency in predictions between the teacher and student models. While this approach can partially alleviate the lack of labeled data, it often exhibits significant prediction fluctuations when dealing with information-dense and complex regions. Furthermore, the inherent characteristics of pericardial fat in cardiac MR images make the model susceptible to small noise artifacts during segmentation, resulting in unstable predictions. This fluctuation reflects the uncertainty of the model within specific regions, making it difficult to further improve segmentation accuracy. Third, clinical data of cardiac MR images are scarce, especially high-quality annotated datasets. Pixel-level accurate annotation of pericardial fat not only consumes a lot of time and manpower, but also requires the annotators to have professional medical knowledge to ensure the accuracy and consistency of the annotation. Since pericardial fat is distributed around the heart and has a complex anatomical location, and it is difficult to distinguish between small areas and boundaries during manual annotation, the annotation results vary between different annotators, which directly limits the effectiveness of model training and makes it difficult for the model to generalize on diverse data. Based on the above problems, the present invention proposes a consistent semi-supervised pericardial fat segmentation method based on difference guidance. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the technical problem to be solved by the present invention is to propose a consistent semi-supervised pericardial fat segmentation method based on difference guidance.

[0005] The present invention solves the technical problem by adopting the following technical solutions:

[0006] A method for consistent semi-supervised pericardial fat segmentation based on difference guidance, characterized in that the method comprises the following steps:

[0007] S1. Acquire a plurality of cardiac MR images containing pericardial fat to form a data set, and preprocess the data set;

[0008] S2, use MCM blocks to replace the VSS blocks of the Mamba-Unet network one by one, and then obtain the teacher model and student model;

[0009] In the MCM block, the input tensor is first split equally into two tensors along the channel dimension. and ; Tensor After convolution, activation, convolution and activation operations, a jump connection is made with itself to obtain the tensor ; Tensor Enter two information flow branches, one of which includes a linear layer, a depth-separable convolutional layer, an SS2D block, and a normalization layer connected in sequence, and the other includes a linear layer and an activation layer connected in sequence; multiply the output tensors of the two information flow branches to obtain a tensor ; The tensor and After concatenation, a 1×1 convolution is performed and then a jump connection is performed with the input tensor in the MCM block to obtain the output tensor of the MCM block;

[0010] S3. Perform semi-supervised training on the collaborative mean teacher model based on difference fusion, and use the trained teacher model as the segmentation model for pericardial fat segmentation;

[0011] The collaborative mean teacher model consists of a teacher model and two student models. For unsupervised training, the unlabeled image is input into the teacher model for segmentation. After adding different noises, the unlabeled image is input into the two student models for segmentation. The segmentation map generated by the teacher model is XORed pixel by pixel with the segmentation maps generated by the two student models to obtain the first difference map and the second difference map. The process is expressed as follows:

[0012] (7)

[0013] (8)

[0014] Where, represents the difference map, represents the argmax function, Represents the student model The generated segmentation map, are the learnable parameters of the student model, represents the unlabeled image after adding noise, Represents the teacher model The generated segmentation map, are the learnable parameters of the teacher model, represents an unlabeled image, Represents exclusive OR operation;

[0015] Unsupervised loss includes consistency loss and cross pseudo-supervision loss ; The calculation formula of consistency loss is as follows:

[0016] (9)

[0017] Where, represents the consistency loss, represents the consistency weight;

[0018] The calculation formula of cross pseudo supervision loss is as follows:

[0019] (11)

[0020] Where, represents the cross pseudo-supervision loss, Indicates dice loss;

[0021] Total training loss The calculation formula is as follows:

[0022] (13)

[0023] Where, is the monitoring loss, is the unsupervised loss, is the weighting factor.

[0024] Furthermore, for supervised training, different noises are added to the labeled images and then input into two student models for segmentation. The supervised loss is calculated based on the segmentation maps generated by the student models and the corresponding labels. The calculation formula is as follows:

[0025] (5)

[0026] (6)

[0027] Where, represents the cross entropy loss, represents the segmentation map generated by the student model, represents the labeled image after adding noise, Represents the label of the labeled image.

[0028] Furthermore, for tensors The convolution and activation operations use 3×3 convolution and LeakyReLU activation function respectively; the convolution kernel size of the depth-wise separable convolution layer is 3×3.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This paper proposes a Mamba-CNN hybrid block. By integrating a convolutional neural network (CNN) with the Mamba architecture, it not only expands the model's ability to perceive global information, but also effectively captures details and local information, leveraging the strengths of both in feature extraction and information integration. CNNs excel at capturing details and local features in images, particularly at processing edge information and small-scale objects. However, their ability to perceive global information is relatively limited. The Mamba architecture, through its global perception design, enhances the model's ability to capture long-range dependencies and global context, thereby compensating for the traditional CNN's shortcomings in global vision. The combination of the two enables the model to simultaneously process global context and detailed features, significantly improving its ability to understand complex scenes. Using the Mamba-CNN hybrid block as the primary module of the segmentation model, the method organically integrates multi-scale information with local features, effectively enhancing the ability to capture image features. This integration is particularly significant in the segmentation of blurred pericardial fat areas. The model not only enhances its overall understanding of blurred areas based on global context, but also refines complex boundaries by combining local features, resulting in higher accuracy and robustness in low-contrast and complex background scenes. In the pericardial fat segmentation task, the model focuses more on information-rich key areas and complex, difficult-to-segment areas, weighting image features at different scales to improve segmentation performance and accuracy. This adaptability enables the model to demonstrate greater robustness and generalization capabilities in complex image environments.

[0031] 2. The present invention uses a collaborative consistent mean teacher model for training and introduces a difference fusion strategy. It not only improves the overall performance of the model through the collaborative training of the teacher model and two student models, but also provides feedback to the student model to correct its deviation. This strategy enables each model to retain its own feature extraction advantages while learning from the learning results of other models to obtain a more reliable feature representation. By fusing the feature representations of different models, attention is focused on the fuzzy areas of pericardial fat. Difference fusion aims to make full use of the information-rich but difficult-to-segment areas in unlabeled data, especially for uncertain areas, so that the model reaches a consensus in these areas and significantly improves the model's ability to segment pericardial fat. Difference fusion guides the model to deeply optimize the parts with inconsistent information by focusing on the feature differences of the model in uncertain areas. Through multiple iterative training, the differences between the teacher model and the student model are gradually eliminated, and the model's performance in difficult-to-segment areas in unlabeled data is strengthened, thereby achieving more accurate prediction consistency. Through this method, the present invention can make full use of the potential information of the data, improve the accuracy and robustness of segmentation, and provide a more efficient and accurate solution for pericardial fat segmentation.

[0032] 3. This invention utilizes a semi-supervised learning approach, leveraging only a small amount of labeled data and a large amount of unlabeled data. This fully exploits the potential of the unlabeled data, effectively reducing the model's reliance on labeled data and significantly improving pericardial fat segmentation performance. This method accurately identifies and segments pericardial fat regions with limited labeled resources, providing a more efficient solution for medical image analysis. This method not only improves the model's accuracy in identifying pericardial fat but also significantly reduces the time required for diagnosis, effectively assisting doctors in quantitatively assessing pericardial fat. This approach is of great significance in clinical practice and facilitates the transition of automated medical diagnostic technology from the laboratory to the clinic. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural diagram of the teacher model and the student model of the present invention;

[0034] Figure 2 This is a structural diagram of the Mamba and CNN hybrid module of the present invention;

[0035] Figure 3 A framework diagram of the collaborative mean teacher model of the present invention;

[0036] Figure 4 This is a flow chart of the segmentation map difference fusion of the present invention;

[0037] Figure 5 Comparison of pericardial fat segmentation results using different methods. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but this does not limit the scope of protection of the present application.

[0039] The present invention provides a consistent semi-supervised pericardial fat segmentation method based on difference guidance (hereinafter referred to as the method, see Figures 1 to 5 ), including the following steps:

[0040] S1. Acquire a number of cardiac MR images containing pericardial fat to form a data set and preprocess the data set;

[0041] A dataset of cardiac MR images of 150 subjects was obtained, including 50 subjects with hypertrophic cardiomyopathy, 50 subjects with acute myocardial infarction, and 50 normal subjects. For each subject, 20, 30, or 40 image sequences were acquired at different stages of a cardiac cycle, with each sequence containing 8 to 18 images. The image resolution was adjusted to 256×256 pixels for preprocessing.

[0042] The preprocessed dataset is divided into training set and test set; the training set is represented as , where there is a label set , Represents a labeled image, express Tags, Indicates the size of the label set; , represents an unlabeled image, represents the size of the unlabeled set.

[0043] During model training, labeled images and unlabeled images account for 20% and 80% respectively. During the training phase, a combination of data augmentation methods is used, including random rotation, contrast enhancement, and random cropping.

[0044] S2, based on the Mamba-Unet network, introduce the Mamba-CNN Mixture Block (MCM block) to build the teacher model and student model with the same structure; Figure 1 As shown in Figure 2, the teacher model and the student model are obtained by replacing the VSS blocks of the Mamba-Unet network one by one using the MCM blocks.

[0045] like Figure 2 As shown, in the MCM block, the input tensor Evenly split into two tensors along the channel dimension and , expressed as:

[0046] (1)

[0047] Where, Represents a split operation;

[0048] Tensor After 3×3 convolution, LeakyReLU activation, 3×3 convolution and LeakyReLU activation, it is skipped and connected to itself to obtain the tensor ; Tensor Entering two information flow branches, one information flow branch includes a linear layer, a depth-wise separable convolution layer, a 2D state space (SS2D) block and a normalization layer connected in sequence, and the other information flow branch includes a linear layer and a SiLU activation layer connected in sequence. After multiplying the output tensors of the two information flow branches, the tensor is obtained. ; The specific process is expressed as:

[0049]

[0050] Where, represents the LeakyReLU activation function, represents a 3×3 convolution operation, represents the normalization operation, represents a 2D state-space module, represents a 3×3 depthwise separable convolution operation, represents the SiLU activation function, Represents a linear operation;

[0051] The tensor and After concatenation, it passes through a 1×1 convolution layer and is combined with the input tensor in the MCM block. Perform skip connection to obtain the output tensor of the MCM block , and the process is expressed by formula (4):

[0052] (4)

[0053] Where, represents a 1×1 convolution operation, Represents a splicing operation.

[0054] S3. Introduce a consistent collaborative mean teacher model based on difference fusion, train the consistent collaborative mean teacher model in a semi-supervised manner, and use the trained teacher model as a segmentation model for pericardial fat segmentation;

[0055] like Figure 3 As shown in the figure, the collaborative mean teacher model consists of a teacher model and two student models. The teacher model is collaboratively trained by the two student models. The training is divided into two parts: supervised and unsupervised. For supervised training, the labeled images are noised and input into the two student models for segmentation. The supervision loss is calculated based on the segmentation maps generated by the student models and the corresponding labels. The calculation formula is as follows:

[0056] (5)

[0057] (6)

[0058] Where, represents the supervision loss, represents the cross entropy loss, Represents the student model The generated segmentation map, represents the labeled image after adding noise, Represents the label of the labeled image.

[0059] For unsupervised training, such as Figure 4As shown in the figure, the unlabeled image is first input into the teacher model for segmentation. After adding different noises, the unlabeled image is input into two student models for segmentation. The segmentation map generated by the teacher model is XORed with the segmentation maps generated by the two student models pixel by pixel to achieve the difference fusion of the segmentation maps and obtain the first difference map and the second difference map. The process is expressed as:

[0060] (7)

[0061] (8)

[0062] Where, represents the difference map, represents the first difference map, represents the second difference map, Represents the student model The generated segmentation map, are the learnable parameters of the student model, represents the unlabeled image after adding noise, Represents the teacher model The generated segmentation map, represents an unlabeled image, are the learnable parameters of the teacher model, represents the exclusive OR operation, Represents the argmax function.

[0063] Difference fusion of segmentation maps aims to adaptively strengthen the model's focus on key areas, making it more accurate and robust when segmenting pericardial fat. The XOR operation highlights regions with significant differences between the teacher and student segmentation results. These regions are typically located along complex edges of pericardial fat or in areas that are difficult to accurately segment. Regions with significant differences between the teacher and student segmentation results are considered to have high uncertainty. This uncertainty identification allows the model to focus on these regions, effectively improving the model's predictive stability and segmentation accuracy.

[0064] Unsupervised training introduces consistency loss and cross pseudo-supervision loss The training loss is calculated jointly. The consistency loss is used to guide the model to focus on learning uncertain areas, learn effective information from them, and make the model produce more consistent predictions in these areas. The calculation formula is as follows:

[0065] (9)

[0066] Where, Represents the consistency weight, which changes with the iteration round, The calculation formula is as follows:

[0067] (10)

[0068] Where, is the maximum value of the weight, Indicates the current iteration number, Indicates the total number of iterations;

[0069] Cross-pseudo-supervision loss It is used to guide the two student models to collaborate and learn from each other and enhance the consistency between them. The calculation formula is as follows:

[0070] (11)

[0071] Where, Indicates dice loss;

[0072] Unsupervised loss It is expressed as follows:

[0073] (12)

[0074] Total training loss of the collaborative mean teacher model The calculation formula is:

[0075] (13)

[0076] Where, Is the weight factor used to balance the supervised loss and unsupervised loss, and the Gaussian warm-up function is used to control changes, thereby optimizing the training process;

[0077] (14)

[0078] Where, is the maximum value of the weight;

[0079] The teacher model is updated by the exponential moving average of the student model, and the process is expressed as:

[0080] (15)

[0081] Where, are the parameters of the teacher model, and are the parameters of the two student models, represents the decay coefficient of the exponential moving average, is the coefficient that controls the fusion ratio of the two student model parameters.

[0082] The trained teacher model is used as a segmentation model, and the segmentation model is used to segment the pericardial fat in the cardiac MR image to obtain a pericardial fat segmentation map.

[0083] Example:

[0084] This example experiments on the MRPEAT dataset, using the proposed method and existing methods to segment pericardial fat. The proposed method is implemented on an NVIDIA GeForce RTX 3090 Ti GPU using the PyTorch framework, trained using the AdamW optimizer, with a fixed learning rate of 0.001 and a maximum consistency weight. and the maximum unsupervised weight All are set to 0.1, the decay rate of the exponential moving average The lag is set to 0.99, the batch size is set to 4, and the training is done for 100 epochs.

[0085] Four metrics are used to evaluate the segmentation performance: Dice coefficient, Jaccard index, 95% Hausdorff distance (95HD) and accuracy. The experimental results and segmentation results of different methods are shown in Tables 1 and Figure 5 .

[0086]

[0087] As can be seen from Table 1, the pericardial fat segmentation results obtained by the method of the present invention significantly outperform existing semi-supervised methods in multiple metrics. Specifically, the method of the present invention achieves significant improvements in both the Dice coefficient and the Jaccard index, demonstrating that the method of the present invention can more accurately capture the boundaries and morphology of pericardial fat, improving the overall accuracy of segmentation. Furthermore, the precision is also higher than that of other methods, further verifying that the method of the present invention has a strong ability to identify the correct regions during the segmentation process. Most importantly, the performance of the method of the present invention is particularly outstanding in the 95HD metric, which is only 5.02mm, significantly lower than that of methods such as MT (13.01mm) and CCT (12.57mm). This demonstrates that the method of the present invention can provide more accurate segmentation results when dealing with complex areas with blurred boundaries and unstable positions, especially for the edge areas of pericardial fat, reducing the distance between the predicted results and the true labels.

[0088] from Figure 5It can be seen that although most methods can better segment the main areas of pericardial fat, they still face challenges when dealing with complex structures. Convolutional neural networks often lose details during the segmentation process, especially for the segmentation of pericardial fat edges and local areas, which can easily lead to overfitting; and networks using the Mamba architecture are prone to inaccuracies in the segmentation of some detail areas. In contrast, the method of the present invention effectively combines convolution and Mamba blocks and introduces difference fusion in semi-supervised training. By identifying areas with significant differences in the segmentation results of the teacher model and the student model, it pays special attention to high uncertainty areas, optimizes the parts with blurred boundaries and difficult to segment, and especially shows stronger accuracy and robustness in the processing of complex edges and local details, thereby achieving more accurate pericardial fat segmentation.

[0089] The sources of the above methods are as follows:

[0090] [1] A. Tarvainen and H. Valpola, Mean teachers are better rolemodels: Weight-averaged consistency targets improve semi-supervised deep learning results, Advances in neural information processing systems 30, 2017.

[0091] [2] Y. Ouali, C. Hudelot, and M. Tami, Semi-supervised semanticsegmentation with cross consistency training, in Proceedings of the IEEE / CVFconference on computer vision and pattern recognition, pages 12674–12684, 2020.

[0092] [3] V. Verma, K. Kawaguchi, A. Lamb, J. Kannala, A. Solin, Y. Bengio, and D. Lopez-Paz, Interpolation consistency training for semi-supervised learning, Neural Networks 145, 90–106, 2022.

[0093] [4] L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng, Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation,in Medical image computing and computer assisted intervention–MICCAI 2019:22nd international conference, Shenzhen, China, October 13–17, 2019,proceedings, part II 22, pages 605–613, Springer, 2019.

[0094] [5] X. Chen, Y. Yuan, G. Zeng, and J. Wang, Semi-supervised semanticsegmentation with cross pseudo supervision, in Proceedings of the IEEE / CVFconference on computer vision and pattern recognition, pages 2613–2622, 2021

[0095] [6] X. Luo, W. Liao, J. Chen, T. Song, Y. Chen, S. Zhang, N. Chen, G. Wang, and S. Zhang, Efficient semi-supervised gross target volume ofnasopharyngeal carcinoma segmentation via uncertainty rectified pyramidconsistency 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24, pages 318–329, Springer,

[0096] [7] Z. Xu, Y. Wang, D. Lu, X. Luo, J. Yan, Y. Zheng, and RK-y.Tong, Ambiguity selective consistency regularization for mean-teacher semi-supervised medical image segmentation, Medical Image Analysis 88, 102880,2023.

[0097] Thanks for reading the book.

Claims

1. A consistent semi-supervised pericardial fat segmentation method based on difference guidance, characterized in that: The method comprises the following steps: S1. Acquire a plurality of cardiac MR images containing pericardial fat to form a data set, and preprocess the data set; S2, use MCM blocks to replace the VSS blocks of the Mamba-Unet network one by one, and then obtain the teacher model and student model; In the MCM block, the input tensor is first split equally into two tensors along the channel dimension. and ; Tensor After convolution, activation, convolution and activation operations, a jump connection is made with itself to obtain the tensor ; Tensor Enter two information flow branches, one of which includes a linear layer, a depth-separable convolutional layer, an SS2D block, and a normalization layer connected in sequence, and the other includes a linear layer and an activation layer connected in sequence; multiply the output tensors of the two information flow branches to obtain a tensor ; The tensor and After concatenation, a 1×1 convolution is performed and then a jump connection is performed with the input tensor in the MCM block to obtain the output tensor of the MCM block; S3. Perform semi-supervised training on the collaborative mean teacher model based on difference fusion, and use the trained teacher model as the segmentation model for pericardial fat segmentation; The collaborative mean teacher model consists of a teacher model and two student models. For unsupervised training, the unlabeled image is input into the teacher model for segmentation. After adding different noises, the unlabeled image is input into the two student models for segmentation. The segmentation map generated by the teacher model is XORed pixel by pixel with the segmentation maps generated by the two student models to obtain the first difference map and the second difference map. The process is expressed as follows: (7) (8) Where, represents the difference map, represents the argmax function, Represents the student model The generated segmentation map, are the learnable parameters of the student model, represents the unlabeled image after adding noise, Represents the teacher model The generated segmentation map, are the learnable parameters of the teacher model, represents an unlabeled image, Represents exclusive OR operation; Unsupervised loss includes consistency loss and cross pseudo-supervision loss ; The calculation formula of consistency loss is as follows: (9) Where, represents the consistency loss, represents the consistency weight; The calculation formula of cross pseudo supervision loss is as follows: (11) Where, represents the cross pseudo-supervision loss, Indicates dice loss; Total training loss The calculation formula is as follows: (13) Where, is the monitoring loss, is the unsupervised loss, is the weighting factor.

2. The difference-guided consistent semi-supervised pericardial fat segmentation method according to claim 1, characterized in that: For supervised training, different noises are added to the labeled images and then input into two student models for segmentation. The supervised loss is calculated based on the segmentation maps generated by the student models and the corresponding labels. The calculation formula is as follows: (5) (6) Where, represents the cross entropy loss, represents the segmentation map generated by the student model, represents the labeled image after adding noise, Represents the label of the labeled image.

3. The method for consistent semi-supervised pericardial fat segmentation based on difference guidance according to claim 1 or 2, characterized in that: For tensors The convolution and activation operations use 3×3 convolution and LeakyReLU activation function respectively; the convolution kernel size of the depth-wise separable convolution layer is 3×3.

Images