Semi-supervised pericardial fat segmentation method based on multi-model cross teaching

Abstract

The invention belongs to the technical field of medical image processing, and particularly relates to a semi-supervised pericardial fat segmentation method based on multi-model cross teaching. The method comprises the following steps: firstly, carrying out ROI extraction on pericardial adipose tissue by using a lightweight SAM model to obtain a rough ROI; the lightweight SAM model comprises an image encoder and a visual Transform decoder, and an adapter layer is embedded in each Transform block of the image encoder; performing morphological processing on the rough ROI to obtain a fine ROI; mapping the fine ROI to an input image and cutting the fine ROI to obtain a fine ROI image; then, a segmentation model composed of an encoder, a bottleneck processing module and a decoder is built based on TransUNet, and the bottleneck processing module comprises an ASPP module and a plurality of WTS blocks; and finally, performing semi-supervised training on the segmentation model, the UNeXt model and the Mamb-UNet model based on multi-model cross teaching, and applying the trained segmentation model to pericardial fat segmentation. According to the method, the edge and detail information of pericardial fat can be better captured, and the segmentation precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically is a semi-supervised pericardial fat segmentation method based on multi-model cross-teaching. Background Art

[0002] Pericardial adipose tissue is a special tissue wrapped around the epicardial surface of the heart and coronary arteries, which is composed of pericardial adipose tissue and epicardial adipose tissue. Pericardial adipose tissue is located in the outer layer of the pericardium. A large number of studies have shown that pericardial adipose tissue is closely related to a variety of heart diseases. Its unique structure not only participates in energy supply and mechanical protection, but also regulates local and systemic metabolism by secreting cytokines, thereby directly affecting heart function and health. Specifically, the increase in the volume of pericardial adipose tissue will not only increase the secretion of proinflammatory adipose factors, but also accelerate the damage of the vascular endothelium through local inflammatory reactions, promote the formation of atherosclerotic plaques and fibrosis of the vascular wall, and thus increase the risk of coronary artery disease. The epicardial adipose tissue is located between the epicardium and the inner layer of the pericardium, mainly concentrated around the ventricles and coronary arteries. The increase in the thickness of the epicardial adipose tissue is positively correlated with the occurrence of hypertrophic cardiomyopathy and acute myocardial infarction. Therefore, accurate segmentation of pericardial adipose tissue is crucial for assessing the risk of heart disease, supporting early diagnosis and personalized treatment.

[0003] With the rapid development of deep learning technology, it has made significant progress in medical image segmentation tasks. However, due to the generally small size of existing clinical databases, it is difficult to effectively train models through full supervision. Semi-supervised training can reduce the reliance on a large amount of manually labeled data while ensuring high accuracy, providing a new solution for the automatic segmentation of pericardial fat tissue. In semi-supervised learning methods, consistency regularization methods, especially the model framework based on the teacher-student architecture, have been widely used. This framework updates the teacher model parameters through the exponential moving average (EMA) method to establish a close connection between the teacher model and the student model. This design requires that the teacher model can reflect the changes of the student model in real time to maintain the stability and consistency of training. Although this method improves segmentation accuracy and robustness, the flexibility of the model structure and learning strategy is limited to a certain extent because the teacher and student models need to be highly consistent, especially when dealing with pericardial fat features with high diversity, blurred edges and irregular distribution. There are still limitations.

[0004] Pericardial fat tissue segmentation is very challenging in medical image processing. The main reasons are: first, pericardial fat is similar to adjacent structures such as myocardium and coronary vessels in density and texture, making it difficult to accurately distinguish. Second, noise, low resolution, and motion blur caused by heartbeats can lead to image quality degradation, further increasing the difficulty of segmentation. Third, the distribution and thickness of pericardial fat vary among different individuals, increasing the complexity of segmentation. Most current studies are still based on U-Net, Transformer and their variants. Although these models have achieved good results in image segmentation tasks, they still have limitations in feature extraction and information propagation. The U-Net model performs well in capturing local features, but has limitations in dealing with complex problems such as blurred boundaries between pericardial fat and surrounding tissues and long-distance dependencies. Although the Transformer model can effectively capture global features and long-distance dependencies, it has high requirements for computing resources, especially when processing high-resolution medical images, which can easily cause pressure on memory and computing resources. Summary of the invention

[0005] In view of the deficiencies in the prior art, the technical problem that the present invention intends to solve is to propose a semi-supervised pericardial fat segmentation method based on multi-model cross-teaching.

[0006] The present invention solves the technical problem by adopting the following technical solution:

[0007] A semi-supervised pericardial fat segmentation method based on multi-model cross-teaching, characterized by comprising the following steps:

[0008] S1. Use the lightweight SAM model to extract the ROI of pericardial fat tissue in cardiac MR images to obtain a rough ROI;

[0009] The lightweight SAM model includes an image encoder and a visual Transformer decoder. An adapter layer is embedded in each Transformer block of the image encoder, and the adapter layer is located after the multi-head attention layer of the Transformer block.

[0010] S2, perform morphological processing on the rough ROI to obtain a fine ROI; map the fine ROI onto the input image and crop it to obtain a fine ROI image;

[0011] S3. Build a segmentation model based on TransUNet. The segmentation model includes an encoder, a decoder, and a bottleneck processing module connecting the encoder and the decoder. The bottleneck processing module includes an ASPP module and multiple WTS blocks.

[0012] In the WTS block, the input features are normalized to obtain normalized input features; the normalized input features are input into two branches, and in one branch, after passing through the linear layer, the depth-separable convolution layer, the SS2D module, and the normalization layer in sequence, they are multiplied pixel by pixel with the features of the normalized input features after passing through the linear layer, and the multiplied features are passed through the linear layer to obtain the output features of the branch; in the other branch, the normalized input features are passed through the frequency domain feature extraction module, the normalization layer, and the multilayer perceptron in sequence to obtain the output features of the branch; the output features of the two branches and the input features of the WTS block are added pixel by pixel to obtain the output features of the WTS block;

[0013] S4. Based on multi-model cross-teaching, semi-supervised training is performed on the segmentation model, UNeXt and Mamba-UNet models, and the trained segmentation model is used for pericardial fat segmentation.

[0014] Furthermore, in step S4, the refined ROI image x i The segmentation maps are input into the Mamba-UNet model and the segmentation model respectively, and the pseudo labels are generated using the segmentation maps. The pseudo labels are expressed as:

[0015]

[0016] In the formula, This is the segmentation map generated by the Mamba-UNet model. The pseudo labels obtained are is the segmentation map generated by the segmentation model The pseudo-label obtained, Softmax(·) is the Softmax function, and T(t) is the temperature parameter during the tth round of training;

[0017] Consistency loss L for unlabeled data con Defined as:

[0018]

[0019] Where, L softdice (·) represents a variant of Dice loss, N is the number of images in the current batch, Represents segmentation map The segmentation result at the position (h,w) in the middle, Represents pseudo labels The result at position (h,w);

[0020] The supervision loss L of labeled data sup Defined as:

[0021] L sup =w CE L CE(G,P)+w Dice L Dice (G,P)+w BD L BD (G,P) (18)

[0022]

[0023] L BD (G,P)=L mse (sobel(G),sobel(P)) (21)

[0024] Where, L CE (G,P) represents the cross entropy loss, L Dice (G,P) represents Dice loss, L BD (G,P) represents the boundary loss, w CE 、w Dice and w BD are weight factors, G is the label, P is the segmentation result, G c and P c Represent the true label and predicted probability of category c, C is the number of categories, L mse (·) is the mean square error loss, and sobel(·) is the Sobel edge operator.

[0025] The loss L between the Mamba-UNet model and the segmentation model part1 It is expressed as:

[0026] L part1 =L sup +λL con (twenty two)

[0027] Similarly, we get the loss L between the UNeXt model and the segmentation model part2 , then the total loss is expressed as:

[0028] L total =w1L part1 +w2L part2 (twenty three)

[0029] In the formula, λ, w1, and w2 are all weight factors.

[0030] Furthermore, in the adapter layer, the input features are subjected to the average pooling operation and then enter the two processing branches. In one processing branch, the features obtained by the average pooling are pre-activated, multiplied with the input features of the adapter layer, and then passed through the Transformer block to obtain the output features of the processing branch; in the other processing branch, the features obtained by the average pooling are sequentially subjected to deep convolution, ReLU activation, deconvolution, and Sigmod activation operations to obtain the output features of the processing branch; after the output features of the two processing branches are added together, they are sequentially subjected to convolution, ReLU activation, deconvolution, and ReLU activation operations, and then added to the input features of the adapter layer to obtain the output features of the adapter layer.

[0031] Furthermore, for the morphological processing of the rough ROI, the rough ROI is first normalized and then binarized; then, the binary ROI is converted into an array, and the connected domain is marked to obtain the maximum connected domain, and then the ROI containing only the maximum connected domain is obtained; the ROI containing only the maximum connected domain is subjected to a dilated convolution operation to obtain a fine ROI.

[0032] Furthermore, in the frequency domain feature extraction module, the normalized input features are decomposed by wavelet to obtain low-frequency sub-bands, horizontal high-frequency sub-bands, vertical high-frequency sub-bands and diagonal high-frequency sub-bands; each sub-band is multiplied pixel by pixel with the corresponding learnable weight matrix to obtain weighted low-frequency sub-bands, weighted horizontal high-frequency sub-bands, weighted vertical high-frequency sub-bands and weighted diagonal high-frequency sub-bands; each weighted sub-band is subjected to inverse wavelet transform to obtain reconstructed low-frequency sub-bands, reconstructed horizontal high-frequency sub-bands, reconstructed vertical high-frequency sub-bands and reconstructed diagonal high-frequency sub-bands; all reconstructed sub-bands are fused to obtain the output features of the frequency domain feature extraction module.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. In order to achieve accurate segmentation of pericardial fat, the SAM model is improved to obtain a lightweight SAM model, and a small amount of annotated ROI data is used to fine-tune the lightweight SAM model. Combined with the large pre-training parameters of the SAM model, the lightweight SAM model can accurately extract the ROI containing pericardial fat tissue from cardiac MR images. The introduction of the adapter layer in the image encoder is helpful to help the image encoder locate the ROI to be extracted, and enhance its ROI extraction ability while ensuring the feature extraction ability of the image encoder itself. Compared with the traditional center point or square box selection method, the lightweight SAM model can more intelligently exclude irrelevant areas and focus on key areas related to pericardial fat tissue, reducing the interference of irrelevant areas on segmentation accuracy, which not only improves the performance of the segmentation model, but also reduces the waste of subsequent computing resources.

[0035] 2. In view of the problem that traditional methods cannot accurately handle complex boundaries, the present invention further introduces morphological processing to refine the rough ROI after completing the extraction of the rough ROI. The morphological processing can effectively remove noise and fill the empty area inside the ROI, making the ROI boundary clearer, so that the segmentation model can be trained and inferred on cleaner and more accurate fine ROI images, which is conducive to improving the segmentation accuracy.

[0036] 3. Innovatively adopt a multi-model cross-teaching strategy to train the segmentation model, and enhance the segmentation performance through a semi-supervised learning framework that combines pseudo-labels and consistency regularization. Specifically, the pseudo-label method generates labels for unlabeled data, expanding the scale of training data; while consistency regularization improves the generalization ability and robustness of the model through data enhancement. The cross-teaching strategy can effectively alleviate the uncertainty introduced by pseudo-labels, and also reduce the sensitivity of consistency regularization to noise, thereby obtaining a more stable and efficient segmentation effect.

[0037] 4. A segmentation model is built based on TransUNet, and the spatial-channel compression and activation module (ASPP) of the attention mechanism is introduced to enhance the global receptive field and improve the model's perception of key areas of the image. At the same time, the frequency domain feature extraction based on wavelet transform and the Mamba feature extraction module (WTS block) are used to replace the Transformer block, which effectively captures global features while focusing on local features, and significantly reduces the number of parameters, thereby optimizing computing resources. Compared with traditional segmentation models such as UNet and Transformer, UNeXt and Mamba-UNet are introduced as auxiliary models, and the backbone model (i.e., segmentation model) is trained through multi-model cross-teaching, so that the backbone model can better capture the edge and detail information of pericardial fat, thereby improving segmentation accuracy. At the same time, multi-model cross-teaching has better training efficiency and stability, reducing the difficulty of training under limited labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall flow chart of the present invention;

[0039] Figure 2 It is a structural diagram of the lightweight SAM model of the present invention;

[0040] Figure 3 is a structural diagram of the segmentation model of the present invention;

[0041] Figure 4 This is a diagram of the semi-supervised training framework for multi-model cross-teaching of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present application is not limited thereto.

[0043] The present invention provides a semi-supervised pericardial fat segmentation method based on multi-model cross-teaching (hereinafter referred to as method, see Figures 1 to 4 ), including the following steps:

[0044] S1. Use the lightweight SAM model to extract the ROI of pericardial fat tissue in cardiac MR images to obtain a rough ROI;

[0045] The lightweight SAM model includes an image encoder and a visual Transformer decoder (ViT). The image embedding features obtained by the image encoder are input into the visual Transformer decoder for decoding to obtain a rough ROI. The visual Transformer decoder replaces the prompt encoder and mask decoder of the original SAM to achieve lightweight. An adapter layer is introduced into the image encoder to enable the image encoder to learn more parameters, which helps the image encoder locate the ROI to be extracted, and enhances its ROI extraction capability while ensuring the feature extraction capability of the image encoder itself. Figure 3 As shown in the figure, each Transformer block of the image encoder is embedded with an adapter layer, and the adapter layer is located after the multi-head attention layer of the Transformer block. In the adapter layer, the input features are subjected to the average pooling operation and then enter the two processing branches. In one processing branch, the features obtained by average pooling are pre-activated and multiplied with the input features of the adapter layer. The multiplied features are passed through the Transformer block to obtain the output features of the processing branch; in the other processing branch, the features obtained by average pooling are sequentially subjected to deep convolution, ReLU activation, deconvolution and Sigmod activation operations to obtain the output features of the processing branch; after the output features of the two processing branches are added, they are sequentially subjected to convolution, ReLU activation, deconvolution and ReLU activation operations, and then added to the input features of the adapter layer to obtain the output features of the adapter layer.

[0046] S2, perform morphological processing on the rough ROI to obtain a fine ROI; map the fine ROI onto the input image and crop it to obtain a fine ROI image;

[0047] The rough ROI has a relatively rough outline and may contain noise and holes. In order to reduce the impact of these problems on the segmentation effect, the ROI is refined through morphological processing.

[0048] First, the rough ROI is normalized and the Sigmoid activation function is used to map the pixel values ​​to the range of 0 to 1. Then, the pixel values ​​greater than 0.5 are set to 1, and the pixel values ​​equal to or less than 0.5 are set to 0. The specific process is as follows:

[0049]

[0050] In the formula, R represents the rough ROI, and R' represents the binarized ROI;

[0051] The binary ROI is converted into an array and its connected domain is marked using the connected domain marking algorithm. This process is defined as:

[0052] (L,N)=cdla(R′,connectivity=2) (2)

[0053] Where L represents the ROI after connected domain labeling, N represents the number of connected domains, cala (Connected Domain Labeling Algorithm) represents the connected domain labeling algorithm, and connectivity represents adjacency;

[0054] If there is more than one connected domain, the number of pixels in each connected domain is calculated, the largest connected domain is identified, and then the ROI containing only the largest connected domain is obtained; a 3×3 dilated convolution operation is performed on the ROI containing only the largest connected domain to eliminate holes in the ROI and obtain a fine ROI; the binary image corresponding to the fine ROI is multiplied pixel by pixel with the input image, and the fine ROI is mapped to the input image to obtain the input image after ROI mapping; the input image after ROI mapping is cropped to obtain a fine ROI image, which is the output of morphological processing.

[0055] S3. Build a segmentation model (WTS-UNet) based on TransUNet; the segmentation model includes an encoder, a decoder, and a bottleneck processing module connecting the encoder and the decoder. The bottleneck processing module includes an ASPP module (atrous spatial convolutional pooling pyramid) and multiple WTS blocks (this embodiment includes 12). The ASPP module combines spatial attention with channel attention mechanisms, significantly expands the receptive field, and enhances the focus on key areas, especially when dealing with complex or small target segmentation tasks; optimizes feature representation through compression and activation operations, effectively reduces redundant information, and improves the responsiveness to useful features, thereby capturing richer feature information at a low computational cost, thereby improving segmentation accuracy.

[0056] In the WTS block, the input features are normalized to obtain normalized input features; the normalized input features are input into two branches, and in one branch, after passing through the linear layer, the depth-separable convolution layer, the SS2D module and the normalization layer in sequence, they are multiplied pixel by pixel with the features of the normalized input features after passing through the linear layer, and the multiplied features are passed through the linear layer to obtain the output features of the branch; in the other branch, the normalized input features are passed through the frequency domain feature extraction module, the normalization layer and the multilayer perceptron in sequence to obtain the output features of the branch; the output features of the two branches and the input features of the WTS block are added pixel by pixel to obtain the output features of the WTS block. The frequency domain feature extraction module extracts frequency domain features based on discrete wavelet transform (DWT). First, the normalized input features are decomposed into four sub-bands through the wavelet decomposition layer, namely the low frequency sub-band (LL), the horizontal high frequency sub-band (LH), the vertical high frequency sub-band (HL) and the diagonal high frequency sub-band (HH), which are expressed as:

[0057] LL[m,n]=∑ i ∑ j I[i,j]·g[i-2m]·g[j-2n] (3)

[0058] LH[m,n]=∑ i ∑ j I[i,j]·g[i-2m]·h[j-2n] (4)

[0059] HL[m,n]=∑ i ∑ j I[i,j]·h[i-2m]·g[j-2n] (5)

[0060] HH[m,n]=∑ i ∑ j I[i,j]·h[i-2m]·h[j-2n] (6)

[0061] Where I[i,j] represents the normalized input feature, LL[m,n] represents the pixel value of the low-frequency subband, which is used to reflect the global features or contour information of the image; LH[m,n] represents the pixel value of the horizontal high-frequency subband, which is used to reflect the detail information in the horizontal direction, that is, the horizontal edge feature; HL[m,n] represents the pixel value of the vertical high-frequency subband, which is used to reflect the detail information in the vertical direction, that is, the vertical edge feature; HH[m,n] represents the pixel value of the diagonal high-frequency subband, which is used to reflect the detail information in the diagonal direction, that is, fine texture and noise; g[·] is a low-pass filter, which uses a scaling function to extract the smooth part of the image; h[·] is a high-pass filter, which uses a wavelet function to extract the detail part of the image; 2m and 2n represent indexes;

[0062] Each subband is multiplied pixel by pixel with the corresponding learnable weight matrix to obtain the weighted low-frequency subband LL'[m,n], the weighted horizontal high-frequency subband LH'[m,n], the weighted vertical high-frequency subband HL'[m,n] and the weighted diagonal high-frequency subband HH'[m,n]; taking the low-frequency subband as an example, the weighted low-frequency subband LL'[m,n] is expressed as:

[0063] LL′[m,n]=W LL LL[m,n] (7)

[0064] Where W LL represents the learnable weight matrix of the low-frequency subband;

[0065] By combining the learning weights with the DWT formula, the features are decomposed into multi-level wavelets to extract detail information at different resolution levels, which helps to capture local structural features.

[0066] Each weighted subband is input into the wavelet reconstruction layer, and after inverse wavelet transform, the reconstructed low-frequency subband I' is obtained. LL [i,j], reconstructed horizontal high frequency subband I' LH [i,j], reconstruct vertical high frequency subband I' HL [i,j] and reconstruct the diagonal high frequency subband I' HH [i,j]; then these sub-bands are fused to obtain frequency domain features and realize feature reconstruction. The process is as follows:

[0067] I′ LL [i,j]=∑ m ∑ n LL′[m,n]·g[i-2m]·g[j-2n] (8)

[0068] I′ LH [i,j]=Σ m Σ n LH′[m,n]·g[i-2m]·h[j-2n] (9)

[0069] I′ HL [i,j]=Σ m Σ n HL′[m,n]·h[i-2m]·g[j-2n] (10)

[0070] I′ HH [i,j]=∑ m ∑ n HH′[m,n]·h[i-2m]·h[j-2n] (11)

[0071] I′[i,j]=I′ LL [i,j]+I′LH [i,j]+I′ HL [i,j]+I′ HH [i,j] (12)

[0072] Where I'[i,j] represents the frequency domain feature.

[0073] S4. Semi-supervised training of WTS-UNet, UNeXt and Mamba-UNet models was performed based on multi-model cross-teaching, and the trained segmentation models were used for pericardial fat segmentation.

[0074] The UNeXt model effectively captures the details and microstructures in the image through the local focusing mechanism, and the Mamba-UNet model achieves a deep understanding of the global context information in the image. Figure 4 As shown in the figure, WTS-UNet is used as the backbone model, UNeXt and Mamba-UNet are used as auxiliary models for cross-teaching to improve the performance of the backbone model. For unlabeled data, the prediction results of the UNeXt and Mamba-UNet models will guide the learning process of the backbone model, and the prediction results of the backbone model will optimize the parameters of the auxiliary model. This cross-teaching mechanism enables the backbone model to combine the advantages of the two auxiliary models in capturing details and understanding global contextual information, which is conducive to improving the segmentation effect.

[0075] For a given refined ROI image x i , the Mamba-UNet model and the WTS-UNet model generate segmentation maps respectively, and the process is expressed as:

[0076]

[0077] In the formula, and Represent the Mamba-UNet model respectively and WTS-UNet model Generated segmentation map;

[0078] Since the Mamba-UNet model and the WTS-UNet model belong to different learning paradigms, the segmentation maps they generate have essentially different characteristics; therefore, pseudo labels are generated based on the segmentation results. The generation process is as follows:

[0079]

[0080] In the formula, and The segmentation maps are and The pseudo labels generated, Softmax(·) is the Softmax function, T(t) is the temperature parameter during the tth round of training, T base is the initial temperature parameter, T final is the lowest temperature parameter at the end of training, decay_rate represents the control decay speed, t total is the total number of training rounds.

[0081] The smoothness of the segmentation result distribution is controlled by dynamically adjusting the temperature parameter T(t). As the training progresses, the temperature parameter T(t) gradually decays exponentially, which not only promotes the model's exploration of a variety of possible label distributions in the early stages of training, but also promotes the model's exploration of a variety of possible label distributions in the later stages of training, enhancing the model's deterministic learning of specific patterns and structures.

[0082] Segmentation maps generated by the Mamba-UNet model and the WTS-UNet model and With pseudo labels and No gradient back propagation is performed between them to maintain the perturbation during training. For unlabeled data, the consistency loss is used to evaluate the consistency of the segmentation results of the Mamba-UNet model and the WTS-UNet model without additional constraints, effectively avoiding the error accumulation that may be caused by the prediction error of a single model, thereby improving the overall robustness and accuracy; the consistency loss of unlabeled data is defined as:

[0083]

[0084] Where, L con represents the consistency loss of unlabeled data, L softdice (·) represents a variant of Dice loss, N is the number of images in the current batch, Represents segmentation map The segmentation result at the position (h,w) in the middle, Represents pseudo labels The result at position (h,w); The calculation process of is the same as that of formula (17).

[0085] For labeled data, the model directly performs supervised learning with the true label, and the supervision loss of the labeled data is defined as:

[0086] L sup =w CE L CE (G,P)+w Dice L Dice (G,P)+w BD L BD (G,P) (18)

[0087]

[0088] L BD (G,P)=L mse (sobel(G),sobel(P)) (21)

[0089] Where, L sup represents the supervision loss of labeled data; L CE (G,P) represents the cross entropy loss, which accelerates the initial learning of the model by imposing a high penalty on incorrect predictions; L Dice (G, P) represents Dice loss, which handles the data imbalance problem in segmentation tasks by optimizing the overlap between the segmented region and the true region; L BD (G,P) represents the boundary loss, which optimizes the model's segmentation of the target boundary, especially for irregular boundaries, significantly improving the segmentation accuracy; w CE 、w Dice and w BD are weight factors, G is the label, P is the segmentation result, G c and P c Represent the true label and predicted probability of category c, C is the number of categories, L mse (·) is the mean square error loss, and sobel(·) is the Sobel edge operator.

[0090] The loss L between the Mamba-UNet model and the WTS-UNet model part1 The supervised loss L of the labeled data sup and the consistency loss L for unlabeled data con Composition, see formula (22); Similarly, the loss L between the UNeXt model and the WTS-UNet model is obtained part2 ;

[0091] L part1 =L sup +λL con (twenty two)

[0092] Finally, the total loss is expressed as:

[0093] L total =w1L part1 +w2L part2 (twenty three)

[0094] Where λ is a weight factor, defined by a time-dependent Gaussian warm-up function: w1 and w2 are both weight factors that change over time. In the early stages of training, since the Mamba-UNet model focuses more on global features, its initial weight is larger, while the UNeXt model focuses more on local information, and its initial weight is smaller. It gradually increases during the training process and eventually reaches a stable value.

[0095] Example

[0096] First, the lightweight SAM model is fine-tuned using 300 manually annotated ROI labels and the ACDC dataset. Then, the results of the downstream segmentation task (segmentation map generated by the UNeXt or Mamba-UNet model) are superimposed with the corresponding fine ROI, and the superimposed fine ROI image is input into the lightweight SAM model as a pseudo-label for further training. The model is fine-tuned again, and its parameters are continuously updated during the segmentation model training process to improve the accuracy of ROI extraction.

[0097] The lightweight SAM model is used for ROI extraction, and IoU and Re are used as evaluation indicators. The results show that IoU and Re reach 88.19% and 95.63% respectively, achieving satisfactory results. This shows that it is feasible to fine-tune the lightweight SAM model using a small amount of data and use the fine-tuned lightweight SAM model for ROI extraction.

[0098] For the segmentation task, the present invention adopts five-fold cross validation to verify the robustness of the segmentation model. The model is trained five times for each fold and the average value and standard deviation are taken. At the same time, the present invention method is compared with the existing semi-supervised method DCT [1] ,ICT [2] , EM [3] UAMT [4] 、CCT [5] 、CPS [6] and URPC [7] For comparison, all methods used the same parameter configuration and training strategy, and all used ROI extraction and morphological processing to achieve the effect of control variables. Pr, Jaccard index, F1 score and HD were selected as evaluation indicators. The higher the Pr, the higher the accuracy of the model in the prediction results of positive samples. The higher the Jaccard index and F1 score, the better the model is in precision and recall. The lower the HD index, the better the model performs in capturing the boundaries and microstructures of the segmented targets. The performance of the five-fold cross-validation results of different methods on the four evaluation indicators is shown in Tables 1-4.

[0099] Table 1 Results of Pr↑ in five-fold cross validation

[0100]

[0101] Table 2 Results of F1↑ in five-fold cross validation

[0102]

[0103]

[0104] Table 3 Results of Jaccard↑ in five-fold cross validation

[0105]

[0106] Table 4 Results of HD↓ in five-fold cross validation

[0107]

[0108]

[0109] From the results of Tables 1, 2 and 3, it can be seen that the segmentation model of the present invention is superior to other methods in Pr, Jaccard index and F1 score, indicating that the model has significant advantages in regional coverage and prediction accuracy, and can achieve higher consistency between the target area and the real area. At the same time, Table 4 shows that the performance of the segmentation model of the present invention on the HD index is also better than other methods, indicating that the model has stronger performance in boundary fitting accuracy and detail capture, and can better achieve boundary and detail segmentation of the target area. In summary, the present invention effectively improves the model's ability in global feature extraction and detail segmentation through a multi-model cross-teaching strategy, and the segmentation model integrates the ASPP module and the WTS block for feature extraction, which effectively enhances the model's ability to segment pericardial fat edges and details.

[0110] The sources of the above existing methods are as follows:

[0111] [1]Qiao S, Shen W, Zhang Z, et al.Deep Co-Training for Semi-SupervisedImage Recognition[M / OL] / / Computer Vision–ECCV 2018,Lecture Notes in ComputerScience.2018:142-159.

[0112] [2]Verma V,Kawaguchi K,Lamb A,et al.Interpolation consistencytraining for semi-supervised learning[J / OL].Neural Networks, 2022:90-106.

[0113] [3]Vu T H,Jain H,Bucher M,et al.ADVENT:Adversarial EntropyMinimization for Domain Adaptation in Semantic Segmentation[C / OL].2019IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),Long Beach,CA,USA,2019,pp.2512-2521.

[0114] [4]Yu L,Wang S,Li X,et al.Uncertainty-Aware Self-ensembling Model forSemi-supervised 3D Left Atrium Segmentation[M / OL] / / Lecture Notes in ComputerScience,Medical Image Computing and Computer Assisted Intervention–MICCAI2019.2019:605-613.

[0115] [5]Ouali Y,Hudelot C,Tami M.Semi-Supervised Semantic SegmentationWith Cross-Consistency Training[C / OL] / / 2020IEEE / CVF Conference on ComputerVision and Pattern Recognition(CVPR),Seattle,WA,USA.2020.pp.12674–12684.

[0116] [6]Chen X,Yuan Y,Zeng G,et al.Semi-Supervised Semantic Segmentationwith Cross Pseudo Supervision[C / OL] / / 2021IEEE / CVF Conference on ComputerVision and Pattern Recognition(CVPR),Nashville,TN,USA.2021.pp.2613–2622.

[0117] [7]Luo X, Wang G, Liao W, et al. Semi-supervised medical imagesegmentation via uncertainty rectified pyramid consistency[J]. Medical ImageAnalysis 80(2022)102517.

[0118] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A semi-supervised pericardial fat segmentation method based on multi-model cross-teaching, characterized in that: The following steps are involved: S1. Use the lightweight SAM model to extract the ROI of pericardial fat tissue in cardiac MR images to obtain a rough ROI; The lightweight SAM model includes an image encoder and a visual Transformer decoder. An adapter layer is embedded in each Transformer block of the image encoder, and the adapter layer is located after the multi-head attention layer of the Transformer block. S2, perform morphological processing on the rough ROI to obtain a fine ROI; map the fine ROI onto the input image and crop it to obtain a fine ROI image; S3. Build a segmentation model based on TransUNet. The segmentation model includes an encoder, a decoder, and a bottleneck processing module connecting the encoder and the decoder. The bottleneck processing module includes an ASPP module and multiple WTS blocks. In the WTS block, the input features are normalized to obtain normalized input features; The normalized input features are input into two branches. In one branch, after passing through the linear layer, the depthwise separable convolutional layer, the SS2D module, and the normalization layer in sequence, they are multiplied pixel by pixel with the features of the normalized input features after passing through the linear layer. The multiplied features are then passed through the linear layer to obtain the output features of the branch. In the other branch, the normalized input features are passed through the frequency domain feature extraction module, the normalization layer, and the multilayer perceptron in sequence to obtain the output features of the branch. The output features of the two branches are added pixel by pixel to the input features of the WTS block to obtain the output features of the WTS block; S4. Based on multi-model cross-teaching, semi-supervised training is performed on the segmentation model, UNeXt and Mamba-UNet models, and the trained segmentation model is used for pericardial fat segmentation.

2. The semi-supervised pericardial fat segmentation method based on multi-model cross-teaching according to claim 1 is characterized in that: In step S4, the refined ROI image x i The segmentation maps are input into the Mamba-UNet model and the segmentation model respectively, and the pseudo labels are generated using the segmentation maps. The pseudo labels are expressed as: In the formula, This is the segmentation map generated by the Mamba-UNet model. The pseudo labels obtained are is the segmentation map generated by the segmentation model The pseudo-label obtained, Softmax(·) is the Softmax function, and T(t) is the temperature parameter during the tth round of training; Consistency loss L for unlabeled data con Defined as: Where, L softdice (·) represents a variant of Dice loss, N is the number of images in the current batch, Represents segmentation map The segmentation result at the position (h, w) in the middle, Represents pseudo labels The result at position (h, w); The supervision loss L of labeled data sup Defined as: L sup =w CE L CE (G,P)+w Dice L Dice (G,P)+w BD L BD (G,P) (18) L BD (G,P)=L mse (sobel(G),sobel(P)) (21) Where, L CE (G, P) represents the cross entropy loss, L Dice (G, P) represents Dice loss, L BD (G, P) represents the boundary loss, W CE 、w Dice and w BD are weight factors, G is the label, P is the segmentation result, G c and P c Represent the true label and predicted probability of category c, C is the number of categories, L mse (·) is the mean square error loss, and sobel(·) is the Sobel edge operator. The loss L between the Mamba-UNet model and the segmentation model part1 It is expressed as: THE part1 =L sup +λL con (22) Similarly, we get the loss L between the UNeXt model and the segmentation model part2 , then the total loss is expressed as: L total =w1L part1 +w2L part2 (23) In the formula, λ, w1, and w2 are all weight factors.

3. The semi-supervised pericardial fat segmentation method based on multi-model cross-teaching according to claim 1 or 2, characterized in that: In the adapter layer, the input features are average-pooled and then enter two processing branches. In one processing branch, the features obtained by average pooling are pre-activated, multiplied with the input features of the adapter layer, and then passed through the Transformer block to obtain the output features of the processing branch. In another processing branch, the features obtained by average pooling are sequentially subjected to depthwise convolution, ReLU activation, deconvolution, and Sigmod activation operations to obtain the output features of the processing branch; After the output features of the two processing branches are added, they are sequentially subjected to convolution, ReLU activation, deconvolution, and ReLU activation operations, and then added to the input features of the adapter layer to obtain the output features of the adapter layer.

4. The semi-supervised pericardial fat segmentation method based on multi-model cross-teaching according to claim 3 is characterized in that: For the morphological processing of the rough ROI, the rough ROI is first normalized and then binarized; then, the binary ROI is converted into an array and the connected domain is marked to obtain the maximum connected domain, and then the ROI containing only the maximum connected domain is obtained; the ROI containing only the maximum connected domain is subjected to a dilated convolution operation to obtain a fine ROI.

5. The semi-supervised pericardial fat segmentation method based on multi-model cross-teaching according to claim 1, characterized in that: In the frequency domain feature extraction module, the normalized input features are decomposed by wavelet to obtain low-frequency sub-bands, horizontal high-frequency sub-bands, vertical high-frequency sub-bands and diagonal high-frequency sub-bands; each sub-band is multiplied pixel by pixel with the corresponding learnable weight matrix to obtain weighted low-frequency sub-bands, weighted horizontal high-frequency sub-bands, weighted vertical high-frequency sub-bands and weighted diagonal high-frequency sub-bands; each weighted sub-band is subjected to inverse wavelet transform to obtain reconstructed low-frequency sub-bands, reconstructed horizontal high-frequency sub-bands, reconstructed vertical high-frequency sub-bands and reconstructed diagonal high-frequency sub-bands; all reconstructed sub-bands are fused to obtain the output features of the frequency domain feature extraction module.

Images