Consistency semi-supervised left atrium segmentation method based on teacher-student model

Abstract

The invention discloses a consistency semi-supervised left atrium segmentation method based on a teacher-student model, the teacher-student model is constructed, each of a teacher image segmentation network and a student image segmentation network comprises an encoder and three decoders, the input of the encoders forms the input of the teacher image segmentation network or the student image segmentation network, and the input of the decoders forms the input of the teacher image segmentation network or the student image segmentation network; the outputs of the three decoders form three outputs of a teacher image segmentation network or a student image segmentation network; the encoder and each decoder form a sub-segmentation network of a U-Net structure, and the intensities of adversarial disturbances added to feature transmission channels of different sub-segmentation networks are different. According to the method, a strategy of performing effective segmentation by using a small amount of marked data and a large amount of unmarked data is utilized, and through mutual consistency learning of resistance disturbance and a teacher-student image segmentation network, the segmentation accuracy can be improved while the marking requirement is reduced, so that the performance of a semi-supervised medical image segmentation task is effectively improved, and the efficiency of the semi-supervised medical image segmentation task is improved. And the training cost is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of nuclear magnetic resonance image segmentation, and in particular to a consistent semi-supervised left atrium segmentation method based on a teacher-student model. Background Art

[0002] Atrial fibrillation is one of the most common persistent cardiac arrhythmias, affecting the quality of life and health of millions of people worldwide. Atrial fibrillation is characterized by a disordered rhythm and extremely rapid frequency of atrial electrical signals, which prevents the heart from pumping blood effectively, thereby increasing the risk of heart failure and stroke. Abnormal hemodynamics causes blood to remain in the atria and form blood clots, which may flow into the brain with the blood and cause a stroke. Therefore, atrial fibrillation not only poses a serious threat to the patient's health, but also imposes a huge burden on the medical system.

[0003] For patients with atrial fibrillation, accurate segmentation of atrial structure helps to assess the degree of atrial fibrosis and develop catheter ablation strategies. Due to the anatomical structure and functional characteristics of the left atrium, atrial fibrillation is more common in the left atrium, so it is of great significance to establish a patient's left atrium segmentation model to assist doctors in treating atrial fibrillation. Traditional medical image segmentation methods rely on manual operations. Professional radiologists or technicians need to manually mark the atrial area in the image. This process is not only time-consuming and labor-intensive, but the accuracy and repeatability of the results are limited by the operator's experience and professional level. In addition, manual segmentation methods are inefficient when processing large amounts of data and are difficult to meet the needs of rapid clinical diagnosis. With the development of medical imaging technology, the amount of image data obtained has increased dramatically, making manual segmentation methods even more impractical. At the same time, manual segmentation methods are difficult to fully utilize the complex information contained in medical image data, which may lead to inaccurate diagnosis and treatment plans.

[0004] In recent years, deep learning technology has made significant progress in the field of medical image segmentation. By automatically learning the feature representation in image data, deep learning models can achieve high-accuracy image segmentation, greatly improving the efficiency and accuracy of segmentation tasks. Especially in the segmentation of left atrial structure in patients with atrial fibrillation, deep learning methods can automatically identify and segment atrial regions by learning the morphological features of the atrium from a large amount of image data, providing important information for the diagnosis and treatment of atrial fibrillation. However, the training of deep learning models usually requires a large amount of labeled data, and in the field of medical imaging, it is costly to obtain such data, which requires a lot of manpower and time. In addition, the quality of data annotation directly affects the performance of the model, and high-quality annotation often requires highly specialized knowledge. Therefore, how to effectively train deep learning models with limited labeled data and reduce dependence on a large amount of labeled data has become a research hotspot.

[0005] In summary, accurate diagnosis and treatment of atrial fibrillation urgently require accurate atrial structure segmentation. The advantages and disadvantages of traditional segmentation methods and deep learning technology indicate the direction of future research, that is, it is necessary to study a method that combines the powerful capabilities of deep learning and a small amount of labeled data to achieve efficient and accurate atrial structure segmentation. Summary of the invention

[0006] Aiming at the challenge of medical image segmentation under the condition of limited annotated data, the present invention provides a consistent semi-supervised left atrium segmentation method based on a teacher-student model.

[0007] To solve the above problems, the present invention is achieved through the following technical solutions:

[0008] The consistent semi-supervised left atrium segmentation method based on the teacher-student model includes the following steps:

[0009] Step 1, construct a teacher-student model consisting of a teacher image segmentation network and a student image segmentation network, wherein the teacher image segmentation network and the student image segmentation network have the same structure, both including 1 encoder and 3 decoders, and the 3 decoders share 1 encoder, the encoder input forms the input of the teacher image segmentation network or the student image segmentation network, and the 3 decoders output form the 3 outputs of the teacher image segmentation network or the student image segmentation network; the encoder and each decoder form a sub-segmentation network of a U-Net structure, that is, the encoder input forms the input of the sub-segmentation network, the encoder output is connected to the decoder input, and the decoder output forms the output of the sub-segmentation network, and a feature transmission channel for adding adversarial perturbations is provided between the ports of the same resolution of the encoder and the decoder; the strength of the adversarial perturbations added to the feature transmission channels of different sub-segmentation networks is different, wherein the adversarial perturbation added to the feature transmission channel between the encoder and the first decoder is less than the adversarial perturbation added to the feature transmission channel between the encoder and the second decoder, and the adversarial perturbation added to the feature transmission channel between the encoder and the second decoder is less than the adversarial perturbation added to the feature transmission channel between the encoder and the third decoder;

[0010] Step 2: Obtain a left atrial image data set, and after processing the left atrial image data set, obtain a labeled data sample set and an unlabeled data sample set;

[0011] Step 3: Initialize the teacher-student model using the labeled data sample set; then mix the labeled data sample set and the unlabeled data sample set into the initialized teacher-student model for consistency learning, calculate the consistency loss by comparing the output difference between the teacher image segmentation network and the student image segmentation network, and use the minimum loss consistency loss back propagation and exponential moving average strategy to update the teacher-student model parameters; then use some labeled data samples in the labeled data sample set to perform full-supervised training on the teacher-student model after consistency learning, and further refine the teacher-student model parameters;

[0012] Step 4: Use an independent left atrium image test set to evaluate the teacher-student model obtained in step 3, and use the sub-segmentation network with the best performance in the student image segmentation network as the final left atrium segmentation model;

[0013] Step 5: Input the left atrium image to be segmented into the left atrium segmentation model obtained in step 4 to obtain the segmentation result.

[0014] In the above scheme, the encoder is composed of 8 convolution and activation modules, 4 downsampling modules and a pyramid pooling module; the input of the first convolution and activation module forms the input of the encoder, the output of the first convolution and activation module is connected to the input of the second convolution and activation module, and the output of the second convolution and activation module is connected to the input of the first downsampling module; the output of the first downsampling module is connected to the input of the third convolution and activation module, the output of the third convolution and activation module is connected to the input of the fourth convolution and activation module, and the output of the fourth convolution and activation module is connected to the input of the second downsampling module; the output of the second downsampling module is connected to the input of the fifth convolution and activation module, the output of the fifth convolution and activation module is connected to the input of the sixth convolution and activation module, and the sixth convolution module is connected to the input of the fifth convolution and activation module. The output of the activation module is connected to the input of the third downsampling module; the output of the third downsampling module is connected to the input of the seventh convolution and activation module, the output of the seventh convolution and activation module is connected to the input of the eighth convolution and activation module, and the output of the eighth convolution and activation module is connected to the input of the fourth downsampling module; the output of the fourth downsampling module is connected to the input of the pyramid pooling module, and the output of the pyramid pooling module forms the output of the encoder; the input of the first convolution and activation module is used as the input of the first feature transmission channel, the input of the third convolution and activation module is used as the input of the second feature transmission channel, the input of the fifth convolution and activation module is used as the input of the third feature transmission channel, and the input of the seventh convolution and activation module is used as the input of the fourth feature transmission channel.

[0015] In the above scheme, the decoder is composed of 8 convolution and activation modules, 4 upsampling modules and 4 fusion modules; the input of the first upsampling module forms the input of the decoder, the output of the first upsampling module is connected to the input of the first convolution and activation module, the output of the first convolution and activation module is connected to the input of the second convolution and activation module, the output of the second convolution and activation module is connected to an input of the first fusion module, and the output of the first fusion module is connected to the input of the second upsampling module; the output of the second upsampling module is connected to the input of the third convolution and activation module, the output of the third convolution and activation module is connected to the input of the fourth convolution and activation module, the output of the fourth convolution and activation module is connected to an input of the second fusion module, and the output of the second fusion module is connected to the input of the third upsampling module; the output of the third upsampling module is connected to the input of the fifth convolution and activation module Input, the output of the fifth convolution and activation module is connected to the input of the sixth convolution and activation module, the output of the sixth convolution and activation module is connected to an input of the third fusion module, and the output of the third fusion module is connected to the input of the fourth upsampling module; the output of the fourth upsampling module is connected to the input of the seventh convolution and activation module, the output of the seventh convolution and activation module is connected to the input of the eighth convolution and activation module, the output of the eighth convolution and activation module is connected to an input of the fourth fusion module, and the output of the fourth fusion module forms the output of the decoder; the other input of the first fusion module is used as the output of the first feature transmission channel, the other input of the second fusion module is used as the output of the first feature transmission channel, the other input of the third fusion module is used as the output of the first feature transmission channel, and the other input of the fourth fusion module is used as the output of the first feature transmission channel.

[0016] In the above scheme, the upsampling module of the first decoder adopts bilinear interpolation, the upsampling module of the second decoder adopts transposed convolution, and the upsampling module of the third decoder adopts sub-pixel convolution.

[0017] In step 3 above, when learning the consistency of the teacher-student model:

[0018] For the student image segmentation network, the parameters are updated by minimizing the consistency loss back propagation, namely:

[0019]

[0020] For the teacher image segmentation network, an exponential moving average strategy is used to update its parameters, namely:

[0021] θ t ←α·θ t +(1-α)·θ s

[0022] Among them, θ s represents the parameters of the student image segmentation network, θ trepresents the parameters of the teacher image segmentation network, η represents the learning rate, α represents the weight of the historical parameters, represents the gradient of the parameters of the student image segmentation network, L represents the consistency loss,

[0023]

[0024] Where i represents the current sample index, j represents the current output port index, represents the output probability distribution of the student image segmentation network at the j output ports for the i-th sample, represents the output probability distribution of the teacher image segmentation network at the j output ports for the i-th sample, ‖*‖ 2 Represents the square of the Euclidean distance.

[0025] Compared with the prior art, the present invention has the following characteristics:

[0026] (1) Construct a teacher-student model consisting of a teacher image segmentation network and a student image segmentation network, wherein the teacher image segmentation network and the student image segmentation network each include an encoder and three decoders, and the three decoders share one encoder, and the encoder and each decoder form a sub-segmentation network of the U-Net structure. The connection channels between the shared encoder and the three decoders are independent of each other, and there is no direct connection between the decoders. During the training process, although the decoders are independent, they can work together with the encoder through the back-propagation algorithm to optimize the model parameters.

[0027] (2) The decoders of the three U-Net sub-segmentation networks of the teacher image segmentation network and the student image segmentation network adopt different upsampling strategy combinations (transposed convolution, bilinear interpolation, sub-pixel convolution), which can combine the advantages of their respective decoders and weaken their respective shortcomings, so as to facilitate the consistency learning of subsequent models and improve the model training effect.

[0028] (3) Adversarial perturbations of different strengths are added to the feature transmission channels of the three U-Net sub-segmentation networks of the teacher image segmentation network and the student image segmentation network. The introduced adversarial perturbations enhance the effectiveness of the training data by directly challenging the weaknesses of the model, forcing the model to learn more generalized feature representations. Adding adversarial perturbations of different strengths to different types of decoders can generate output differences containing richer abstract information, thereby improving the generalization ability of the model and significantly improving the robustness of the model.

[0029] (4) The constructed teacher-student model can be trained by mixing a small amount of labeled data and a large amount of unlabeled data, which significantly reduces the dependence on a large amount of manually labeled data, reduces the cost of data preparation and the cost of obtaining data samples, and effectively improves the performance of medical image segmentation tasks under conditions of limited annotation resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Flowchart of the consistent semi-supervised left atrium segmentation method based on the teacher-student model.

[0031] Figure 2 Model for teachers and students.

[0032] Figure 3 This is the schematic diagram of the sub-segmentation network of the U-Net structure. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0034] The consistency of the teacher-student model based on the semi-supervised left atrium segmentation method, such as Figure 1 As shown, the steps include:

[0035] Step 1: Construct the teacher-student model.

[0036] The teacher-student model of the present invention is composed of a teacher image segmentation network and a student image segmentation network. Figure 2 As shown, the structures of the teacher image segmentation network and the student image segmentation network are the same, both including 1 encoder and 3 decoders, and the 3 decoders share the same encoder, the input of the encoder forms the input of the teacher image segmentation network or the student image segmentation network, and the output of the 3 decoders forms the 3 outputs of the teacher image segmentation network or the student image segmentation network. The shared encoder is connected to the three decoders through independent connection channels, and the decoders are not directly connected to each other. During the training process, although the decoders are independent of each other, they can act on the shared encoder together through the back propagation algorithm to optimize the model parameters. The shared encoder and each decoder together constitute a sub-segmentation network based on the U-Net structure, and the strength of the adversarial perturbation added to the feature transmission channel of each sub-segmentation network is different. Based on the decoding network fused by multiple feature maps, feature information from different levels is more effectively integrated to improve the ability to restore the details of the segmentation. The adversarial perturbation calculates the gradient of the loss function of the model to the input image to find the direction in which the model output is most sensitive to the input change, that is, the direction that misleads the model to make an incorrect judgment, and generates a small but meaningful adversarial perturbation according to this direction.

[0037] The sub-segmentation network of the present invention draws on the classic U-Net structure, and its structure is as follows Figure 3 As shown in the figure, the input of the encoder forms the input of the sub-segmentation network, the output of the encoder is connected to the input of the decoder, the output of the decoder forms the output of the sub-segmentation network, and a feature transmission channel for adding adversarial perturbations is set between the ports of the same resolution of the encoder and the decoder. The encoder part extracts features of different resolutions through multiple downsampling layers, while the decoder part uses multiple upsampling layers to restore the resolution of the image and uses the jump connection of the feature transmission channel to fuse features so that the model can understand the data more deeply.

[0038] The encoder is composed of 8 convolution and activation modules, 4 downsampling modules and a pyramid pooling module. The input of the first convolution and activation module forms the input of the encoder, the output of the first convolution and activation module is connected to the input of the second convolution and activation module, and the output of the second convolution and activation module is connected to the input of the first downsampling module; the output of the first downsampling module is connected to the input of the third convolution and activation module, the output of the third convolution and activation module is connected to the input of the fourth convolution and activation module, and the output of the fourth convolution and activation module is connected to the input of the second downsampling module; the output of the second downsampling module is connected to the input of the fifth convolution and activation module, the output of the fifth convolution and activation module is connected to the input of the sixth convolution and activation module, and the output of the sixth convolution and activation module is connected to the input of the third downsampling module; the output of the third downsampling module is connected to the input of the seventh convolution and activation module, the output of the seventh convolution and activation module is connected to the input of the eighth convolution and activation module, and the output of the eighth convolution and activation module is connected to the input of the fourth downsampling module; the output of the fourth downsampling module is connected to the input of the pyramid pooling module, and the output of the pyramid pooling module forms the output of the encoder. The input of the first convolution and activation module is used as the input of the first feature transmission channel, the input of the third convolution and activation module is used as the input of the second feature transmission channel, the input of the fifth convolution and activation module is used as the input of the third feature transmission channel, and the input of the seventh convolution and activation module is used as the input of the fourth feature transmission channel.

[0039] The above decoder consists of 8 convolution and activation modules, 4 upsampling modules and 4 fusion modules. The input of the first upsampling module forms the input of the decoder, the output of the first upsampling module is connected to the input of the first convolution and activation module, the output of the first convolution and activation module is connected to the input of the second convolution and activation module, the output of the second convolution and activation module is connected to an input of the first fusion module, and the output of the first fusion module is connected to the input of the second upsampling module; the output of the second upsampling module is connected to the input of the third convolution and activation module, the output of the third convolution and activation module is connected to the input of the fourth convolution and activation module, the output of the fourth convolution and activation module is connected to an input of the second fusion module, and the output of the second fusion module is connected to the input of the third upsampling module; the output of the third upsampling module is connected to the input of the fifth convolution and activation module, the output of the fifth convolution and activation module is connected to the input of the sixth convolution and activation module, the output of the sixth convolution and activation module is connected to an input of the third fusion module, and the output of the third fusion module is connected to the input of the fourth upsampling module; the output of the fourth upsampling module is connected to the input of the seventh convolution and activation module, the output of the seventh convolution and activation module is connected to the input of the eighth convolution and activation module, the output of the eighth convolution and activation module is connected to an input of the fourth fusion module, and the output of the fourth fusion module forms the output of the decoder. Another input of the first fusion module is used as the output of the first feature transmission channel, another input of the second fusion module is used as the output of the first feature transmission channel, another input of the third fusion module is used as the output of the first feature transmission channel, and another input of the fourth fusion module is used as the output of the first feature transmission channel.

[0040] In view of the fact that the present invention adds two decoders with different upsampling strategies on the basis of the classic U-Net, it is necessary to explain its design considerations. Although the transposed convolution used by the classic U-Net decoder can learn effective parameters in the upsampling process and is conducive to the restoration of image details and textures, its inherent uneven pixel arrangement may cause checkerboard artifacts, and the number of parameters is relatively large, and the computational cost is high. To this end, considering that subsequent consistency learning can integrate the advantages of different decoders and complement their deficiencies, the present invention additionally adds two decoders that use bilinear interpolation and sub-pixel convolution as upsampling strategies respectively.

[0041] For bilinear interpolation: 1) Bilinear interpolation is a non-learning upsampling method with fast processing speed and low computational cost. This makes the model more efficient in real-time or resource-constrained applications. 2) Bilinear interpolation mainly estimates new pixel values ​​by weighted averaging of adjacent pixel values. This method can naturally smooth the image and help provide better visual effects in some medical image processing tasks that require smooth edges and transitions. For areas with delicate boundaries and structures such as the left atrium, bilinear interpolation can provide smoother image transitions and avoid abrupt changes caused by upsampling, which is particularly important for accurate identification of cardiac structures. 3) Transposed convolution may cause a checkerboard effect when upsampling, because the convolution kernel and step size may cause uneven pixel arrangement. In left atrium segmentation, this effect may cause false positives or false negatives in the segmentation results, affecting the accuracy of diagnosis. Bilinear interpolation does not have this problem. It ensures the continuity and naturalness of the image, which is very critical for medical applications that require high-quality edge details. 4) Bilinear interpolation provides a non-learning upsampling method with higher consistency and stability of output than transposed convolution. In transposed convolution, insufficient model training or slight changes in data may lead to very different output results, while bilinear interpolation can ensure consistent output under the same input conditions. At the same time, because bilinear interpolation does not depend on the parameters learned by the model, it can reduce the risk of overfitting to a certain extent. In specific applications, this can help the model better generalize to new, unseen data. In general, in applications such as left atrium segmentation that require extremely high image quality, bilinear interpolation can make up for the corresponding shortcomings of transposed convolution in consistency learning with its smooth image quality, efficient computing performance and excellent stability. This helps to improve segmentation accuracy, reduce potential diagnostic errors, and speed up processing. Therefore, the present invention selects bilinear interpolation as one of the upsampling methods of the decoder.

[0042] For sub-pixel convolution: 1) Sub-pixel convolution achieves upsampling by increasing the number of channels in the feature map and then performing a pixel rearrangement process. This method allows the model to increase the details and complexity of the data without directly increasing the spatial dimension, thereby better maintaining the details and clarity of the image after upsampling. For left atrium segmentation, this means that the edges of the heart structure can be more accurately portrayed, improving the accuracy of segmentation, especially in key areas such as the edges of the heart. 2) Sub-pixel convolution avoids the checkerboard effect through pixel rearrangement, providing smoother and continuous image features, which is conducive to generating more accurate and natural segmentation results. 3) In sub-pixel convolution, upsampling is achieved by first performing a deep convolution to increase the number of channels and then performing pixel rearrangement. This process allows the model to make better use of network parameters when extracting features, enhancing the ability to extract features. This is particularly beneficial for the segmentation of complex structural areas such as the left atrium, because the model needs to accurately understand and parse the complex structure of the heart. 4) Because sub-pixel convolution can restore image details more naturally, it helps the model show better robustness when processing a variety of different medical images. Especially when faced with the complex and ever-changing image features of the left atrium, it is possible to more reliably produce high-quality segmentation results. In general, the use of sub-pixel convolution in left atrium segmentation can bring higher image segmentation accuracy than traditional transposed convolution, reduce common problems in the image processing process such as the checkerboard effect, and improve the computational efficiency and robustness of the model. The corresponding shortcomings of transposed convolution can be compensated in consistency learning. This helps to improve segmentation accuracy, reduce potential diagnostic errors, and speed up processing speed. Therefore, the present invention selects sub-pixel convolution as another upsampling method for the decoder.

[0043] Accordingly, the three decoders of the three U-Net structured sub-segmentation networks of the teacher image segmentation network and the student image segmentation network of the present invention respectively adopt different upsampling strategies, wherein the upsampling module of the first decoder adopts bilinear interpolation, the upsampling module of the second decoder adopts transposed convolution, and the upsampling module of the third decoder adopts sub-pixel convolution.

[0044] Since accurate segmentation of the left atrium is crucial for the diagnosis and treatment planning of heart disease, it is necessary to ensure that the segmentation model can show high reliability under various conditions. Adversarial perturbation can be regarded as a carefully designed input perturbation designed to challenge the decision boundary of the model. It can effectively reveal the potential defects and deficiencies of the model by directly challenging the weaknesses of the model to enhance the effectiveness of the training data. Adversarial perturbation can reveal the vulnerability of the model in the face of carefully designed input perturbations by purposefully generating perturbations. Compared with ordinary perturbations, adversarial perturbations can significantly improve the robustness of the model. Adversarial perturbations simulate input changes at the edge of the data distribution or at the boundary conditions where the model is prone to errors, forcing the model to learn more generalized feature representations, thereby performing better when facing unknown data. These characteristics provide important clues for improving model structure and training strategies. To this end, in the specific medical image processing task of left atrium segmentation, the use of adversarial perturbations can bring a series of advantages and significance, especially in improving the robustness, accuracy and generalization ability of segmentation models. In terms of improving the robustness of the model, in the real world, medical images are often affected by various noises, such as different scanning equipment and changes in operating conditions. Adversarial perturbations can train the model to maintain its performance in the face of real-world data by simulating these potential interferences, especially when the data quality is uneven. And because the left atrium may show significant morphological differences between different patients, through adversarial training, the model can learn a wider range of feature representations to better handle this anatomical variation. In terms of enhancing the generalization ability of the segmentation model, adversarial perturbations can simulate situations that are not covered in the training data, helping the model to make accurate predictions when encountering new or rare situations, which is crucial to ensure the wide applicability of the model in clinical applications. In terms of improving the accuracy of the segmentation model, during the training process that includes adversarial samples, the model needs to learn how to correctly segment the left atrium from complex or subtly changing images. This high-difficulty training allows the model to learn deeper and more robust features, ultimately improving its accuracy in standard and challenging tasks.

[0045] Although compound perturbations can simulate a variety of actual changes, as shown in the Chinese invention patent application "Left Atrial Image Segmentation Method Based on Consistency Learning" with announcement number CN116958544A, compound perturbations can improve the robustness of the model to natural interference, but adversarial perturbations focus on the most vulnerable parts of the model and provide some unique advantages. First, adversarial perturbations are more targeted than compound perturbations. Adversarial perturbations are specially designed to mislead the model. They are calculated to precisely target the weaknesses of the model. This method can effectively reveal the most unstable aspects of the model, thereby helping developers to understand and improve the model more deeply. Secondly, adversarial perturbations are also more efficient. Adversarial perturbations are usually generated through gradient optimization, which makes them more effective in testing and enhancing models. In contrast, compound perturbations may include a series of untargeted changes that increase the general robustness of the model but may not be enough to touch the key weaknesses of the model. Furthermore, adversarial perturbations can better improve the generalization ability of the model. Since adversarial perturbations are systematic and targeted, they can help the model learn deeper and more abstract feature representations during training, thereby improving the generalization ability of the model on unseen data. This training method forces the model to learn not only how to deal with specific and obvious perturbations, but also to learn to maintain stability in a wider range of adversarial scenarios. Finally, adversarial perturbations can also provide clearer optimization goals compared to compound perturbations, because when using adversarial perturbations to train models, the optimization goal is very clear-reducing the error on adversarial samples. This clear goal helps the direction of model training, while compound perturbations may involve optimization in multiple different directions, which may sometimes lead to a dispersion of training focus. In summary, for the scenario of left atrium segmentation, adversarial perturbations can better improve the robustness, accuracy and generalization ability of the segmentation model compared to general random perturbations and even compound perturbations (a method that includes multiple different types of perturbations, such as random noise, blur, brightness changes, etc.).

[0046] Accordingly, adversarial perturbations are used in the feature transmission channels between the encoders and decoders of the three U-Net structured sub-segmentation networks of the teacher image segmentation network and the student image segmentation network of the present invention.

[0047] Since the teacher image segmentation network and the student image segmentation network add two decoders with different upsampling methods on the basis of the classic U-net, it is necessary to design adversarial perturbations of different strengths for different upsampling structures in their respective feature transmission channels, aiming to generate more diverse feature maps, thereby promoting the model to learn more robust and generalized feature representations. For the decoder using transposed convolution as upsampling: Since the weights of the transposed convolution are learned, it has a certain adaptability to small changes in the input, but it will still be affected by large perturbations, especially when the perturbation affects the learned convolution kernel. Applying moderate perturbations to the decoder can help test and enhance its ability to handle more complex image structures while avoiding performance degradation caused by excessive perturbations. Therefore, moderate adversarial perturbations are applied to the decoder using transposed convolution as upsampling. For the decoder using bilinear interpolation as upsampling: Since bilinear interpolation directly depends on the values ​​of neighboring pixels, small perturbations to these pixel values ​​may directly affect the interpolation results, especially at the edges or details of the image. Applying a smaller adversarial perturbation to the decoder can prevent excessive impact on the accuracy of the interpolation, while still being able to test its robustness to small changes. Therefore, a smaller adversarial perturbation is applied to the decoder that uses bilinear interpolation as upsampling. For the decoder that uses sub-pixel convolution as upsampling: sub-pixel convolution is usually tolerant to input perturbations due to its operation mode. This is because it obtains high-resolution output by rearranging pixels, rather than directly relying on the precise values ​​of surrounding pixels, which has a higher tolerance. Applying a larger perturbation to the decoder can help further enhance the model's ability to handle complex input changes, ensuring good performance even under larger input perturbations. Therefore, a larger adversarial perturbation is applied to the decoder that uses sub-pixel convolution as upsampling.

[0048] The shared encoder captures the core features and information of the input data, and transmits these features and information to different decoders respectively, and adds adversarial perturbations of different strengths in their respective feature transmission channels. The three decoders will receive features with different interferences and output different segmentation results. This can test and enhance the robustness of each decoder in a targeted manner, especially when they process different types of image features (such as details of different scales). Each decoder can be trained to specialize in its specific task, such as one processing finer texture details and another processing more macroscopic structures.

[0049] Compared with the method of adding perturbations to different layers of different decoders adopted by the Chinese invention patent application "Left Atrial Image Segmentation Method Based on Consistency Learning" with announcement number CN116958544A, the present invention applies perturbations to the entire decoder structure, which can more systematically evaluate and improve the robustness of the decoder as a whole, avoiding the local optimization problems that may be caused by focusing only on inter-layer robustness. At the same time, by applying perturbations of different intensities to each decoder, the entire network can be made more robust when integrating information obtained from different decoders. This method can optimize the model's understanding and processing of complex image structures better than simply adding perturbations to different layers. And because consistency constraints are required later, perturbations of different intensities on different decoders will enhance the model's ability to maintain output consistency, which can further improve the accuracy and robustness of the model, which is particularly important for left atrium segmentation. And because applying adversarial perturbations of different intensities to different decoders provides greater flexibility, the intensity of the perturbation can be adjusted according to specific application requirements to maximize the performance of the model in specific scenarios. This method has strong scalability and is convenient for further optimization and adaptation to more task requirements in the future. In summary, applying adversarial perturbations of different strengths to different decoders can more effectively utilize the advantages of the multi-decoder architecture, specifically enhance the robustness of the model, optimize the effect of consistency loss, and facilitate more refined model performance tuning. This strategy is particularly advantageous in dealing with left atrium segmentation tasks with complex structures and requirements.

[0050] Accordingly, the feature transmission channels of the three U-Net structured sub-segmentation networks of the teacher image segmentation network and the student image segmentation network of the present invention adopt adversarial perturbations of different intensities, wherein the adversarial perturbation added to the feature transmission channel between the encoder and the first decoder is smaller than the adversarial perturbation added to the feature transmission channel between the encoder and the second decoder, and the adversarial perturbation added to the feature transmission channel between the encoder and the second decoder is smaller than the adversarial perturbation added to the feature transmission channel between the encoder and the third decoder.

[0051] Step 2: Build sample data.

[0052] First, a Magnetic Resonance Imaging (MRI) three-dimensional heart data set is obtained. The images in the left atrial image data set can be derived from existing data on the Internet or collected by a nuclear magnetic resonance imaging device. After the left atrial image data set is processed, that is, the images in the left atrial image data set are subjected to image preprocessing (such as normalization processing, etc.) and data enhancement (such as flipping, scaling and / or random cropping, etc.), a part of the images that have not been calibrated by professional physicians directly form an unlabeled data sample set, and the other part of the images that have been calibrated by professional physicians form a labeled data sample set.

[0053] Step 3: Use sample data to learn the teacher-student model.

[0054] Step 3.1 Initialization: Initialize the teacher-student model using the labeled data sample set.

[0055] The encoder part of the teacher-student model is initialized with pre-trained weights, which enables the encoder to have better feature extraction capabilities. The decoder part uses Xavier initialization, which can automatically adjust the initial distribution of weights according to the number of input and output neurons, thereby effectively alleviating the gradient vanishing and gradient exploding problems, and contributing to the stability and efficiency of neural network training. Finally, the entire network is fine-tuned using the labeled left atrial image dataset.

[0056] Step 3.2, consistency learning: The labeled data sample set and the unlabeled data sample set are mixed and input into the initialized teacher-student model for consistency learning. The consistency loss is calculated by comparing the output differences between the teacher image segmentation network and the student image segmentation network, and the teacher-student model parameters are updated by minimizing the loss consistency loss back propagation and EMA strategy.

[0057] The labeled data sample set and the unlabeled data sample set are mixed as input data and input into the teacher-student model for consistency learning. In the initial stage of training, more labeled data samples are used for model training. As the training progresses, the proportion of unlabeled data samples is gradually increased to increase the weight of the impact of unlabeled data on performance. During consistency learning, the encoders of the teacher image segmentation network and the student image segmentation network of the teacher-student model capture the core features and information of the input data, and transmit these features and information to different decoders respectively, and add adversarial perturbations of different strengths in their respective feature transmission channels to obtain multiple segmentation results. The mean square error of the output probability distribution of each student image segmentation network and all teacher image segmentation networks is calculated between the three results of the teacher image segmentation network and the student image segmentation network, that is, the consistency loss. The consistency loss is calculated by comparing the output difference between the student image segmentation network and the teacher image segmentation network. The parameters of the student image segmentation network are first updated by backpropagation by minimizing the consistency loss, and then the parameters of the teacher image segmentation network are updated using the EMA strategy. This process is repeated until all data are trained.

[0058] For the student image segmentation network, the parameters are updated by minimizing the consistency loss back propagation. Let P t is the output probability distribution of the student image segmentation network, N is the total number of data samples, then the mean square error consistency loss (L) can be defined as:

[0059]

[0060] Among them, L represents the calculated consistency loss value, which is used to measure the difference between the output of the student image segmentation network and the teacher image segmentation network. N represents the total number of data samples, i represents the current sample index, traversing all samples from 1 to N. j represents the current output port index, traversing all output ports 1, 2, 3. It represents the square of the Euclidean distance between the output probability distributions of the student image segmentation network and the teacher image segmentation network at the j output port for the i-th sample, which measures the difference between the output of the student image segmentation network at this port and the output of the teacher image segmentation network at this port.

[0061] During the training process, our goal is to minimize this loss L. Specifically, the gradient of the loss function L with respect to the parameters of the student image segmentation network is calculated, and the parameters of the student image segmentation network are updated:

[0062]

[0063] Among them, θ s represents the parameters of the student image segmentation network. η is the learning rate, which is a positive scalar that determines the step size of each parameter update. Represents the gradient of the parameters of the student image segmentation network. The gradient indicates the direction in which the loss function increases fastest, and its negative direction is the direction in which the loss function decreases fastest. is the loss function L with respect to the parameters θ of the student image segmentation network s gradient.

[0064] For the teacher image segmentation network, the EMA (exponential moving average) strategy is used to update its parameters to ensure that the update of the teacher image segmentation network parameters is smoother:

[0065] θ t ←α·θ t +(1-α)·θ s

[0066] Among them, θ t is the parameter of the teacher image segmentation network. α is a coefficient close to 1, which represents the weight of the historical parameters. It controls the weight of the historical parameters when the teacher network parameters are updated.

[0067] In this way, the model ensures that the student image segmentation network can learn stable and accurate knowledge from the teacher image segmentation network by minimizing the consistency loss, thereby improving the performance of the student image segmentation network. Compared with traditional consistency learning, teacher-student model consistency learning increases the stability of the model output through the exponential moving average (EMA) mechanism. The EMA strategy helps to smooth the parameter update process of the teacher image segmentation network, thereby suppressing potential fluctuations during training and improving the consistency and stability of the teacher network output. This smooth output is more reliable when generating pseudo labels and can reduce noise interference during the student network learning process. This smooth output is particularly important in left atrium segmentation because it ensures that the model can maintain good performance even in the face of individual differences and technical deviations in the dataset. Traditional consistency learning usually relies on multiple outputs of a single model (or multiple models processing the same task), which are directly compared through consistency loss to encourage the model output to be consistent when accepting slightly perturbed inputs. Although this also helps to improve robustness, without the help of the EMA mechanism, the model output may be more susceptible to random fluctuations during training. At the same time, in terms of the use of unlabeled data, in the teacher-student model, the output of the teacher image segmentation network (based on the parameters of the student image segmentation network smoothed by EMA) is used to generate pseudo labels, which are fed back to the student image segmentation network as consistency targets. This not only enhances the guiding role of the teacher image segmentation network, but also enables the student image segmentation network to learn effectively without explicit annotations. For left atrium segmentation, this means that the model can better learn from a large number of unlabeled medical images in the absence of sufficient labeled data, improving the accuracy and consistency of its predictions. Traditional consistency learning usually requires enforcing consistency between different outputs of the model or between changing inputs generated by data augmentation. This method can also utilize unlabeled data, but lacks a teacher image segmentation network whose output is smoothed and stabilized by EMA to guide the learning process. At the same time, the combination of smoothing and consistency learning provided by EMA enables the teacher-student model to better generalize to new and unseen data. In left atrium segmentation, this helps the model process image data from different patients, different devices, or obtained under different conditions, ensuring the accuracy and reliability of the segmentation results.

[0068] Step 3.3, fully supervised training: Use some labeled data samples in the labeled data sample set to perform fully supervised training on the teacher-student model after consistency learning, further refine the parameters of the teacher-student model, impose semantic constraints, and use the early stopping method to monitor the training process and save the optimal network parameters to improve the model segmentation accuracy;

[0069] Step 4: Use an independent left atrium image test set to evaluate the teacher-student model obtained in step 3 to verify its performance, and use the sub-segmentation network with the best performance in the student image segmentation network as the final left atrium segmentation model.

[0070] In order to test the generalization ability of the obtained network, the test set is used to verify it. The performance indicators for measuring the segmentation results are Dice coefficient, Jaccard coefficient, 95% Hausdorff distance (95% HD) and average surface distance ASD.

[0071] The Dice coefficient mainly evaluates the accuracy of segmentation by measuring the similarity of the set. The formula of the Dice coefficient is as follows:

[0072]

[0073] Among them, A and B represent the true data label of the left atrium and the predicted left atrium area, respectively.

[0074] The Jaccard coefficient mainly evaluates the accuracy of segmentation by measuring the degree of overlap of regions. The specific definition of Jaccard is as follows:

[0075]

[0076] Among them, A and B represent the true data label of the left atrium and the predicted left atrium area, respectively.

[0077] 95% HD is evaluated by calculating the distance to the segmentation boundary. 95% HD is defined as follows:

[0078] HD 95 (A,B)=max(h 95 (A,B),h 95 (B,A))

[0079] Among them, A represents the point set of the left atrium boundary in the predicted segmentation result, and B represents the point set of the left atrium boundary in the real segmentation annotation.

[0080] The average surface distance (ASD) is a measure of the difference between two surfaces. ASD is defined as follows:

[0081]

[0082] Among them, A represents the set of left atrium boundary points predicted from the segmentation model, a∈A: refers to a specific point in point set A, representing a point on the left atrium boundary predicted by the model. B represents the set of points on the actual left atrium boundary, b∈B: refers to a specific point in point set B, representing a point on the actual left atrium boundary.

[0083] Step 5: Input the left atrium image to be segmented into the left atrium segmentation model obtained in step 4 to obtain the segmentation result.

[0084] The present invention designs an average teacher segmentation network consisting of a shared encoder and three slightly different independent decoders, and sets adversarial noises of different strengths according to the sensitivity of different decoders. When training the average teacher segmentation network, the teacher-student model is first initialized with a labeled data sample set; then a mixed labeled data sample set and an unlabeled data sample set are input into the teacher-student model for consistency learning; and then a part of the labeled data samples in the labeled data sample set are used to perform full-supervised training on the teacher-student model to further refine the parameters of the teacher-student model. Afterwards, the trained teacher-student model is evaluated using an independent test set, and the sub-segmentation network with the best performance is used as the final left atrium segmentation model to segment the left atrium image to be segmented. The present invention uses a strategy of effective segmentation using a small amount of labeled data and a large amount of unlabeled data. Through adversarial perturbations and mutual consistency learning of the teacher-student image segmentation network, it can reduce the need for annotation while improving segmentation accuracy, thereby effectively improving the performance of semi-supervised medical image segmentation tasks and greatly reducing training costs.

[0085] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A consistent semi-supervised left atrium segmentation method based on a teacher-student model, characterized by: The steps include: Step 1, construct a teacher-student model consisting of a teacher image segmentation network and a student image segmentation network, wherein the teacher image segmentation network and the student image segmentation network have the same structure, both including 1 encoder and 3 decoders, and the 3 decoders share 1 encoder, the encoder input forms the input of the teacher image segmentation network or the student image segmentation network, and the 3 decoders output form the 3 outputs of the teacher image segmentation network or the student image segmentation network; the encoder and each decoder form a sub-segmentation network of a U-Net structure, that is, the encoder input forms the input of the sub-segmentation network, the encoder output is connected to the decoder input, and the decoder output forms the output of the sub-segmentation network, and a feature transmission channel for adding adversarial perturbations is provided between the ports of the same resolution of the encoder and the decoder; the strength of the adversarial perturbations added to the feature transmission channels of different sub-segmentation networks is different, wherein the adversarial perturbation added to the feature transmission channel between the encoder and the first decoder is less than the adversarial perturbation added to the feature transmission channel between the encoder and the second decoder, and the adversarial perturbation added to the feature transmission channel between the encoder and the second decoder is less than the adversarial perturbation added to the feature transmission channel between the encoder and the third decoder; Step 2: Obtain a left atrial image data set, and after processing the left atrial image data set, obtain a labeled data sample set and an unlabeled data sample set; Step 3: Initialize the teacher-student model using the labeled data sample set; then mix the labeled data sample set and the unlabeled data sample set into the initialized teacher-student model for consistency learning, calculate the consistency loss by comparing the output difference between the teacher image segmentation network and the student image segmentation network, and use the minimum loss consistency loss back propagation and exponential moving average strategy to update the teacher-student model parameters; then use some labeled data samples in the labeled data sample set to perform full-supervised training on the teacher-student model after consistency learning, and further refine the teacher-student model parameters; Step 4: Use an independent left atrium image test set to evaluate the teacher-student model obtained in step 3, and use the sub-segmentation network with the best performance in the student image segmentation network as the final left atrium segmentation model; Step 5: Input the left atrium image to be segmented into the left atrium segmentation model obtained in step 4 to obtain the segmentation result.

2. The consistent semi-supervised left atrium segmentation method based on the teacher-student model according to claim 1 is characterized in that: The encoder consists of 8 convolution and activation modules, 4 downsampling modules and a pyramid pooling module; The input of the first convolution and activation module forms the input of the encoder, the output of the first convolution and activation module is connected to the input of the second convolution and activation module, and the output of the second convolution and activation module is connected to the input of the first downsampling module; the output of the first downsampling module is connected to the input of the third convolution and activation module, the output of the third convolution and activation module is connected to the input of the fourth convolution and activation module, and the output of the fourth convolution and activation module is connected to the input of the second downsampling module; the output of the second downsampling module is connected to the input of the fifth convolution and activation module, the output of the fifth convolution and activation module is connected to the input of the sixth convolution and activation module, and the output of the sixth convolution and activation module is connected to the input of the third downsampling module; the output of the third downsampling module is connected to the input of the seventh convolution and activation module, the output of the seventh convolution and activation module is connected to the input of the eighth convolution and activation module, and the output of the eighth convolution and activation module is connected to the input of the fourth downsampling module; the output of the fourth downsampling module is connected to the input of the pyramid pooling module, and the output of the pyramid pooling module forms the output of the encoder; The input of the first convolution and activation module is used as the input of the first feature transmission channel, the input of the third convolution and activation module is used as the input of the second feature transmission channel, the input of the fifth convolution and activation module is used as the input of the third feature transmission channel, and the input of the seventh convolution and activation module is used as the input of the fourth feature transmission channel.

3. The consistent semi-supervised left atrium segmentation method based on the teacher-student model according to claim 1, characterized in that: The decoder consists of 8 convolution and activation modules, 4 upsampling modules and 4 fusion modules; The input of the first upsampling module forms the input of the decoder, the output of the first upsampling module is connected to the input of the first convolution and activation module, the output of the first convolution and activation module is connected to the input of the second convolution and activation module, the output of the second convolution and activation module is connected to an input of the first fusion module, and the output of the first fusion module is connected to the input of the second upsampling module; the output of the second upsampling module is connected to the input of the third convolution and activation module, the output of the third convolution and activation module is connected to the input of the fourth convolution and activation module, the output of the fourth convolution and activation module is connected to an input of the second fusion module, and the output of the second fusion module is connected to the input of the third upsampling module; the output of the third upsampling module is connected to the input of the fifth convolution and activation module, the output of the fifth convolution and activation module is connected to the input of the sixth convolution and activation module, the output of the sixth convolution and activation module is connected to an input of the third fusion module, and the output of the third fusion module is connected to the input of the fourth upsampling module; the output of the fourth upsampling module is connected to the input of the seventh convolution and activation module, the output of the seventh convolution and activation module is connected to the input of the eighth convolution and activation module, the output of the eighth convolution and activation module is connected to an input of the fourth fusion module, and the output of the fourth fusion module forms the output of the decoder; Another input of the first fusion module is used as the output of the first feature transmission channel, another input of the second fusion module is used as the output of the first feature transmission channel, another input of the third fusion module is used as the output of the first feature transmission channel, and another input of the fourth fusion module is used as the output of the first feature transmission channel.

4. The consistent semi-supervised left atrium segmentation method based on the teacher-student model according to claim 3 is characterized in that: The upsampling module of the first decoder adopts bilinear interpolation, the upsampling module of the second decoder adopts transposed convolution, and the upsampling module of the third decoder adopts sub-pixel convolution.

5. The consistent semi-supervised left atrium segmentation method based on the teacher-student model according to claim 1, characterized in that: In step 3, when learning the consistency of the teacher-student model: For the student image segmentation network, the parameters are updated by minimizing the consistency loss back propagation, namely: For the teacher image segmentation network, an exponential moving average strategy is used to update its parameters, namely: i t ←a·i t +(1-a)·θ s Among them, θ s represents the parameters of the student image segmentation network, θ t represents the parameters of the teacher image segmentation network, η represents the learning rate, α represents the weight of the historical parameters, represents the gradient of the parameters of the student image segmentation network, L represents the consistency loss, Where i represents the current sample index, j represents the current output port index, represents the output probability distribution of the student image segmentation network at the j output ports for the i-th sample, represents the output probability distribution of the teacher image segmentation network at the j output ports for the i-th sample, ‖*‖ 2 Represents the square of the Euclidean distance.

Images