Semi-supervised image segmentation method based on knowledge distillation

Abstract

The invention discloses a semi-supervised image segmentation method based on knowledge distillation. The method comprises the following steps: obtaining a to-be-processed original image data set; preprocessing the original image data set to obtain annotated data and unannotated data; constructing a teacher-student network model, performing feature extraction and learning on a small amount of annotation data through a teacher network, generating initial annotation information, and taking the initial annotation information as training guidance of a student network; a feature distillation method is adopted, and students are guided to learn and correct target feature information through the feature information provided by the teacher network; based on a semi-supervised learning strategy, adopting unlabeled data to carry out adaptive training on the teacher-student network model to obtain a trained teacher-student network model; and inputting a to-be-segmented image into the trained teacher-student network model to realize image segmentation. The method is suitable for various image segmentation tasks, and provides reliable technical support for the fields of medical image analysis, automatic driving, video monitoring and the like.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a semi-supervised image segmentation method based on knowledge distillation. Background Art

[0002] Image segmentation plays a key role in the field of computer vision and is widely used in tasks such as autonomous driving, video surveillance, and object detection. In these application scenarios, accurate segmentation of the target area can not only improve the overall recognition effect of the algorithm, but also significantly improve the practicality and stability of the system. For example, the autonomous driving system needs to accurately segment elements such as pedestrians, vehicles, and roads to ensure driving safety; in video surveillance, accurate segmentation of specific target areas can achieve more effective event detection and behavior recognition; in brain tumor diagnosis, accurate segmentation of tumor areas in MRI or CT images can help doctors better understand the size, shape, and location of the tumor, thereby providing an important reference for formulating surgical plans or radiotherapy plans.

[0003] In recent years, with the development of deep learning technology, convolutional neural networks have achieved remarkable results in the field of image segmentation with their powerful feature extraction capabilities. However, convolutional neural networks have limitations in their ability to capture global contextual information, making it difficult to meet the needs of global information in certain application scenarios. To make up for this shortcoming, the attention mechanism has gradually been introduced into image segmentation tasks due to its advantages in capturing global information, complementing the local feature extraction capabilities of convolutional neural networks.

[0004] Although these methods have improved the segmentation performance to a certain extent, they usually rely on a large amount of accurately labeled data for training. Accurate labeling of data often requires a lot of time and labor costs, especially in the labeling of complex targets or scenes that are difficult to accurately label, it is difficult to obtain sufficient high-quality labeled data. Therefore, the problem of insufficient data has become a bottleneck restricting the further improvement of image segmentation performance. In addition, existing methods still have limitations in utilizing unlabeled data or coarsely labeled data, resulting in insufficient extraction of target area features, affecting the accuracy and stability of segmentation.

[0005] In response to the above problems, semi-supervised learning and knowledge distillation techniques have gradually attracted attention. Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data, and can effectively learn even when data resources are limited; knowledge distillation uses a high-performance teacher model to guide the training of the student model, thereby improving the learning effect and generalization ability of the student model. However, existing semi-supervised learning and knowledge distillation methods fail to fully integrate the advantages of different architectures in image segmentation tasks, and lack targeted feature learning strategies in segmentation tasks, resulting in segmentation accuracy and robustness failing to achieve the expected results. Therefore, innovative methods are needed to improve segmentation accuracy and robustness. Summary of the invention

[0006] In order to solve the above problems, the technical solution adopted by the present invention is: a semi-supervised image segmentation method based on knowledge distillation, comprising the following steps:

[0007] Obtain raw image data for image segmentation tasks;

[0008] Preprocess the original image dataset, check and delete duplicate images in the dataset, remove damaged or incorrectly read image files, annotate some images, and obtain a set of images with coarse-grained annotations, a set of images with fine-grained annotations, and a set of unannotated images;

[0009] Constructing a teacher-student network model using a semi-supervised learning strategy, wherein the teacher-student network model constructed using the semi-supervised learning strategy includes a teacher network and a student network;

[0010] The teacher network learns basic segmentation knowledge by training on coarse-grained labeled data, and this knowledge is passed to the student network through knowledge distillation;

[0011] The feature distillation method and the teacher network generate the annotation information of the unlabeled image to guide the student network to learn the features and segment the object of the unlabeled image, thus improving the overall segmentation accuracy.

[0012] Based on a set of fine-grained annotated images, the segmentation performance of the teacher-student network is verified by comparing the image annotation results generated by the teacher-student network with the real image annotations.

[0013] The image to be segmented is input into the trained teacher-student network model to achieve image segmentation.

[0014] Further: The preprocessed image dataset is divided into three parts: a coarse-grained annotated image set for training the teacher network, including image data X t and its corresponding coarse-grained annotation image SL t ;

[0015] Unlabeled image dataset X for training student network s ,

[0016] A collection of fine-grained annotated images for evaluating the teacher-student network model, including image data X test and the corresponding fine-grained annotation image FL test .

[0017] Further: the process of adaptively training the teacher-student network model based on the semi-supervised learning strategy to obtain the trained teacher-student network model is as follows:

[0018] Using image data X t and its corresponding coarse-grained annotation image SL t to train the teacher network;

[0019] Using the trained teacher network, for the image data X s Generate annotated image PL s , annotated image PL s With image data X s Together they form the image data for training the student network;

[0020] Use image data X respectively s And the annotation image PL s to jointly train the student network; in this way, the teacher-student network model can not only integrate the teacher network from the labeled image PL s The information learned in s Learn more features related to the target;

[0021] Generate image data X using teacher network and student network test The annotated image and its corresponding fine-grained annotated image FL test Compare and verify the segmentation performance of the teacher and student network models.

[0022] Furthermore: the teacher network uses a deep learning segmentation network, ResUNet, AttUNet, SwinUNet or UTransformer network to annotate the image.

[0023] Furthermore: the student network adopts a hybrid convolutional neural network-attention mechanism architecture, based on an encoder-decoder framework design;

[0024] The encoder consists of three layers of convolution, each of which contains a normalization layer and an activation function. The normalization layer is used to accelerate the training process and stabilize the network structure, while the activation function gives the network the ability to learn more complex feature representations. Through continuous three-layer convolution operations, the encoder gradually extracts local features of the image, while gradually reducing the spatial resolution of the feature map to capture higher-level semantic information.

[0025] After the local feature extraction is completed, the feature map generated by the encoder is input into the attention mechanism module for further processing. The attention mechanism module adopts a sliding window strategy. First, the feature map is divided into multiple regular non-overlapping windows, and self-attention calculation is performed in each window; this helps to capture the dependency relationship within the local area;

[0026] Subsequently, the window is shifted to the right by half the width to generate new overlapping windows, and self-attention calculations are performed again in these overlapping windows. This sliding window approach can effectively capture global information across windows and enhance the continuity of feature representation. In each self-attention calculation process, the two spatial dimensions of the feature map are merged into one dimension to form an input sequence for the attention mechanism to process. In addition, a learnable relative position offset is introduced to enhance the model's ability to model spatial relationships. The attention mechanism module can perform detailed analysis and processing of feature maps at different scales and positions, thereby improving the performance of the entire model in image understanding tasks.

[0027] The decoder also consists of three layers, each of which contains upsampling, normalization, and activation functions; the upsampling operation gradually restores the spatial resolution of the feature map so that the final output can reach the same size as the input image;

[0028] The normalization layer is used to speed up the training process and stabilize the network, ensuring the consistency of features between different layers. The activation function introduces nonlinearity, enabling the network to learn and express more complex feature representations.

[0029] Through the continuous operation of these three layers, the decoder not only restores the detailed information of the image, but also combines the features of the corresponding layers in the encoder to finally achieve high-resolution segmentation results. This design enables the model to capture local details and global semantics to achieve the final image segmentation goal.

[0030] Further: the attention mechanism introduces a sliding window operation. The attention mechanism consists of 12 layers. Each layer performs an attention calculation. The attention calculation is performed in a regular non-overlapping window, and then the window is shifted to the right by half the width to generate a new window for calculation. In each layer of the attention mechanism, the self-attention calculation of each window is performed through a specific equation, and the two dimensions H and W of the feature map are merged into the same dimension to obtain the input sequence of the attention mechanism. Where N = HW / 16 is the number of elements in the input sequence, and the self-attention calculation of each window is calculated using the following formula:

[0031] [Q; K; V] = [W q ; W k ; W v ]·x i (1)

[0032]

[0033] in, and is the parameter matrix, d is the feature matrix in each window These matrices are multiplied with the feature vector to produce three attention vectors: and Self-attention computation via a learnable relative position offset Introduce location information to enhance the expression of spatial relationships.

[0034] A semi-supervised image segmentation device based on knowledge distillation, comprising:

[0035] Acquisition module: used to obtain the original image data set to be processed;

[0036] Preprocessing module: used to preprocess the original image dataset, check and delete duplicate images in the dataset, remove damaged or incorrectly read image files, annotate some images, and obtain the preprocessed image dataset, a set of images with coarse-grained annotations, a set of images with fine-grained annotations, and a set of unannotated images;

[0037] Construction module: used to construct a teacher-student network model using a semi-supervised learning strategy, wherein the teacher-student network model using a semi-supervised learning strategy includes a teacher network and a student network;

[0038] Training module: used for the teacher network to learn basic segmentation knowledge by training on coarse-grained labeled data, and this knowledge is passed to the student network through knowledge distillation;

[0039] The feature distillation method and the teacher network generate the annotation information of the unlabeled image to guide the student network to learn the features and segment the object of the unlabeled image, thus improving the overall segmentation accuracy.

[0040] Based on a set of fine-grained annotated images, the segmentation performance of the teacher-student network is verified by comparing the image annotation results generated by the teacher-student network with the real image annotations.

[0041] Implementation module: used to input the image to be segmented into the trained teacher-student network model to achieve image segmentation

[0042] A computer device comprises: a processor and a memory, wherein the memory stores a program module, and wherein the program module runs on the processor to implement any of the methods described above.

[0043] The present invention provides a semi-supervised image segmentation method based on knowledge distillation, which adopts a hybrid architecture design of convolutional neural network and attention mechanism and a feature distillation strategy, which can not only effectively utilize unlabeled data when labeled data is insufficient, but also improve the segmentation accuracy of the model for the target area, and meet the requirements of image segmentation accuracy in different application scenarios. The beneficial effects of the present invention are:

[0044] The present invention can solve the limitations of traditional methods in image segmentation tasks. First, by combining the advantages of convolutional neural networks and attention mechanisms, a hybrid architecture design is adopted to retain fine-grained spatial details and capture global contextual information, thereby achieving a more accurate segmentation effect, which is suitable for complex image segmentation tasks. At the same time, with the help of a semi-supervised learning strategy, unlabeled data is effectively utilized, the dependence on large-scale labeled data is reduced, and the data labeling cost is significantly reduced. In addition, the present invention introduces knowledge distillation technology, and guides the training of the student model through the teacher model, so that the student model can learn more accurate feature information from the coarsely labeled data, thereby improving the generalization ability of the model. The sliding window operation of the attention mechanism further reduces the computational complexity of the self-attention mechanism, making the model suitable for practical applications while maintaining high performance. The method of the present invention is applicable to a variety of image segmentation tasks, and provides reliable technical support for fields such as medical image analysis, autonomous driving, and video surveillance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 Schematic diagram of teacher-student framework training.

[0047] Figure 2 This is a schematic diagram of the overall structure of the model.

[0048] Figure 3 Schematic diagram of the original CT images of lung nodules in the LUNA16 dataset, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C, (d) CT scan image of individual D, and (e) CT scan image of individual E.

[0049] Figure 4 Schematic diagram of the comparison before and after sliding the LUNA16 dataset window, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C.

[0050] Figure 5 Schematic diagram of cropping the LUNA16 dataset.

[0051] Figure 6Schematic diagram of the original CT images of lung nodules in the LC183 dataset, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C, (d) CT scan image of individual D, and (e) CT scan image of individual E.

[0052] Figure 7 Schematic diagram of the comparison before and after sliding the window of the LC183 dataset, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C, (d) CT scan image of individual D, and (e) CT scan image of individual E.

[0053] Figure 8 Schematic diagram of the lung nodule segmentation effect under different teacher networks in the LUNA16 dataset.

[0054] Fig. 9 Schematic diagram of lung nodule segmentation effect under different teacher networks of LC183 dataset.

[0055] Fig.10 Schematic diagram of rough lung nodule annotations generated for different teacher models.

[0056] Fig.11 Schematic diagram of the lung nodule segmentation effect under different student segmentations of the LUNA16 dataset.

[0057] Fig.12 Schematic diagram of the lung nodule segmentation effect under different student segmentations of the LC183 dataset. DETAILED DESCRIPTION

[0058] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] A semi-supervised image segmentation method based on knowledge distillation, characterized in that it comprises the following steps:

[0061] S1: Obtain the original image data set to be processed, collect and summarize the original image data for the image segmentation task;

[0062] S2: Preprocess the original image dataset, check and delete duplicate images in the dataset, remove damaged or incorrectly read image files, annotate some images, and obtain a set of images with coarse-grained annotations (accounting for 30% of all data), a set of images with fine-grained annotations (accounting for 20% of all data), and a set of unannotated images (accounting for 50% of all data);

[0063] S3: A teacher-student network model is constructed based on a semi-supervised learning strategy. This model significantly improves the accuracy and efficiency of image segmentation by effectively combining labeled data and unlabeled data.

[0064] S4: The teacher network learns basic segmentation knowledge by training on coarse-grained labeled data, and this knowledge is passed to the student network through knowledge distillation;

[0065] S5: Generate annotation information of unlabeled images through feature distillation method and teacher network to guide student network to perform feature learning and target segmentation on unlabeled images, thus improving the overall segmentation accuracy;

[0066] S6: Based on the fine-grained annotated image set, the segmentation performance of the teacher-student network is verified by comparing the image annotation results generated by the teacher-student network with the real image annotations;

[0067] S7: Input the image to be segmented into the trained teacher-student network model to achieve image segmentation.

[0068] Steps S1 / S2 / S3 / S4 / S5 / S6 / S7 are executed in sequence, as detailed below;

[0069] Furthermore, the teacher-student network model includes a teacher network and a student network. The model significantly improves the accuracy and efficiency of segmentation in image segmentation by effectively combining labeled data and unlabeled data.

[0070] Furthermore, the preprocessed image dataset is divided into three parts: a coarse-grained annotated image set for training the teacher network, including image data X t and its corresponding coarse-grained annotation image SL t ;

[0071] Unlabeled image dataset X for training student network s ,

[0072] A collection of fine-grained annotated images for evaluating the teacher-student network model, including image data X testand the corresponding fine-grained annotation image FL test .

[0073] Furthermore, based on semi-supervised learning and knowledge distillation strategy, the process of training and validating the teacher-student network model is as follows:

[0074] Using image data X t and its corresponding coarse-grained annotation image SL t To train the teacher network to learn image annotation, in this stage, a model with good performance in the field of image segmentation, simple structure and easy implementation is selected as the teacher network.

[0075] Using the trained teacher network, we can train the unlabeled image data X s Generate annotated image PL s , annotated image PL s With image data X s Together they constitute the data for training the student network; in this way, the student network can not only learn from the annotation information PL provided by the teacher network, but also s Learning features can also be done through X s It learns more features related to the segmentation target.

[0076] Generate image data X using the teacher-student network model test The annotation results and the corresponding fine-grained annotation image FL test Compare and verify the segmentation performance of the teacher-student network model.

[0077] Figure 1 A diagram for teacher-student framework training;

[0078] Figure 2 It is a schematic diagram of the overall structure of the model;

[0079] After the training process is set up, a deep learning-based segmentation network is first selected as the teacher network. Previous studies have proven the effectiveness of the UNet network, so UNet is selected as the basic teacher network. Then a network such as ResUNet, AttUNet, SwinUNet or UTransformer is selected to annotate the image.

[0080] Furthermore, the student network adopts a hybrid convolutional neural network-attention mechanism architecture; based on the encoder-decoder framework design, the framework aims to efficiently extract and reconstruct image features to achieve accurate image segmentation. It combines the high-resolution spatial information extraction capability of the convolutional neural network with the global context capture capability of the attention mechanism to improve segmentation performance and positioning accuracy. By introducing a hybrid design, the architecture effectively solves the problem of feature resolution loss that may be caused by the attention mechanism during the segmentation process, ensuring that the global features in the image are accurately captured while retaining fine-grained spatial details.

[0081] The encoder consists of three layers of convolution, each of which contains a normalization layer and an activation function. The normalization layer is used to accelerate the training process and stabilize the network structure, while the activation function gives the network the ability to learn more complex feature representations. Through continuous three-layer convolution operations, the encoder gradually extracts local features of the image while reducing the spatial resolution of the feature map layer by layer to capture higher-level semantic information.

[0082] After completing local feature extraction, the feature map generated by the encoder is input into the attention mechanism module for further processing.

[0083] The attention mechanism consists of two parts: channel attention and spatial attention. Channel attention includes global pooling, which performs global average pooling on the feature map to generate channel descriptors; nonlinear transformation, which performs nonlinear transformation on the descriptors through a multi-layer perceptron to generate attention weights for each channel; weighted processing, which applies the generated attention weights to the original feature map to enhance the feature expression ability of important channels and suppress irrelevant or redundant features. Spatial attention includes convolution operations, which apply two-dimensional convolution operations to the feature map after channel attention processing to generate a spatial attention map; regional highlighting, which uses the generated spatial attention map to highlight the key areas in the image and suppress background noise, thereby improving the segmentation accuracy of the target area. Through the combination of channel attention and spatial attention, the attention mechanism module can dynamically adjust the important information in the feature map, so that the student network pays more attention to key features and areas, and improves the overall segmentation performance.

[0084] Furthermore: the attention mechanism introduces a sliding window operation. The attention mechanism consists of 12 layers. Each layer performs an attention calculation. The attention calculation is performed in a regular non-overlapping window, and then the window is shifted to the right by half the width to generate a new window for calculation. The sliding window operation significantly reduces the computational burden of the model, making it more suitable for image segmentation tasks. In each layer of the attention mechanism, the self-attention calculation of each window is performed through a specific equation, and the two dimensions H and W of the feature map are merged into the same dimension to obtain the input sequence of the attention mechanism. Where N = HW / 16 is the number of elements in the input sequence, and the self-attention calculation of each window is calculated using the following formula:

[0085] [Q; K; V] = [W q ; W k ; W v ]·x i (1)

[0086]

[0087] in, and is the parameter matrix, d is the feature matrix in each window These matrices are multiplied with the feature vector to produce three attention vectors: and Self-attention computation via a learnable relative position offset Position information is introduced to enhance the expression of spatial relationships. After the above operations, the shape of the feature map is consistent with the output of the encoder.

[0088] The decoder also consists of three layers, each of which contains upsampling, normalization, and activation functions. The upsampling operation gradually restores the spatial resolution of the feature map so that the final output can reach the same size as the input image. The normalization layer is used to accelerate the training process and stabilize the network to ensure the consistency of features between different layers. The activation function introduces nonlinearity, enabling the network to learn and express more complex feature representations. Through the continuous operation of these three layers, the decoder not only restores the detailed information of the image, but also combines the features of the corresponding layers in the encoder to ultimately achieve high-resolution segmentation results. This design enables the model to capture local details and global semantics at the same time to achieve the final image segmentation goal.

[0089] A semi-supervised image segmentation device based on knowledge distillation, comprising:

[0090] Acquisition module: used to obtain the original image data set to be processed;

[0091] Preprocessing module: used to preprocess the original image dataset, check and delete duplicate images in the dataset, remove damaged or incorrectly read image files, annotate some images, and obtain the preprocessed image dataset, a set of images with coarse-grained annotations, a set of images with fine-grained annotations, and a set of unannotated images;

[0092] Construction module: used to construct a teacher-student network model using a semi-supervised learning strategy, wherein the teacher-student network model using a semi-supervised learning strategy includes a teacher network and a student network;

[0093] Training module: used for the teacher network to learn basic segmentation knowledge by training on coarse-grained labeled data, and this knowledge is passed to the student network through knowledge distillation;

[0094] The feature distillation method and the teacher network generate the annotation information of the unlabeled image to guide the student network to learn the features and segment the object of the unlabeled image, thus improving the overall segmentation accuracy.

[0095] Based on a set of fine-grained annotated images, the segmentation performance of the teacher-student network is verified by comparing the image annotation results generated by the teacher-student network with the real image annotations.

[0096] Implementation module: used to input the image to be segmented into the trained teacher-student network model to achieve image segmentation Example 1: The present invention designs a semi-supervised image segmentation method based on knowledge distillation. In order to more comprehensively measure the performance of the model, the performance of the teacher-student model is tested using an open source dataset and a constructed private dataset, and a series of preprocessing operations are performed on the two datasets to enhance their usability in model training. The following is a detailed description of the dataset and preprocessing operations:

[0097] (1)LUNA16: Figure 3 As shown, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C, (d) CT scan image of individual D, and (e) CT scan image of individual E.

[0098] The LUNA16 dataset is a subset of the largest public lung nodule dataset LIDC-IDRI, and its main purpose is to achieve automatic detection and segmentation of lung cancer. The LUNA16 dataset contains 888 lung CT scan images, each containing 1-4 nodules, for a total of 1186 nodules. The image size ranges from 512×512×95 to 512×512×733, and the voxel size is 0.78×0.78×1.25mm 2 The annotations of the dataset are generated using an automatic segmentation algorithm and include independent labels for the left and right lungs and trachea. The annotation information of LUNA16 includes the location of the nodules (x, y, and z axis coordinates), and the original image size is 512×512.

[0099] The patient lung cancer images in the LUNA16 dataset are computed tomography (CT) images. CT imaging scans a specific thickness cross section of the human body through precisely guided X-rays and reconstructs a three-dimensional image of the human body part based on the received X-ray signals, providing higher spatial resolution and clearer three-dimensional lesion images than traditional X-rays. The LUNA16 dataset contains multiple CT images containing lung nodules, such as Figure 3The information stored in the original CT file includes images of all organs in the scanned area (including interference areas such as air and blood vessels). In order to enhance the model's ability to segment lung nodules, the CT image is first windowed to highlight the lung area.

[0100] Figure 4 Schematic diagram of the comparison before and after sliding the LUNA16 dataset window, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C.

[0101] Figure 4 The comparison of several CT images in the LUNA16 dataset before and after the window sliding is shown. Subsequently, the CT images in the LUNA16 dataset are resampled using a linear interpolation algorithm to ensure that the spacing between the CT images of each patient is 1 mm between each scanning plane, thereby solving the problem of inconsistent spatial scale caused by different spacing between scanning planes.

[0102] The LUNA16 dataset includes the coordinate information of lung nodules in CT images. Previous studies have generated fine-grained annotations of lung nodules based on nodule coordinates. In order to explore the learning ability of the proposed model under coarse-grained annotations and its effect in identifying lung nodules, the nodule coordinate information is used to generate both fine-grained and coarse-grained annotations. Figure 5 As shown in the figure, for the original CT image of size 512×512, it is first cropped near the location of the lung nodule to generate an image of size 96×96. Subsequently, the lung nodules are annotated in the CT image with reference to the LUNA16 nodule coordinate information. The fine-grained annotation is an irregular shape that accurately covers the nodule area, while the coarse-grained annotation is a rectangular box that covers the nodule area and its surroundings. Based on these fine-grained and coarse-grained annotations, detailed annotation maps and coarse annotation maps are generated for model training and verification.

[0103] The preprocessed LUNA16 dataset is randomly divided into three non-overlapping subsets with a ratio of 3:5:2. The first set contains lung nodule image data X t and its corresponding coarse-grained annotation image SL t , used to train the teacher network. The second group contains lung nodule image data X s , and use the teacher network to generate X s Pulmonary nodules marked with PL s .X s and PL s Used together to train the student network. The third group X test and its corresponding fine-grained annotation image FL test Used to test model performance.

[0104] Figure 6Schematic diagram of the original CT images of lung nodules in the LC183 dataset, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C, (d) CT scan image of individual D, and (e) CT scan image of individual E.

[0105] (2) LC183: Figure 6 As shown in the figure, CT data of 183 lung cancer patients collected from the hospital. This dataset is called LC183, and the collection of data has obtained informed consent from all participants and complies with the ethical guidelines of the hospital ethics committee. Each patient's image contains at least 1 lung nodule and a maximum of 3 lung nodules. The image size ranges from 512×512×49 to 512×512×368, and the voxel size ranges from 0.50×0.50×0.50mm 2 Up to 1.25×1.25×5mm 2 The pulmonary nodule annotation file format of the LC183 dataset is nii.gz, which stores the mask image of the pulmonary nodule.

[0106] The present invention invited a number of clinicians with extensive experience in diagnosing lung nodules to annotate the patient CT images in the LC183 dataset, and the annotation process was the same as that of the above-mentioned LUNA16 dataset.

[0107] Figure 7 Schematic diagram of the comparison before and after sliding the window of the LC183 dataset, (a) CT scan image of individual A, (b) CT scan image of individual B, (c) CT scan image of individual C, (d) CT scan image of individual D, and (e) CT scan image of individual E.

[0108] Figure 7 An image of a lung nodule from the LC183 dataset is shown, and the tool used by clinicians to annotate it is Slicer5.2.1.

[0109] In combination with the scheme of the present invention, the experimental analysis is carried out as follows:

[0110] (1) Experimental details

[0111] The present invention uses Pytorch 1.10 to implement the network and trains the network for 150 epochs on NVIDIA GeForce RTX 3090. The network is optimized using the Adam optimizer with a learning rate of 10 -5 If the validation loss does not improve within 10 consecutive epochs, the training is terminated early to avoid overfitting of the network. During the test, accuracy, Dice coefficient, sensitivity, specificity and Hausdorff distance are used to evaluate the network performance.

[0112] (2) Evaluation indicators

[0113] After training, the segmentation model needs to be tested and evaluated. In order to evaluate the proposed method, the present invention uses the following indicators to evaluate the performance of the model: TP: true positive, FP: false positive, TN: true negative, FN: false negative.

[0114] Accuracy (Acc): determines the performance in correctly classifying the classes in the test SSSSSSSSSSSSSSS set, as shown in Formula 3.

[0115]

[0116] Dice coefficient (Dice): used to evaluate the overlap between the prediction result and the ground truth. The range of Dice is from 0 to 1. The better the prediction result, the larger the Dice value, as shown in Formula 4.

[0117]

[0118] Sensitivity (SENS): measures the proportion of actual positives that are correctly identified, as shown in Equation 5.

[0119]

[0120] Specificity (SPE): The ability to correctly exclude irrelevant samples, that is, the ability to accurately identify negative samples. The higher the specificity, the higher the probability of diagnosis, as shown in Formula 6.

[0121]

[0122] Hough distance (Hausdorff Distance, HF95): The distance between the boundary of the prediction result and the ground truth. The better the prediction result, the smaller the HF95 value, where d(x, y) represents the Euclidean distance between x and y, as shown in Formula 7.

[0123] HF95=maxd(y,y′),d(y′,y) (7)

[0124] (3) Experimental results analysis

[0125] In the experiment, we firstly conduct ablation experiments to investigate the effects of different parameter configurations on model performance. Then, we study the effects of replacing the teacher network on the performance of the student network in the pulmonary nodule segmentation task. Finally, we explore the performance differences between the proposed method and other competing models in the segmentation field, and analyze the reasons for these differences.

[0126] 1) Ablation experiment: In order to prove the effectiveness of the dataset and understand the importance of each component in the present invention, ablation experiments were conducted on the dataset, and the results are shown in Table 1 and Table 2. The ablation experiment results in Table 1 on the LUNA16 dataset and Table 2 on the LC183 dataset show that whether removing the attention mechanism layer in the model or not using the teacher-student training strategy during model training will lead to a decrease in model performance, verifying the effectiveness of the proposed training strategy.

[0127] Table 1 Ablation experiment results of LUNA16 dataset

[0128] method Acc Dice SENS SPE HF95 Removing the attention mechanism 0.9836 0.4554 0.7563 0.9850 18.94 Removal of teacher-student training 0.9780 0.4829 0.7887 0.9810 20.05 This method 0.9850 0.5253 0.7603 0.9931 8.50

[0129] Note ①: The results in bold indicate the best results among the relevant indicators.

[0130] Table 2 Ablation experiment results of LC183 dataset

[0131] method Acc Dice SENS SPE HF95 Removing the attention mechanism 0.9838 0.6245 0.7601 0.9852 78.90 Removal of teacher-student training 0.9785 0.6524 0.7703 0.9940 20.00 This method 0.9892 0.6837 0.8493 0.9813 8.03

[0132] Note ①: The results in bold indicate the best results among the relevant indicators.

[0133] 2) The impact of different teacher networks: In the real world, teachers usually need to have a broader knowledge base than students, or play a guiding role in the learning process. The UNet model has been proven to be highly effective and efficient in the field of image segmentation. Therefore, the UNet model is selected as the basic teacher network, and combined with its variants such as ResUNet, AttUNet, SwinUNet and UTransformer, the impact of different teacher network models on the segmentation performance of the student network is deeply studied and compared.

[0134] Table 3 shows the segmentation performance of the LUNA16 dataset under different teacher network conditions. As can be seen from Table 3, the performance is better when the teacher network is a UNet model, a ResUNet model, or a UTransformer. In particular, when the teacher network is a UNet, most indicators show high performance. However, when the teacher network is an AttUNet model or a SwinUNet, the performance is relatively poor in various indicators. Analysis shows that this result may be related to the increase in the number of parameters of the teacher model. Model parameters are generally related to representation and learning capabilities. More parameters mean that more detailed features and patterns can be learned, thereby better fitting the training data. In some cases, more parameters can enhance performance by adapting to complex data distributions, but when the amount of training data for the teacher network is limited, too many parameters may lead to insufficient model training. This will lead to insufficient data learning, making the generated annotations inaccurate, which will affect learning and ultimately lead to performance degradation.

[0135] Table 4 shows the performance of the proposed method under different teacher network conditions on the LC183 dataset. As can be seen from Table 4, the best performance is achieved when the teacher network is a UNet model, and the performance drops significantly when the teacher network is a SwinUNet or AttUNet model. The performance difference may be due to the differences in the architecture and complexity of the teacher models. The teacher model based on the attention mechanism architecture has relatively more parameters, which makes it difficult to obtain sufficient training effects on the experimental dataset. Therefore, the encoder as a teacher is not accurate enough in identifying lung nodules, resulting in misleading guidance in the subsequent training process.

[0136] Figure 8 The lung nodule segmentation effect of this method under different teacher networks on the LUNA16 dataset is demonstrated. Fig. 9 The results of lung nodule segmentation under different teacher networks on the LC183 dataset are shown. Fig.10 Rough lung nodule annotations generated by different teacher models are shown in.

[0137] Table 3 Performance of different teacher annotators on the LUNA16 dataset

[0138] teacher Acc Dice SENS SPE HF95 SwinUNet 0.9701 0.3153 0.4642 0.9851 26.71 AttUNet 0.9686 0.3446 0.6248 0.9783 20.11 UTransformer 0.9806 0.7520 0.7304 0.9905 9.98 ResUNet 0.9736 0.4612 0.8127 0.9833 20.13 UNet 0.9850 0.5253 0.7603 0.9931 8.50

[0139] Note ①: The results in bold indicate the best results among the relevant indicators.

[0140] Table 4 Performance of different teacher annotators on the LC183 dataset

[0141]

[0142]

[0143] Note ①: The results in bold indicate the best results among the relevant indicators.

[0144] 3) Performance of different student networks: In order to explore the competitiveness and advantages of this segmentation method, the performance of other models (such as UNet, ResUNet and ttUNet) in the lung nodule segmentation task was compared horizontally while keeping the teacher network consistent. Table 5 shows the segmentation performance of the LUNA16 dataset under different student network conditions of this method. Table 6 shows the segmentation performance of the LC183 dataset under different student network conditions of this method. It can be seen from Tables 5 and 6 that this method shows stronger learning and error correction capabilities than AttUNet, ResUNet, SwinUNet and UTransformer when trained using the lung nodule annotations generated by the teacher network. It can not only learn the lung nodule features provided by the teacher network, but also has the ability to correct the inaccurate information provided by the teacher network to a certain extent. Fig.11and Fig.12 The segmentation results of lung nodules by this method and other models under the same teacher network on the LUNA16 and LC183 datasets are shown respectively.

[0145] Table 5 Performance of different student networks on the LUNA16 dataset

[0146] student Acc Dice SENS SPE HF95 UTransformer 0.9665 0.3920 0.5445 0.9750 27.45 AttUNet 0.9705 0.3783 0.6122 0.9868 12.40 ResUNet 0.9932 0.4573 0.7389 0.9850 16.20 SwinUNet 0.9806 0.7623 0.7401 0.9855 11.78 UNet 0.9850 0.5253 0.7603 0.9931 8.50

[0147] Note ①: The results in bold indicate the best results among the relevant indicators.

[0148] Table 6 Performance of different student networks on the LC183 dataset

[0149]

[0150]

[0151] Note ①: The results in bold indicate the best results among the relevant indicators.

[0152] In the present invention, a new image segmentation method is designed, which is based on a hybrid architecture of a convolutional neural network and an attention mechanism, aiming to achieve accurate segmentation of the target area to meet the needs of different application scenarios. In order to make full use of a large amount of data that lacks annotations, a training process based on a teacher-student network is designed, including training a teacher network to generate initial annotated data, and using these annotated data to train a student network to improve segmentation accuracy. The experiment verified the superior performance of this method in the image segmentation task. In addition, the experimental results also prove the effectiveness of the training process proposed in the present invention. In future research, it is planned to further explore how to effectively incorporate a large amount of unlabeled data into the training process of the segmentation network, and to design a better segmentation network to improve the segmentation effect in different task scenarios.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semi-supervised image segmentation method based on knowledge distillation, characterized in that: The following steps are involved: Obtaining raw image data for image segmentation tasks; Preprocess the original image dataset, check and delete duplicate images in the dataset, remove damaged or incorrectly read image files, annotate some images, and obtain the preprocessed image dataset, a set of images with coarse-grained annotations, a set of images with fine-grained annotations, and a set of unannotated images; Constructing a teacher-student network model using a semi-supervised learning strategy, wherein the teacher-student network model constructed using the semi-supervised learning strategy includes a teacher network and a student network; The teacher network learns basic segmentation knowledge by training on coarse-grained labeled data, and this knowledge is passed to the student network through knowledge distillation; The feature distillation method and the teacher network generate the annotation information of the unlabeled image to guide the student network to learn the features and segment the object of the unlabeled image, thus improving the overall segmentation accuracy. Based on a set of fine-grained annotated images, the segmentation performance of the teacher-student network is verified by comparing the image annotation results generated by the teacher-student network with the real image annotations. The image to be segmented is input into the trained teacher-student network model to achieve image segmentation.

2. The semi-supervised image segmentation method based on knowledge distillation according to claim 1, characterized in that: The preprocessed image dataset is divided into three parts: a coarse-grained annotated image set for training the teacher network, including image data X t and its corresponding coarse-grained annotation image SL t ; Unlabeled image dataset X for training student network s , A collection of fine-grained annotated images for evaluating the teacher-student network model, including image data X test and the corresponding fine-grained annotation image FL test .

3. The semi-supervised image segmentation method based on knowledge distillation according to claim 1, characterized in that: The process of adaptively training the teacher-student network model based on the semi-supervised learning strategy to obtain the trained teacher-student network model is as follows: Using image data X t and its corresponding coarse-grained annotation image SL t to train the teacher network; Using the trained teacher network, for the image data X s Generate annotated image PL s , annotated image PL s With image data X s Together they form the image data for training the student network; Use image data X respectively s And the annotation image PL s to jointly train the student network; in this way, the teacher-student network model can not only integrate the teacher network from the labeled image PL s The information learned in s Learn more features related to the target; Generate image data X using teacher network and student network test The annotated image and its corresponding fine-grained annotated image FL test Compare and verify the segmentation performance of the teacher and student network models.

4. The semi-supervised image segmentation method based on knowledge distillation according to claim 1, characterized in that: The teacher network uses a deep learning segmentation network, ResUNet, AttUNet, SwinUNet or UTransformer network to annotate the image.

5. The semi-supervised image segmentation method based on knowledge distillation according to claim 1, characterized in that: The student network adopts a hybrid convolutional neural network-attention mechanism architecture and is designed based on an encoder-decoder framework; The encoder consists of three layers of convolution, each of which contains a normalization layer and an activation function. The normalization layer is used to accelerate the training process and stabilize the network structure, while the activation function gives the network the ability to learn more complex feature representations. Through continuous three-layer convolution operations, the encoder gradually extracts local features of the image, while gradually reducing the spatial resolution of the feature map to capture higher-level semantic information. After the local feature extraction is completed, the feature map generated by the encoder is input into the attention mechanism module for further processing. The attention mechanism module adopts a sliding window strategy. First, the feature map is divided into multiple regular non-overlapping windows, and self-attention calculation is performed in each window; this helps to capture the dependency relationship within the local area; Subsequently, the window is shifted to the right by half the width to generate new overlapping windows, and self-attention calculations are performed again in these overlapping windows. This sliding window approach can effectively capture global information across windows and enhance the continuity of feature representation. In each self-attention calculation process, the two spatial dimensions of the feature map are merged into one dimension to form an input sequence for the attention mechanism to process. In addition, a learnable relative position offset is introduced to enhance the model's ability to model spatial relationships. The attention mechanism module can perform detailed analysis and processing of feature maps at different scales and positions, thereby improving the performance of the entire model in image understanding tasks; The decoder also consists of three layers, each of which contains upsampling, normalization, and activation functions; The upsampling operation gradually restores the spatial resolution of the feature map so that the final output can reach the same size as the input image; The normalization layer is used to speed up the training process and stabilize the network, ensuring the consistency of features between different layers. The activation function introduces nonlinearity, enabling the network to learn and express more complex feature representations. Through the continuous operation of these three layers, the decoder not only restores the detailed information of the image, but also combines the features of the corresponding layers in the encoder to finally achieve high-resolution segmentation results. This design enables the model to capture local details and global semantics to achieve the final image segmentation goal.

6. The semi-supervised image segmentation method based on knowledge distillation according to claim 5, characterized in that: The attention mechanism introduces a sliding window operation. The attention mechanism consists of 12 layers. Each layer performs an attention calculation. The attention calculation is performed in a regular non-overlapping window, and then the window is shifted to the right by half the width to generate a new window for calculation. In each layer of the attention mechanism, the self-attention calculation of each window is performed through a specific equation, and the two dimensions H and W of the feature map are merged into the same dimension to obtain the input sequence of the attention mechanism. Where N = HW / 16 is the number of elements in the input sequence, and the self-attention calculation of each window is based on the following formula: [Q;K;V]=[W q ;W k ;W v ]·x i (1) in, and is the parameter matrix, d is the feature matrix in each window These matrices are multiplied with the feature vector to produce three attention vectors: and Self-attention computation via a learnable relative position offset Introduce location information to enhance the expression of spatial relationships.

7. A semi-supervised image segmentation device based on knowledge distillation, characterized in that: include: Acquisition module: used to obtain the original image data set to be processed; Preprocessing module: used to preprocess the original image dataset, check and delete duplicate images in the dataset, remove damaged or incorrectly read image files, annotate some images, and obtain the preprocessed image dataset, a set of images with coarse-grained annotations, a set of images with fine-grained annotations, and a set of unannotated images; Construction module: used to construct a teacher-student network model using a semi-supervised learning strategy, wherein the teacher-student network model using a semi-supervised learning strategy includes a teacher network and a student network; Training module: used for the teacher network to learn basic segmentation knowledge by training on coarse-grained labeled data, and this knowledge is passed to the student network through knowledge distillation; The feature distillation method and the teacher network generate the annotation information of the unlabeled image to guide the student network to learn the features and segment the object of the unlabeled image, thus improving the overall segmentation accuracy. Based on a set of fine-grained annotated images, the segmentation performance of the teacher-student network is verified by comparing the image annotation results generated by the teacher-student network with the real image annotations. Implementation module: used to input the image to be segmented into the trained teacher-student network model to achieve image segmentation.

8. A computer device comprising: A processor and a memory, wherein the memory stores a program module, wherein the program module runs on the processor to implement the method according to any one of claims 1 to 7.

Images