Semi-supervised medical image segmentation method based on sequence convolution relation
Through the teacher-student training framework and data augmentation technology, combined with the encoder-decoder network structure, the dependence problem on labeled data in medical image segmentation is solved, segmentation accuracy and model generalization capabilities are improved, and efficient medical image segmentation is achieved.
Patent Information
- Application Number
- CN202510467778.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-15
- Publication Date
- 2025-07-29
AI Technical Summary
The existing medical image segmentation methods rely on a large number of manual annotations, which are costly and subjectively biased. The semi-supervised learning method fails to effectively reduce the annotation dependence and improve the generalization ability of the model.
A semi-supervised medical image segmentation method based on sequence convolution relationships is adopted, and a semi-supervised medical image segmentation method is used to guide students' model learning through teacher-student training framework and data augmentation technology. A trained teacher model is used to guide students' model learning, and a mixed image is generated for model optimization in combination with the encoder-decoder network structure and data copy-paste operation.
It improves the accuracy and model generalization ability of medical image segmentation, reduces dependence on labeled data, reduces costs, improves diagnostic efficiency and accuracy, and reduces misdiagnosis and missed diagnosis.
Smart Images
Figure CN120388174A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical image segmentation, and specifically relates to a semi-supervised medical image segmentation method based on sequential convolution relationships. Background Art
[0002] Medical image segmentation refers to the process of accurately extracting and annotating specific structures or lesion areas in medical images, and the segmentation results are usually used as important bases for doctors' clinical diagnosis and treatment decisions. In recent years, with the rapid development of deep learning technology, significant progress has been made in the field of medical image analysis, and deep learning methods have become the mainstream technology in medical image segmentation. In particular, various medical image segmentation methods based on supervised learning have been widely applied in practice.
[0003] However, the annotation process of medical images usually requires manual operation by experienced radiologists, which not only involves high time and economic costs, but also may introduce certain biases due to subjective factors of doctors. In view of these challenges, semi-supervised learning methods have gradually become a research hotspot because they can effectively reduce the dependence on a large amount of labeled data, thereby reducing the cost and time burden of manual annotation. Summary of the Invention
[0004] In order to overcome the problems in the prior art, the present invention proposes a semi-supervised medical image segmentation method based on sequential convolution relationships.
[0005] The technical solution of the present invention to solve the above technical problems is as follows:
[0006] The present invention provides a semi-supervised medical image segmentation method based on sequential convolution relationships, including the following steps:
[0007] Step 100: Perform data preprocessing on the original medical images to obtain medical images with the same size, where the original medical images include unlabeled medical images and labeled medical images;
[0008] Step 200: Perform a copy-paste operation on the preprocessed unlabeled medical images and the preprocessed labeled medical images to generate mixed images;
[0009] Step 300: Construct a teacher-student training framework, input the mixed images into the student network model to generate segmentation masks; input the preprocessed unlabeled medical images into the teacher network model to generate pseudo-labels, and perform a copy-paste operation again with the preprocessed labeled medical images to generate new mixed images; calculate the loss between the new mixed images and the segmentation masks, and optimize the model parameters to determine the final image segmentation model.
[0010] Further, in the step 200, a copy-paste operation is performed on the preprocessed unlabeled medical image and the preprocessed labeled medical image to generate a mixed image, which specifically includes: generating a mixed image by mixing the foreground and background of the preprocessed unlabeled medical image and the preprocessed labeled medical image.
[0011] Further, a new mixed image is generated by mixing the foreground and background of the preprocessed unlabeled medical image and the labeled medical image, which specifically includes:
[0012] Cropping a region of size M×M from the preprocessed labeled medical image as the background, and cropping a region of size M×M from the preprocessed unlabeled medical image as the foreground, and mixing the foreground and the background to form a first mixed image Similarly, cropping a region of size M×M from the preprocessed unlabeled medical image as the background, and cropping a region of size M×M from the preprocessed labeled medical image as the foreground, and mixing the foreground and the background to form a second mixed image
[0013] Further, the calculation formulas for the first mixed image and the second mixed image are as follows:
[0014]
[0015] where ⊙ represents the dot product between pixels.
[0016] Further, the teacher-student training framework is based on an encoder-decoder image network structure; the encoder is used to gradually downsample the input image and extract features; the decoder is used to gradually restore the compressed feature map extracted by the encoder to the spatial resolution of the original image for final pixel-level segmentation.
[0017] Further, in the optimization of the model parameters, the student network model uses the stochastic gradient descent method to optimize the parameters, and the teacher network model updates the parameters based on the parameters of the student network model and the EMA method; the output of the teacher network model is used as a guiding signal, and the student network model learns knowledge according to the prediction of the teacher network model.
[0018] Compared with the prior art, the present invention has the following technical effects:
[0019] (1) The data augmentation technique in the present invention, namely the copy-paste technique, effectively increases the diversity of the training set by synthesizing lesion regions or organ samples, while maintaining the authenticity of the images, which helps to improve the adaptability of the model to complex medical images.
[0020] (2) The training learning framework in the present invention, namely the teacher-student framework, uses the trained teacher model to guide the learning of the incompletely trained student model, realizing knowledge distillation, reducing the dependence on a large amount of labeled data, and improving the generalization ability of the model.
[0021] (3) The network structure design in the present invention: Encoder part: The depthwise separable convolutional attention module is introduced, which reduces the computational amount while enhancing the feature extraction ability. Through the attention mechanism, it focuses on the key regions and improves the segmentation accuracy. Decoder part: The high-order gating mechanism is added to dynamically adjust the feature fusion process, further improving the accuracy of the segmentation boundary.
[0022] (4) The present invention can accurately segment the lesion regions or organs, providing more accurate diagnostic information for doctors, improving the diagnostic efficiency and accuracy. In surgeries such as tumor treatment and organ transplantation, accurate image segmentation helps doctors formulate more reasonable treatment plans, reduce surgical risks, and improve the treatment effect. Over-segmenting the diseased tissue can provide a deeper understanding of the pathological process of the disease, providing strong support for scientific research. Accurate medical image segmentation helps to reduce misdiagnosis and missed diagnosis, lower medical costs, and improve the utilization efficiency of medical resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0024] Figure 1 is the flowchart of the present invention;
[0025] Figure 2 is the teacher-student training framework of the present invention;
[0026] Figure 3 is the image network structure diagram of the encoder-decoder of the present invention;
[0027] Figure 4 is the schematic diagram of the preprocessing process. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0028] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following specifically describes the implementation manner, structure, features, and effects of the technical solution proposed according to the present invention in detail in conjunction with the accompanying drawings and preferred embodiments. Specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.
[0029] The present invention aims to solve the problem of empirical mismatch between unlabeled data and the distribution of labeled data, and at the same time improve the accuracy of medical image segmentation.
[0030] In one embodiment of the present invention, there is provided a reference Figures 1 - 3 , a semi-supervised medical image segmentation method based on sequence convolution relationship, comprising the following steps:
[0031] Step 100: First perform data preprocessing on the original medical images to obtain medical images of the same size, wherein the original medical images include unlabeled medical images and labeled medical images;
[0032] Step 200: Perform a copy-paste operation on the preprocessed unlabeled medical images and the preprocessed labeled medical images to generate a mixed image;
[0033] Step 300: Based on the teacher-student training framework, input the mixed image into the student network model to generate a segmentation mask; input the preprocessed unlabeled medical images into the teacher network model to generate pseudo-labels, and perform a copy-paste operation again with the preprocessed labeled medical images to generate a new mixed image; calculate the loss between the new mixed image and the segmentation mask, and optimize the model parameters to determine the final image segmentation model.
[0034] The following details each of the above steps:
[0035] Step 100: First perform data preprocessing on the original medical images to obtain medical images of the same size, wherein the original medical images include unlabeled medical images and labeled medical images.
[0036] First perform data preprocessing on the original medical images. The preprocessing includes rotation and cropping operations, and each original medical image is cropped into the same size for subsequent processing.
[0037] The specific preprocessing includes: according to the characteristics of the original medical image and the actual application requirements, selecting one or more rotation angles; using the rotation function provided by the image processing library to rotate the original medical image, and the rotation operation usually involves the selection of an interpolation algorithm to minimize the loss of image quality; after rotation, the size of the image needs to be adjusted to maintain consistency, which can be achieved by cropping, padding, or adjusting the resolution; ensuring that the rotated image remains visually clear and accurate without introducing any artifacts or deformations; manually or automatically defining the cropping area according to the position and size of the region of interest in the medical image, which can be achieved by marking tools, algorithm detection, or predefined rules; using the cropping function provided by the image processing library to extract the defined cropping area from the original medical image; during cropping, the original ratio of the image should be maintained as much as possible to avoid introducing additional deformations, and if the size needs to be adjusted, appropriate scaling methods should be used. Ensure that the cropped image contains all important anatomical structures or lesion areas and does not lose any key information.
[0038] The Pancreas dataset plays an important role in the field of medical image processing, especially in the computer-aided detection and diagnosis of pancreatic diseases. The ACDC (Automatic Cardiac Diagnosis Challenge) dataset is part of the MICCAI 2017 challenge, aiming to segment the left ventricle (LV), right ventricle (RV), and myocardium (Myo) in the end-diastolic (ED) and end-systolic (ES) frames of cardiac cine-MRI. In the field of medical images, the LA (Left Atrium) dataset refers to a dataset related to the heart or other medical images.
[0039] In this embodiment, the size of the Pancreas dataset is cropped to 96×96×96, the size of the ACDC dataset is cropped to 256×256, and the size of the LA dataset is cropped to 112×112×80. The dataset is divided into training set, validation set, and test set in a ratio of 7:1:2. The preprocessing process is Figure 4 .
[0040] Step 200: Perform a copy-paste operation on the preprocessed unlabeled medical images and the preprocessed labeled medical images to generate mixed images.
[0041] Take two preprocessed unlabeled medical images and two preprocessed labeled medical images Use the copy-paste technique to enhance the data by combining the preprocessed unlabeled medical images and the preprocessed labeled medical images to construct richer semantic information, specifically including: from the preprocessed labeled medical images Crop a region of size M×M from it as the foreground from the pre - processed unlabeled medical image Crop a region of size M×M from it as the foreground, and mix the foreground with the background to form the first mixed image Similarly, from the pre - processed unlabeled medical image Crop a region of size M×M from it as the background, and crop a region of size M×M from the pre - processed labeled medical image as the foreground, and mix the foreground with the background to form the second mixed image The two mixed images contain common information of unlabeled and labeled data. It should be noted that in the same group of images, the unlabeled medical image and the labeled medical image are cropped at the same position as the foreground, and the cropped part is mixed with the background of the uncropped unlabeled image and labeled image
[0042] The mixed image is calculated by the following formula:
[0043]
[0044] where ⊙ represents the dot - product between pixels; the size of M is taken as 2 / 3, namely 64×64×64, 170×170, 74×74×53
[0045] For the selection of the size of the foreground region, four values of 1 / 3, 1 / 2, 2 / 3, and 5 / 6 are selected for comparison. When the value is too small, the ability of copying - pasting the foreground region to transfer semantic information to the background information is limited. When the value is too large, non - critical information is added, reducing the segmentation accuracy. Finally, 2 / 3 is selected
[0046] The unlabeled medical image can learn comprehensive semantic information from the labeled medical image in both positive and negative directions, solving the problem of empirical mismatch between the two image data distributions
[0047] Step 300: Based on the teacher - student training framework, input the mixed image into the student network model to generate a segmentation mask; input the pre - processed unlabeled medical image into the teacher network model to generate a pseudo - label, and perform a copy - paste operation again with the pre - processed labeled medical image to generate a new mixed image; calculate the loss between the new mixed image and the segmentation mask, and optimize the model parameters to determine the final image segmentation model
[0048] As an example, this step may include the following sub - steps
[0049] Step 310: Construct a teacher - student training framework, which includes a student network model and a teacher network model
[0050] The student network model and the teacher network model are encoder-decoder based image network structures.
[0051] In the encoder part, the combination of traditional convolution and depthwise separable convolution attention module ensures that the model can capture both fine-grained local information and broader global patterns; in the decoder part, a high-order gating device is introduced, and convolutional layer combinations are used to learn spatial interactions and fuse features across different dimensions, and the attention at different scales is adaptively adjusted through mean and max pooling operations.
[0052] The pre-trained teacher model is used to predict and generate pseudo-labels for the pre-processed unlabeled medical images. The pseudo-labels and the pre-processed labeled medical images are replicated and pasted again to generate new mixed images, and the loss is calculated with the segmentation mask generated by the student network model, while the parameters are updated; the iteration is updated until the training is completed to generate the final image segmentation model. Using the pre-trained teacher model to guide the under-trained student model to learn realizes knowledge distillation, reduces the dependence on a large amount of labeled data, and improves the generalization ability of the model.
[0053] Specifically, the encoder is used to gradually downsample the input image and extract features, ensuring that the model can capture both fine-grained local information and broader global patterns, while also improving the computational efficiency. The encoder gradually reduces the spatial resolution of the image to reduce the size of the feature map, while increasing the number of channels of the feature map, which helps to capture broader context information. As the number of layers increases, the encoder gradually compresses the spatial information of the image into a smaller feature map, which contains the abstract features of the image and provides information for the decoder.
[0054] The encoder adopts a multi-level feature extraction architecture, including an encoder convolutional block, a depthwise separable convolution attention module LSKA, and a Down Conv block connected in sequence.
[0055] The encoder convolutional block includes a convolutional layer and a sequence convolutional module. The sequence convolutional module includes Clone, Split (segmentation), Slicing (slicing), and Partial Conv (partial convolution), and the input data is input into the encoder to obtain the feature map of each training sample.
[0056] Depthwise Separable Convolution Attention Module LSKA. It includes a first standard convolutional layer Conv, GELU, Local-Spatial Attention Unit LSK, a second standard convolutional layer Conv, and Shortcut connected in sequence. The first standard convolutional layer performs feature dimension upsampling. GELU is used as the activation function. LSK extracts local and spatial features through two branches respectively. The first branch includes a third standard convolutional layer Conv and a third convolutional block Conv3 connected in sequence. The second branch includes Conv-spatial and a fourth convolutional block Conv4 connected in sequence. The outputs of the third convolutional block Conv3 and the fourth convolutional block Conv4 are concatenated in the channel dimension. The average value and the maximum value of the concatenated feature map in the channel dimension are calculated, and the two are concatenated, processed through Squeeze, and then passed through the Sigmoid activation function to obtain the attention weight sig. sig has two parts, which are used to weight attn1 and attn2 respectively. Finally, the weighted attention feature is calculated through attn = attn1 * attn1_sig + attn2 * attn2_sig. It is mapped back to the original channel dimension through the second standard convolutional layer Conv1x1 convolution, and the output is the result of weighting the original input X: X * attn.
[0057] Each standard convolutional layer includes a Layer Norm layer, a convolution, and a non-linear activation layer. The Layer Norm layer includes Lynorm1 and lynorm2, which are applied to the convolutional output features and the processed input features respectively to ensure that the activation values of each layer are within a reasonable range, thereby improving the stability of training. For the convolution and non-linear activation layer, three convolutional layers and the corresponding non-linear activation layer (ReLu) are applied to the input data to gradually expand the feature dimension and form a multi-level representation.
[0058] Down Conv realizes feature downsampling through convolution operations. Specifically, it includes Drop Path and skip connection operations. Drop Path randomly discards certain paths to prevent the model from overfitting and enhance the generalization ability of the model. Skip connections are used to retain the high-resolution features in the encoding stage and perform tensor concatenation with the corresponding features in the decoder. In the last layer, the decoder usually applies a convolutional layer to reduce the number of channels to the required number of classes and generate the final segmentation result.
[0059] The decoder gradually restores the compressed feature maps extracted by the encoder to the spatial resolution of the original image, enabling pixel-level segmentation. The decoder consists of a sequentially connected upsampling layer, an attention gate, and a decoder convolutional block. Upsampling increases the size of the feature map, using bilinear interpolation to achieve resolution improvement. This is to match the size of the feature map in the encoder for subsequent feature fusion or decoding operations.
[0060] The gated attention module Attention includes the fourth standard convolutional layer Conv, the GELU activation function, the gated unit Gated Unit and the fifth standard convolutional layer Conv. The fourth standard convolutional layer and the fifth standard convolutional layer form a residual connection.
[0061] The output of the fourth standard convolutional layer (Conv) is used for feature extraction. The gated unit (Gated Unit) consists of two branches: one for generating a gating signal, which is activated by a sigmoid function, and the other for generating the features to be gated. The gated signal is multiplied by the features to control the flow of features. The fifth standard convolutional layer (Conv) receives the output of the Gated Unit for further feature extraction and transformation. The output of the fourth convolutional layer is weighted with the output of the fifth convolutional layer to form the final attention feature. The decoder convolution block is used to further extract and transform the output of the attention module to generate the final decoding result.
[0062] The decoder's convolutional block structure consists of deconvolution (transposed convolution) layers and skip connections. Deconvolution (transposed convolution) layers upsample the encoder's output feature maps, restoring them to a higher spatial resolution. The decoder restores the image's spatial dimensions layer by layer while still learning high-level features. At each layer, the decoder not only uses upsampling to increase resolution but also fuses encoder and decoder features to recover more detailed information.
[0063] Step 320: Input the mixed image into the student network model to generate a segmentation mask (Q in , Q out );
[0064] Q in =F s (X in ;Θ s );
[0065] Q out =F s (X out ;Θ s );
[0066] In the above formula, F s represents the student network training function; X in represents the mixed image with unlabeled data as the foreground input into the student network; Θ s represents the student network parameters; X out represents the mixed image with labeled data as the foreground input into the student network.
[0067] Step 330: Input the preprocessed unlabeled medical image into the teacher network model to generate pseudo-labels and the preprocessed labeled medical image Perform the copy-paste operation again to generate a new mixed image (Y in , Y out ) as the supervision signal, so that the strong supervision from the true annotation helps the weak supervision from the pseudo-labels.
[0068] Among them, Y in and Y out , Q in and Q out The obtained calculation formulas are as follows:
[0069]
[0070] Step 340: Calculate the loss between the new mixed image and the segmentation mask, and stop iterating until the iteration number threshold is met to obtain the final parameter model, thereby determining the final image segmentation model.
[0071] The loss calculation formula is as follows:
[0072] I in = L seg (Q in , Y in )⊙M + αL seg (Q in , Y in )⊙(1 - M);
[0073] L out = L seg (Q out , Y out )⊙(1 - M) + αL seg (Q out , Y out )⊙M;
[0074] In the above formula, L in represents the loss function with unlabeled data as the foreground; L seg represents the loss function between the segmentation mask generated after training the student network and the pseudo-labels; α represents the influence of the unlabeled image on the loss function; L outThe loss function representing the label data as the foreground.
[0075] During the training process, the student network model optimizes the parameter θ using the stochastic gradient descent method s , and the teacher network model updates its parameters based on the parameters of the student network model and the EMA (Exponential Moving Average) method; the output of the teacher network model serves as a guiding signal, and the student network model learns knowledge according to the predictions of the teacher network model. Among them, the parameters of the teacher network model are updated by the EMA method to enhance its stability and help the student model learn better. The EMA method updates the parameters of the teacher network model smoothly by combining the current parameters and historical parameters of the student network model.
[0076] Obtain the model parameters according to the loss function of the training sample set, including: adjust the model parameters according to the loss function of the training sample set to obtain the training model parameters, thereby determining the initial network segmentation model, input the validation sample set into the initial network segmentation model, and adjust the training model parameters according to the loss function of the training sample set to determine the model parameters; iterative steps until the iteration number threshold is met and stop the iteration to obtain the final model parameters, including: after stopping the iteration, use multiple candidate operations that meet the selection probability threshold as the final operations, and update the final operation parameters into the teacher network model and the student network model.
[0077] Experiments were conducted on three datasets (ACDC, LA, Pancreas), and the segmentation accuracy was improved in all cases. In the case of a labeling ratio of 10%, the Dice index of the ACDC dataset was 89.74%, which was 0.9% higher than the current advanced method, the Jaccard index was 81.95%, an increase of 1.33%, and the 95HD(↓) index decreased by 1.22%. In the LA dataset with a labeling ratio of 10%, the Dice index was 90.01%, the Jaccard index was 81.90%, the 95HD index was 6.23, and the ASD index was 1.68, all of which were improved compared to the indicators of the current advanced method. All indicators in the Pancreas dataset with a labeling ratio of 20% were also improved.
[0078] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A semi-supervised medical image segmentation method based on sequence convolution relationship, characterized in that It includes the following steps: Step 100: First, perform data preprocessing on the original medical images to obtain medical images with the same size. Among them, the original medical images include unlabeled medical images and labeled medical images; Step 200: Perform a copy-paste operation on the preprocessed unlabeled medical images and the preprocessed labeled medical images to generate mixed images; Step 300: Construct a teacher-student training framework. Input the mixed images into the student network model to generate segmentation masks; input the preprocessed unlabeled medical images into the teacher network model to generate pseudo-labels, and perform a copy-paste operation again with the preprocessed labeled medical images to generate new mixed images; calculate the loss between the new mixed images and the segmentation masks, and optimize the model parameters to determine the final image segmentation model.
2. The semi-supervised medical image segmentation method based on sequence convolution relationship according to claim 1, wherein In the step 200, performing a copy-paste operation on the preprocessed unlabeled medical images and the preprocessed labeled medical images to generate mixed images specifically includes: generating mixed images by mixing the foreground and background of the preprocessed unlabeled medical images and the preprocessed labeled medical images.
3. A semi-supervised medical image segmentation method based on sequence convolution relationship according to claim 2, wherein Generating new mixed images by mixing the foreground and background of the preprocessed unlabeled medical images and the labeled medical images specifically includes: Crop a region of size M*M from the preprocessed labeled medical image as the background, and crop a region of size M*M from the preprocessed unlabeled medical image as the foreground. Mix the foreground with the background to form the first mixed image Similarly, crop a region of size M*M from the preprocessed unlabeled medical image as the background, and crop a region of size M*M from the preprocessed labeled medical image as the foreground. Mix the foreground with the background to form the second mixed image The calculation formulas for the first mixed image and the second mixed image are as follows: Among them, ⊙ represents the dot product between pixels.
4. A semi-supervised medical image segmentation method based on sequence convolution relationship according to claim 1, characterized in that The teacher-student training framework is based on the encoder-decoder image network structure; the encoder is used to gradually downsample the input image and extract features; The decoder is used to gradually restore the compressed feature maps extracted by the encoder to the spatial resolution of the original image for final pixel-level segmentation.
5. A semi-supervised medical image segmentation method based on sequence convolution relationship according to claim 4, characterized in that, The encoder integrates traditional convolution and depthwise separable convolution attention modules to capture local and global features.
6. A semi-supervised medical image segmentation method based on sequence convolution relationship according to claim 4, characterized in that In the decoder, the spatial resolution of the image is restored through deconvolution layers and skip connections.
7. A semi-supervised medical image segmentation method based on sequence convolution relationship according to claim 1, characterized in that, In the model parameter optimization, the student network model uses the stochastic gradient descent method to optimize the parameters, and the teacher network model updates the parameters based on the parameters of the student network model and the EMA method; the output of the teacher network model is used as a guiding signal, and the student network model learns knowledge according to the prediction of the teacher network model.