Semi-supervised medical image segmentation method and system based on visual language model

Abstract

The invention discloses a semi-supervised medical image segmentation method and system based on a visual language model. The method comprises the following steps: S1, obtaining a medical image data set to be segmented containing labeled and unlabeled samples; s2, pre-training the visual language model; s3, training the teacher model by using the labeled samples, and then inputting the unlabeled samples into the trained teacher model to obtain pseudo labels; s4, training the pre-trained visual language model to obtain a trained visual language model; s5, optimizing the trained visual language model; s6, training the segmentation model; s7, optimizing the trained segmentation model; and S8, adjusting the learning efficiency and the batch size, circulating the steps S1 to S7, and integrating the prediction results of the plurality of models to obtain a final segmentation result. According to the method, various visual and language features can be learned, a good basis is provided for transfer learning in a medical image segmentation task, and the model can adapt to different image features.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a semi-supervised medical image segmentation method and system based on a visual language model. Background Art

[0002] Medical image segmentation can be used to extract important organs or lesions from abnormal images, and plays an important role in computer-aided diagnosis and treatment research. In recent years, many medical image segmentation encoder-decoder networks based on supervised learning have achieved remarkable results, such as U-Net, U-Net++, H-DenseUNet, etc. However, these supervised learning methods rely heavily on a large amount of pixel-level labeled data, and annotating medical images in practice is usually very expensive. In addition, medical image annotation requires more expertise than natural images. Compared with supervised learning, semi-supervised learning is a new learning paradigm to solve the problem of incomplete data supervision. It uses a small amount of labeled data and a large amount of unlabeled data to achieve joint training. Obviously, in medical image segmentation, semi-supervised learning is more important than supervised learning and is more in line with the requirements of actual clinical scenarios. In semi-supervised medical image segmentation, the performance of segmentation is closely related to the quality of pseudo-labels. Current semi-supervised medical image segmentation methods may lead to insufficient complementarity between models when performing joint training, making the predictions of the same sample inconsistent, thus affecting the quality of pseudo-labels.

[0003] However, a major problem facing current semi-supervised image segmentation is the scarcity of labeled samples in training data. Summary of the invention

[0004] The present invention provides a semi-supervised medical image segmentation method and system based on a visual language model to solve the technical problems mentioned in the background technology.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] The present invention provides a semi-supervised medical image segmentation method based on a visual language model, comprising the following steps:

[0007] S1. Obtain a medical image dataset to be segmented that includes labeled and unlabeled samples, and preprocess the medical image dataset;

[0008] S2. Pre-train the visual language model to obtain a pre-trained visual language model;

[0009] S3. Select a segmentation model as a teacher model, and use the labeled samples to train the teacher model to obtain a trained teacher model;

[0010] S4. Using the medical image dataset to train the pre-trained visual language model to obtain a trained visual language model;

[0011] S5, enhancing the original medical image to obtain a variant image, then optimizing the trained visual language model using the original medical image and the variant image to obtain an optimized visual language model, inputting the unlabeled sample into the optimized visual language model, and inputting the output into the trained teacher model again to obtain an optimized pseudo label;

[0012] S6, select a segmentation model, and train the segmentation model using the labeled samples and the pseudo labels obtained in S5 to obtain a trained segmentation model;

[0013] S7, calculating a unit term according to a preliminary segmentation probability map of each pixel of the original medical image, defining an interaction term according to the similarity between adjacent pixels, and then optimizing the trained segmentation model according to the unit term and the interaction term to obtain an optimized segmentation model;

[0014] S8. Adjust the learning efficiency and batch size of the optimized segmentation model, loop S1 to S7, obtain multiple segmentation models, integrate the prediction results of multiple models, and obtain the final segmentation result.

[0015] Furthermore, the preprocessing in S1 at least includes image standardization, denoising and enhancement, and data enhancement;

[0016] Among them, image normalization is used to adjust the input image to a uniform size;

[0017] Denoising and enhancement is used to enhance the image contrast using contrast-limited adaptive histogram equalization and denoise the input image using image filters;

[0018] Data augmentation is used to enhance the input image using rotation, flipping, scaling, and translation methods.

[0019] Furthermore, the S2 specifically includes the following steps:

[0020] S21. Selecting a plurality of original medical images from a medical image dataset, where the original medical images are labeled samples or unlabeled samples, and performing data enhancement on the plurality of original medical images to obtain a plurality of enhanced images;

[0021] S22, providing a text description for the multiple original medical images and the multiple enhanced images;

[0022] S23. Use a convolutional neural network (CNN) or a visual network (Transformer) as a visual encoder; use a BERT model or a GPT model as a text encoder, and then construct a momentum visual encoder and a momentum text encoder respectively;

[0023] S24, inputting the original medical image into the visual encoder 1 to obtain the original image embedding; inputting the enhanced image into the momentum visual encoder to obtain the enhanced image embedding;

[0024] S25, inputting the original medical image into a text encoder 1 to obtain an original text embedding; inputting the enhanced image into a momentum text encoder to obtain a momentum text embedding;

[0025] S26, calculating a contrast loss function 1 according to the outputs of S24 and S25;

[0026] S27, loop S24 to S26, maximize the similarity between the original image embedding and the corresponding text embedding, and minimize the similarity between the original image embedding and the enhanced image embedding;

[0027] S28, using a momentum update strategy to update the parameters of the momentum visual encoder and the momentum text encoder respectively, so that the current parameters of the momentum visual encoder are close to the weights of the visual encoder one, and the current parameters of the momentum text encoder are close to the weights of the text encoder one;

[0028] The momentum update strategy is expressed in the following formula:

[0029] θ momentum =m×θ momentum +(1-m)×θ main

[0030] Where m is the momentum decay factor, θ momentum is the current parameter of the momentum encoder, which includes the momentum visual encoder and the momentum text encoder, θ main is the weight of the original encoder; the original encoder includes visual encoder 1 and text encoder 1.

[0031] Furthermore, the S3 specifically includes the following steps:

[0032] S31, select the U-Net segmentation model as the teacher model and initialize the teacher model;

[0033] S32. Use the labeled samples to train the initialized teacher model to obtain a trained teacher model. The training objective is to minimize the following formula:

[0034]

[0035] Among them, N1 is the number of labeled samples, f is the teacher model, and L teacher is the loss function of the teacher model, and Respectively represent the i-th original medical image in the labeled sample and its corresponding segmentation annotation mask;

[0036] S33, inputting the unlabeled samples into the trained teacher model to obtain the corresponding pseudo labels;

[0037] S34, set a confidence threshold T, and then use the confidence threshold T to filter multiple pseudo labels to obtain the retained pseudo labels The filtering process is expressed by the formula as follows:

[0038]

[0039] Among them, X u represents unlabeled data, f(X u ) represents the pseudo labels generated by the teacher model for the unlabeled data.

[0040] Furthermore, the S4 specifically includes the following steps:

[0041] S41, inputting the original medical image X into the pre-trained visual language model, extracting features using a visual encoder in the pre-trained visual language model, and obtaining an image feature embedding h(X); then extracting text features from the corresponding text description T of the original medical image X using a text encoder in the pre-trained visual language model, and obtaining a text feature embedding g(T);

[0042] S42, based on the cosine similarity, the similarity between the image feature embedding h(X) and the text feature embedding g(T) is calculated, and the formula is as follows:

[0043]

[0044] Among them, h(X)·g(T) represents the vector inner product, ||·|| is the Euclidean norm of the vector;

[0045] S43, constructing a contrast loss function 2 based on the similarity obtained in S42, the contrast loss function 2 is specifically as follows:

[0046]

[0047] Among them, L contrastive represents the loss value of contrast loss function 2; T is the text description matching the original medical image X, T j is the description of all candidate texts; N is the total number of labeled samples and unlabeled samples;

[0048] S44, loop S41 to S43, maximize the similarity between the image and the corresponding text through the contrast loss function 2, and minimize the similarity between the image and the unmatched text; optimize the parameters in the visual encoder 1 and the text encoder 1 through the back propagation algorithm and based on the contrast loss function 2 to obtain the trained visual language model.

[0049] Furthermore, the S5 specifically includes the following steps:

[0050] S51, enhancing the original medical image X to generate a variant image X aug ;Enhancement methods include rotation, cropping or brightness adjustment;

[0051] S52: the original medical image X and the variant image X aug Input to the visual encoder 1, extract the image features of the two and embed them into h(X) and h(X) respectively. aug );

[0052] S53, embedding h(X) and h(X) based on the image features of the two aug ) and the corresponding text embeddings to construct loss function three;

[0053] S54, loop S51 to S53, minimize the loss function three, and in the process update the weights of the trained visual language model through the back propagation algorithm to obtain the optimized visual language model.

[0054] S55, unlabeled sample X u Input the optimized visual language model to extract image features h(X u ) and text features g(T u ), where T u is the same as the unlabeled sample X u Corresponding text description; the image feature h(X u ) and text features g(T u ) together with the pixel-level features of the original medical image X are input into the teacher model of S3 to generate optimized pseudo labels

[0055] Furthermore, the S6 specifically includes the following steps:

[0056] S61. According to the segmentation task requirements, select a model from the segmentation model U-Net and the segmentation model Mask R-CNN as the selected segmentation model;

[0057] S62: Mix the pseudo labels obtained in S5 with the annotated samples and input them into the selected segmentation model to obtain the model prediction output.

[0058] S63. Output based on model prediction Construct loss function 4, the specific loss function 4 is as follows:

[0059]

[0060] Among them, L seg is the loss value of loss function 4; Y i The true labels corresponding to the labeled samples and the pseudo labels corresponding to the unlabeled samples in S3;

[0061] S64, loop S62 to S63, minimize loss function four, and in the process update the weights of the segmentation model through the back propagation algorithm to obtain the trained segmentation model.

[0062] Furthermore, the S7 specifically includes the following steps:

[0063] S71, calculate the unit item U(y) according to the preliminary segmentation probability map of each pixel point of the original medical image X. i ), whose formula is,

[0064] U(y i )=-logP(y i )

[0065] Among them, P(y i ) indicates that pixel i belongs to label y i probability;

[0066] S72, define the interaction term V(y i ,y j ), whose formula is,

[0067]

[0068] In the formula, μ(y i ≠y j ) is a penalty term, indicating the energy values ​​of different labels of adjacent pixels; p i , p j are the position coordinates of pixel points i and j; σ is the smoothing parameter of the Gaussian kernel;

[0069] S73, according to the unit item U(y i ), the interaction term V(y i ,y j ) construct an energy function, and loop S71 to S72 to minimize the energy function to obtain an optimized segmentation model; the specific details of minimizing the energy function are as follows:

[0070]

[0071] Among them, E CRF (Y) represents the energy function; Y represents the segmentation label set; y i ,y j Represent the segmentation labels of pixel i and , respectively.

[0072] Furthermore, the S8 specifically includes the following steps:

[0073] S81, dynamically adjust the learning rate, batch size and structure of the optimized segmentation model;

[0074] S82. Repeat S1 to S7 to obtain multiple segmentation models, train the multiple segmentation models, and then perform weighted averaging on the results of the multiple segmentation models to improve the segmentation accuracy of the final model and obtain the final segmentation result.

[0075] On the other hand, the present invention also provides a semi-supervised medical image segmentation system based on a visual language model, including a computer device, and the computer device is programmed or configured to execute the above semi-supervised medical image segmentation method.

[0076] Beneficial effects of the present invention:

[0077] 1. The present invention can extract richer contextual information based on the visual language model, so that when processing medical images, it can more accurately identify and segment key structures or abnormal areas. In addition, the text description provides additional semantic information for the visual language model, making it easier to identify clinically relevant features. The visual language model can be pre-trained on a large-scale data set and can learn a variety of visual and language features, which provides a good foundation for transfer learning in medical image segmentation tasks, allowing the visual language model to adapt to different image features.

[0078] 2. The present invention reduces the dependence on expensive annotation resources and improves the robustness of the segmentation model by utilizing a large amount of unlabeled data for training. The visual language model combines visual and language information, so that the visual language model can not only understand the visual features of the image, but also understand the related text descriptions (such as medical reports), thereby improving the visual language model's ability to analyze complex medical images.

[0079] 3. The present invention effectively increases the diversity of training data through data augmentation, so that the visual language model and the segmentation model can be exposed to a wider range of sample variations during training, which helps to improve the generalization ability of the segmentation model. In the case of scarce labeled data, direct training may cause the model to overfit the training samples. By applying data augmentation, the model can see different image variants in each training iteration, reducing dependence on specific samples, thereby improving performance on unseen samples. At the same time, image enhancement technology can highlight important anatomical structures or pathological changes, so that the model can learn features more effectively during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 Flow chart of the semi-supervised medical image segmentation method in the present invention. DETAILED DESCRIPTION

[0081] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0082] Reference Figure 1 The present application embodiment provides a semi-supervised medical image segmentation method based on a visual language model, comprising the following steps:

[0083] S1. Obtain a medical image dataset to be segmented that includes labeled and unlabeled samples, and preprocess the medical image dataset;

[0084] S2. Pre-training the visual language model to obtain a pre-trained visual language model; the visual language model is specifically any one of a CLIP model, a BLIP model or a ViLT model;

[0085] S3. Select a segmentation model as the teacher model, and use the labeled samples to train the teacher model to obtain the trained teacher model, and then input the unlabeled samples into the trained teacher model to obtain pseudo labels;

[0086] S4. Use the medical image dataset to train the pre-trained visual language model to improve the visual language model's ability to model the matching relationship between images and texts, and obtain an optimized visual language model;

[0087] S5, enhance the original medical image to obtain a variant image, then use the original medical image and the variant image to optimize the trained visual language model to obtain an optimized visual language model, and improve the quality of pseudo-label generation through the optimized visual language model; input the unlabeled sample into the optimized visual language model, and input the output into the trained teacher model again to obtain an optimized pseudo-label; the goal of S5 optimization is to improve the model's ability to understand image and text features through self-supervised learning, so that it can better capture the semantic association between the two;

[0088] S6, select a segmentation model, and train the segmentation model using the labeled samples and the pseudo labels obtained in S5 to obtain a trained segmentation model;

[0089] S7, calculating a unit term according to a preliminary segmentation probability map of each pixel of the original medical image, defining an interaction term according to the similarity between adjacent pixels, and then optimizing the trained segmentation model according to the unit term and the interaction term to obtain an optimized segmentation model;

[0090] S8. Adjust the learning efficiency and batch size of the optimized segmentation model, loop S1 to S7, obtain multiple segmentation models, integrate the prediction results of multiple models, and obtain the final segmentation result.

[0091] In some embodiments, the preprocessing in S1 includes at least image normalization, denoising and enhancement, and data enhancement;

[0092] Among them, image normalization is used to adjust the input image to a uniform size;

[0093] Denoising and enhancement is used to enhance the image contrast using contrast-limited adaptive histogram equalization and denoise the input image using image filters;

[0094] Data augmentation is used to enhance the input image using rotation, flipping, scaling, and translation methods.

[0095] The present invention effectively increases the diversity of training data through data augmentation, so that the visual language model and segmentation model can be exposed to a wider range of sample variations during training, which helps to improve the generalization ability of the model. In the case of scarce labeled data, direct training may cause the model to overfit the training samples. By applying data augmentation, the model can see different image variants in each training iteration, reducing dependence on specific samples, thereby improving performance on unseen samples. At the same time, image enhancement technology can highlight important anatomical structures or pathological changes, so that the model can learn features more effectively during training.

[0096] In some embodiments, S2 specifically includes the following steps:

[0097] S21. Selecting a plurality of original medical images from a medical image dataset, where the original medical images are labeled samples or unlabeled samples, and performing data enhancement on the plurality of original medical images to obtain a plurality of enhanced images;

[0098] S22, providing a text description for the multiple original medical images and the multiple enhanced images;

[0099] S23. Use a convolutional neural network (CNN) or a visual network (Transformer) as a visual encoder; use a BERT model or a GPT model as a text encoder, and then construct a momentum visual encoder and a momentum text encoder respectively;

[0100] S24, inputting the original medical image into the visual encoder 1 to obtain the original image embedding; inputting the enhanced image into the momentum visual encoder to obtain the enhanced image embedding;

[0101] S25, inputting the original medical image into a text encoder 1 to obtain an original text embedding; inputting the enhanced image into a momentum text encoder to obtain a momentum text embedding;

[0102] S26, calculating a contrast loss function 1 according to the outputs of S24 and S25;

[0103] S27, loop S24 to S26, maximize the similarity between the original image embedding and the corresponding text embedding, and minimize the similarity between the original image embedding and the enhanced image embedding;

[0104] S28, using a momentum update strategy to update the parameters of the momentum visual encoder and the momentum text encoder respectively, so that the current parameters of the momentum visual encoder are close to the weights of the visual encoder one, and the current parameters of the momentum text encoder are close to the weights of the text encoder one;

[0105] The momentum update strategy is expressed in the following formula:

[0106] θ momentum =m×θ momentum +(1-m)×θ main

[0107] Where m is the momentum decay factor, θ momentum is the current parameter of the momentum encoder, which includes the momentum visual encoder and the momentum text encoder, θ main is the weight of the original encoder; the original encoder includes visual encoder 1 and text encoder 1.

[0108] In some embodiments, S3 specifically includes the following steps:

[0109] S31, select the U-Net segmentation model as the teacher model and initialize the teacher model;

[0110] S32. Use the labeled samples to train the initialized teacher model to obtain a trained teacher model. The training objective is to minimize the following formula:

[0111]

[0112] Among them, N1 is the number of labeled samples, f is the teacher model, and L teacher is the loss function of the teacher model, and Respectively represent the i-th original medical image in the labeled sample and its corresponding segmentation annotation mask;

[0113] S33, inputting the unlabeled samples into the trained teacher model to obtain multiple pseudo labels;

[0114] S34, set the confidence threshold τ, and then use the confidence threshold T to filter multiple pseudo labels to obtain the retained pseudo labels The filtering process is expressed by the formula as follows:

[0115]

[0116] Among them, X u represents unlabeled data, f(X u ) represents the pseudo labels generated by the teacher model for the unlabeled data.

[0117] In some embodiments, the S4 specifically includes the following steps:

[0118] S41, inputting the original medical image X into the pre-trained visual language model, extracting features using a visual encoder in the pre-trained visual language model, and obtaining an image feature embedding h(X); then extracting text features from the corresponding text description T of the original medical image X using a text encoder in the pre-trained visual language model, and obtaining a text feature embedding g(T);

[0119] S42, based on the cosine similarity, the similarity between the image feature embedding h(X) and the text feature embedding g(T) is calculated, and the formula is as follows:

[0120]

[0121] Among them, h(X)·g(T) represents the vector inner product, ||·|| is the Euclidean norm of the vector;

[0122] S43, constructing a contrast loss function 2 based on the similarity obtained in S42, the contrast loss function 2 is specifically as follows:

[0123]

[0124] Among them, L contrastive represents the loss value of contrast loss function 2; T is the text description matching the original medical image X, T j is the description of all candidate texts; N is the total number of labeled samples and unlabeled samples;

[0125] S44, loop S41 to S43, maximize the similarity between the image and the corresponding text through the contrast loss function 2, and minimize the similarity between the image and the unmatched text; optimize the parameters in the visual encoder 1 and the text encoder 1 through the back propagation algorithm and based on the contrast loss function 2 to enhance the visual language model's modeling ability for the image and text matching relationship, and obtain the trained visual language model.

[0126] The present invention reduces the dependence on expensive annotation resources and improves the robustness of the model by utilizing a large amount of unlabeled data for training. The visual language model combines visual and language information so that the visual language model can not only understand the visual features of the image, but also understand the related text descriptions (such as medical reports), thereby improving the visual language model's ability to analyze complex medical images.

[0127] The visual language model extracts richer contextual features by combining the visual features of medical images and the semantic information of text descriptions. The text provides semantic guidance for key structures or abnormal areas (such as location, morphology, etc.). At the same time, the visual and language features obtained by pre-training the model on large-scale data enhance its generalization ability, making it able to adapt to different scenarios and complex structures, thereby more accurately identifying and segmenting the target area.

[0128] In some embodiments, S5 specifically includes the following steps:

[0129] S51, enhancing the original medical image X to generate a variant image X aug ;Enhancement methods include rotation, cropping or brightness adjustment;

[0130] S52: the original medical image X and the variant image X aug Input to the visual encoder 1, extract the image features of the two and embed them into h(X) and h(X) respectively. aug );

[0131] S53, embedding h(X) and h(X) based on the image features of the two aug ) and the corresponding text embeddings to construct loss function three;

[0132] S54, loop S51 to S53, minimize the loss function three, and in the process update the weights of the trained visual language model through the back propagation algorithm to obtain the optimized visual language model.

[0133] S55, unlabeled sample X u Input the optimized visual language model to extract image features h(X u ) and text features g(T u ), where T u is the same as the unlabeled sample X u Corresponding text description; the image feature h(X u ) and text features g(T u ) together with the pixel-level features of the original medical image X are input into the teacher model of S3 to generate optimized pseudo labels

[0134] In some embodiments, the S6 specifically includes the following steps:

[0135] S61. According to the segmentation task requirements, select a model from the segmentation model U-Net and the segmentation model Mask R-CNN as the selected segmentation model; the segmentation model U-Net is suitable for smaller-scale data sets and has efficient feature extraction and context information fusion capabilities, and the segmentation model Mask R-CNN is suitable for more complex tasks and can perform target detection and segmentation at the same time;

[0136] S62: Mix the pseudo labels obtained in S5 with the annotated samples and input them into the selected segmentation model to obtain the model prediction output.

[0137] S63. Output based on model prediction Construct loss function 4, the specific loss function 4 is as follows:

[0138]

[0139] Among them, L seg is the loss value of loss function 4; Y i The true labels corresponding to the labeled samples and the pseudo labels corresponding to the unlabeled samples in S3;

[0140] S64, loop S62 to S63, minimize loss function four, and in the process update the weights of the segmentation model through the back propagation algorithm to obtain the trained segmentation model.

[0141] In some embodiments, the S7 specifically includes the following steps:

[0142] S71, calculate the unit item U(y) according to the preliminary segmentation probability map of each pixel point of the original medical image X.i ), whose formula is,

[0143] U(y i )=-logP(y i )

[0144] Among them, P(y i ) indicates that pixel i belongs to label y i probability;

[0145] S72, define the interaction term V(y i ,y j ), whose formula is,

[0146]

[0147] In the formula, μ(y i ≠y j ) is a penalty term, indicating the energy values ​​of different labels of adjacent pixels; p i , p j are the position coordinates of pixel points i and j; σ is the smoothing parameter of the Gaussian kernel;

[0148] S73, according to the unit item U(y i ), the interaction term V(y i ,y j ) construct an energy function, and loop S71 to S72 to minimize the energy function, so that the segmentation result is smoother and more accurate, and an optimized segmentation model is obtained; the specific energy function minimization is as follows:

[0149]

[0150] Among them, E CRF (Y) represents the energy function; Y represents the segmentation label set; y i ,y j Represent the segmentation labels of pixel i and j respectively. i ) indicates that pixel i belongs to segmentation label y i The energy value, V(y i ,y j ) represents the interaction energy between pixel i and pixel ,.

[0151] In some embodiments, the S8 specifically includes the following steps:

[0152] S81, dynamically adjust the learning rate, batch size and structure of the optimized segmentation model;

[0153] S82. Repeat S1 to S7 to obtain multiple segmentation models, train the multiple segmentation models, and then perform weighted averaging on the results of the multiple segmentation models to improve the segmentation accuracy of the final model as well as the segmentation performance and robustness of the model to obtain the final segmentation result.

[0154] On the other hand, the present invention also provides a semi-supervised medical image segmentation system based on a visual language model, including a computer device, and the computer device is programmed or configured to execute the above semi-supervised medical image segmentation method.

[0155] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A semi-supervised medical image segmentation method based on a visual language model, characterized in that: The steps include: S1. Obtain a medical image dataset to be segmented that includes labeled and unlabeled samples, and preprocess the medical image dataset; S2. Pre-train the visual language model to obtain a pre-trained visual language model; S3. Select a segmentation model as a teacher model, and use the labeled samples to train the teacher model to obtain a trained teacher model; S4. Using the medical image dataset to train the pre-trained visual language model to obtain a trained visual language model; S5, enhancing the original medical image to obtain a variant image, then optimizing the trained visual language model using the original medical image and the variant image to obtain an optimized visual language model, inputting the unlabeled sample into the optimized visual language model, and inputting the output into the trained teacher model again to obtain an optimized pseudo label; S6, select a segmentation model, and train the segmentation model using the labeled samples and the pseudo labels obtained in S5 to obtain a trained segmentation model; S7, calculating a unit term according to a preliminary segmentation probability map of each pixel of the original medical image, defining an interaction term according to the similarity between adjacent pixels, and then optimizing the trained segmentation model according to the unit term and the interaction term to obtain an optimized segmentation model; S8. Adjust the learning efficiency and batch size of the optimized segmentation model, loop S1 to S7, obtain multiple segmentation models, integrate the prediction results of multiple models, and obtain the final segmentation result.

2. The semi-supervised medical image segmentation method according to claim 1, characterized in that: The preprocessing in S1 at least includes image standardization, denoising and enhancement, and data enhancement; Among them, image normalization is used to adjust the input image to a uniform size; Denoising and enhancement is used to enhance the image contrast using contrast-limited adaptive histogram equalization and denoise the input image using image filters; Data augmentation is used to enhance the input image using rotation, flipping, scaling, and translation methods.

3. The semi-supervised medical image segmentation method according to claim 1, characterized in that: The S2 specifically includes the following steps: S21. Selecting a plurality of original medical images from a medical image dataset, where the original medical images are labeled samples or unlabeled samples, and performing data enhancement on the plurality of original medical images to obtain a plurality of enhanced images; S22, providing a text description for the multiple original medical images and the multiple enhanced images; S23. Use a convolutional neural network (CNN) or a visual network (Transformer) as a visual encoder; use a BERT model or a GPT model as a text encoder, and then construct a momentum visual encoder and a momentum text encoder respectively; S24, inputting the original medical image into the visual encoder 1 to obtain the original image embedding; inputting the enhanced image into the momentum visual encoder to obtain the enhanced image embedding; S25, inputting the original medical image into the text encoder 1 to obtain the original text embedding; Input the enhanced image into the momentum text encoder to obtain the momentum text embedding; S26, calculating a contrast loss function 1 according to the outputs of S24 and S25; S27, loop S24 to S26, maximize the similarity between the original image embedding and the corresponding text embedding, and minimize the similarity between the original image embedding and the enhanced image embedding; S28, using a momentum update strategy to update the parameters of the momentum visual encoder and the momentum text encoder respectively, so that the current parameters of the momentum visual encoder are close to the weights of the visual encoder one, and the current parameters of the momentum text encoder are close to the weights of the text encoder one; The momentum update strategy is expressed in the following formula: i momentum =m×θ momentum +(1-m)×θ main Where m is the momentum decay factor, θ momentum is the current parameter of the momentum encoder, which includes the momentum visual encoder and the momentum text encoder, θ main is the weight of the original encoder; the original encoder includes visual encoder 1 and text encoder 1.

4. The semi-supervised medical image segmentation method according to claim 1, characterized in that: The S3 specifically includes the following steps: S31, select the U-Net segmentation model as the teacher model and initialize the teacher model; S32. Use the labeled samples to train the initialized teacher model to obtain a trained teacher model. The training objective is to minimize the following formula: Among them, N1 is the number of labeled samples, f is the teacher model, and L teacher is the loss function of the teacher model, and Y1 i Respectively represent the i-th original medical image in the labeled sample and its corresponding segmentation annotation mask; S33, inputting the unlabeled samples into the trained teacher model to obtain the corresponding pseudo labels; S34, set the confidence threshold τ, and then use the confidence threshold T to filter multiple pseudo labels to obtain the retained pseudo labels The filtering process is expressed by the formula as follows: Among them, X u represents an unlabeled sample; f(X u ) represents the pseudo labels of unlabeled samples generated by the teacher model.

5. The semi-supervised medical image segmentation method according to claim 4, characterized in that: The S4 specifically includes the following steps: S41, inputting the original medical image X into the pre-trained visual language model, extracting features using a visual encoder in the pre-trained visual language model, and obtaining an image feature embedding h(X); then extracting text features from the corresponding text description T of the original medical image X using a text encoder in the pre-trained visual language model, and obtaining a text feature embedding g(T); S42, based on the cosine similarity, the similarity between the image feature embedding h(X) and the text feature embedding g(T) is calculated, and the formula is as follows: Among them, h(X)·g(T) represents the vector inner product, ||·|| is the Euclidean norm of the vector; S43, constructing a contrast loss function 2 based on the similarity obtained in S42, the contrast loss function 2 is specifically as follows: Among them, L contrastive represents the loss value of contrast loss function 2; T is the text description matching the original medical image X, T j is the description of all candidate texts; N is the total number of labeled samples and unlabeled samples; S44, loop S41 to S43, maximize the similarity between the image and the corresponding text through the contrast loss function 2, and minimize the similarity between the image and the unmatched text; optimize the parameters in the visual encoder 1 and the text encoder 1 through the back propagation algorithm and based on the contrast loss function 2 to obtain the trained visual language model.

6. The semi-supervised medical image segmentation method according to claim 5, characterized in that: The S5 specifically includes the following steps: S51, enhancing the original medical image X to generate a variant image X aug ;Enhancement methods include rotation, cropping or brightness adjustment; S52: the original medical image X and the variant image X aug Input to the visual encoder 1, extract the image features of the two and embed them into h(X) and h(X) respectively. aug ); S53, embedding h(X) and h(X) according to the image features of the two aug ) and the corresponding text embeddings to construct loss function three; S54, looping from S51 to S53, minimizing the loss function three, and in the process updating the weights of the trained visual language model through the back propagation algorithm to obtain the optimized visual language model; S55, unlabeled sample X u Input the optimized visual language model to extract image features h(X u ) and text features g(X u ), where T u is the same as the unlabeled sample X u Corresponding text description; the image feature h(X u ) and text features g(T u ) together with the pixel-level features of the original medical image X are input into the trained teacher model in S3 to generate optimized pseudo labels 7. The semi-supervised medical image segmentation method according to claim 6, characterized in that: The S6 specifically includes the following steps: S61. According to the segmentation task requirements, select a model from the segmentation model U-Net and the segmentation model Mask R-CNN as the selected segmentation model; S62: Mix the pseudo labels obtained in S5 with the annotated samples and input them into the selected segmentation model to obtain the model prediction output. S63. Output based on model prediction Construct loss function 4, the specific loss function 4 is as follows: Among them, L seg is the loss value of loss function 4; Y i The true labels corresponding to the labeled samples and the pseudo labels corresponding to the unlabeled samples in S3; S64, loop S62 to S63, minimize loss function four, and in the process update the weights of the segmentation model through the back propagation algorithm to obtain the trained segmentation model.

8. The semi-supervised medical image segmentation method according to claim 7, characterized in that: The S7 specifically includes the following steps: S71. Calculate the unit term U(yi) according to the preliminary segmentation probability map of each pixel of the original medical image X. The formula is: U(and i )=-logP(y i ) Among them, P(y i ) indicates that pixel i belongs to label y i probability; S72, define the interaction term V(y i ,y i ), whose formula is, In the formula, μ(y i ≠y i ) is a penalty term, indicating the energy values ​​of different labels of adjacent pixels; p i , p j are the position coordinates of pixel points i and j; σ is the smoothing parameter of the Gaussian kernel; S73, according to the unit item U(y i ), the interaction term V(y i ,y j ) construct an energy function, and loop S71 to S72 to minimize the energy function to obtain an optimized segmentation model; the specific details of minimizing the energy function are as follows: Among them, E CRF (Y) represents the energy function; Y represents the segmentation label set; y i ,y j Represent the segmentation labels of pixel i and j respectively.

9. The semi-supervised medical image segmentation method according to claim 8, characterized in that: The S8 specifically includes the following steps: S81, dynamically adjust the learning rate, batch size and structure of the optimized segmentation model; S82. Repeat S1 to S7 to obtain multiple segmentation models, train the multiple segmentation models, and then perform weighted averaging on the results of the multiple segmentation models to improve the segmentation accuracy of the final model and obtain the final segmentation result.

10. A semi-supervised medical image segmentation system based on a visual language model, comprising a computer device, characterized in that: The computer device is programmed or configured to execute the semi-supervised medical image segmentation method according to any one of claims 1 to 9.

Images