Medical image segmentation method and system based on prompt learning

Abstract

The invention provides a medical image segmentation method and system based on prompt learning, and belongs to the technical field of medical image segmentation, and the method comprises the steps: obtaining a medical image and a text prompt; the medical images are respectively input into a teacher model and a student model, and the text prompt is input into the teacher model; the teacher model forms semantic guidance based on the text prompt, modulates the features of the medical image through the semantic guidance and outputs a first segmentation result; the student model processes the received medical image to obtain a second segmentation result; using mixed supervision to update student model parameters for the first segmentation result and the second segmentation result; and inputting a to-be-segmented medical image into the student model after parameter updating to obtain a segmentation result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image segmentation, and particularly relates to a medical image segmentation method and system based on prompt learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Medical image segmentation refers to the process of using computer algorithms to separate the regions of interest from the background or other tissues in medical images such as CT, MRI, X-ray, etc. In simple terms, it is to assign a label to each pixel or voxel in the image, such as "tumor", "bone", "blood vessel" or "background". Medical image segmentation plays a crucial role in computer-aided diagnosis, treatment planning and disease monitoring. With the widespread application of medical imaging technology, there is an increasing demand for efficient, accurate and automated image segmentation methods.

[0004] Traditional segmentation methods based on convolutional neural networks (CNN), especially the encoding-decoding architecture such as U-Net, have made significant progress in multiple medical image segmentation tasks. However, these methods still have some limitations that are difficult to overcome. For example, in tasks involving small target lesions or complex morphological structures, the segmentation accuracy is often greatly affected by factors such as class imbalance, low contrast, fuzzy boundaries and high annotation costs. These problems limit the robustness and generalizability of existing models in real clinical environments.

[0005] In recent years, with the rapid development of visual-linguistic models, prompt learning has been gradually introduced into the field of medical imaging. The representative work CLIP model achieves cross-modal alignment and rich semantic understanding by jointly training image and text features. Inspired by this, some studies attempt to introduce text prompts in image classification, retrieval or coarse-grained segmentation tasks to enhance the semantic information of feature representation. However, such methods often rely on text input at the inference stage, not only increasing deployment complexity, but also limiting feasibility in actual clinical applications. In addition, existing methods mostly target a single imaging modality or specific disease, lacking universality across tasks and modalities. SUMMARY

[0006] To overcome the shortcomings of the prior art, the present application provides a medical image segmentation method and system based on prompt learning, which improves segmentation accuracy and generalization performance while avoiding dependence on text at the inference stage.

[0007] To achieve the above-mentioned purposes, one or more embodiments of the present application provide the following technical solutions: In a first aspect, a medical image segmentation method based on prompt learning is disclosed, comprising: obtaining a medical image and a text prompt; inputting the medical image into a teacher model and a student model respectively, and inputting the text prompt into the teacher model; the teacher model forms a semantic guide based on the text prompt, modulates the features of the medical image through the semantic guide, and outputs a first segmentation result; the student model processes the received medical image to obtain a second segmentation result; updating the student model parameters using hybrid supervision for the first segmentation result and the second segmentation result; inputting the medical image to be segmented into the student model with updated parameters to obtain a generated segmentation result.

[0008] As a further technical solution, the teacher model comprises a text encoder, a semantic guide module, an image encoder, a feature modulation module, and a decoder. The image encoder is configured to extract features from the input medical image to obtain multi-layer hierarchical features, gradually abstracting from low-level edges to high-level semantics. The text encoder is configured to encode the input text prompt to generate a text embedding vector representing the semantic information of the text prompt. The semantic guide module is configured to generate affine modulation parameters using the text embedding vector. The feature modulation module is configured to use affine modulation parameters to modulate the image features of the corresponding layer for each layer of features output by the image encoder, to achieve cross-modal semantic guidance and focus on regions with specific semantics. The decoder is configured to decode the semantically modulated features step by step, recover the spatial resolution through skip connection and upsampling, and obtain the first segmentation result.

[0009] As a further technical solution, the student model comprises an image encoder and a decoder. The image encoder extracts features from the input medical image. The decoder decodes the extracted features to obtain the second segmentation result.

[0010] As a further technical solution, if the text prompt is missing, a text prompt is automatically generated.

[0011] As a further technical solution, the step of automatically generating a text prompt is: input a binary mask image into a processing class, add a text label to each picture according to the number of connected domains, the total pixel area and white pixel area of each connected domain, and the circularity of the connected domain. The image information is combined into a structured text prompt through a text tag splicing program.

[0012] As a further technical solution, the semantic guidance module adopts a multi-layer perception machine, which generates modulation parameters matched with the number of visual feature channels according to the text embedding vector, for performing affine transformation and feature modulation on feature maps of different scales in the image encoder.

[0013] As a further technical solution, in the feature modulation module, text prompt information is introduced into the feature space of different scales, the semantic response of the visual feature map is adjusted, and feature fusion is modulated.

[0014] As a further technical solution, the hybrid supervision package adopts a total loss composed of output distillation loss and feature distillation loss, which jointly optimizes the performance and generalization ability of the student model.

[0015] In a second aspect, a medical image segmentation system based on prompt learning is disclosed, comprising: A data acquisition module configured to acquire a medical image and a text prompt; A model processing module configured to input the medical image into a teacher model and a student model, and input the text prompt into the teacher model; The teacher model forms a semantic guide based on the text prompt, modulates the features of the medical image through the semantic guide, and outputs a first segmentation result; The student model processes the received medical image to obtain a second segmentation result; A parameter updating module configured to update the student model parameters using hybrid supervision for the first segmentation result and the second segmentation result; A segmentation module configured to input a medical image to be segmented into the student model with updated parameters to obtain a generated segmentation result.

[0016] The above one or more technical solutions have the following beneficial effects: The technical solution of the present application discloses a prompt guidance type semantic distillation framework, namely UText. The method introduces text prompts in the training stage to obtain richer semantic representations, guides feature learning through text prompts in the training process, thereby achieving stronger semantic alignment and generalization ability, and realizes semantic distillation through a teacher-student structure, thereby completely eliminating the dependence on text input in the inference stage.

[0017] In the technical scheme of the present application, the teacher model embeds disease-related prompts into multiple decoder stages through a text encoder, and realizes hierarchical feature modulation with the help of semantic guidance. Subsequently, through a semantic distillation mechanism, the student model gradually learns the semantic information of the teacher model in the process of aligning multi-scale features and representation levels, thereby completing reasoning only relying on images, and being able to balance the semantic understanding ability and the clinical deployment feasibility.

[0018] The UText framework effectively integrates the semantic guidance and semantic distillation mechanism, improves the segmentation accuracy and generalization performance, avoids the dependence on text in the reasoning stage, and provides a new technical path for cross-modal medical image analysis.

[0019] The technical scheme of the present application is verified on public and local data sets, and shows stronger accuracy, robustness and interpretability. Compared with the existing method which depends on text prompts in the reasoning stage, the present method not only maintains the performance improvement brought by semantic alignment, but also takes into account the simplicity and efficiency in actual deployment. At the same time, the design of UText has high flexibility, can respond to various text prompts and realize personalized adjustment, and provides a broader application prospect for clinical decision support.

[0020] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of the present application, are used to provide further understanding of the present application, and the illustrative embodiments of the present application and their description are used to explain the present application, and do not constitute improper limitations on the present application.

[0022] Figure 1 The figure is a schematic diagram of the overall structure of the embodiment system of the present application. Figure 2 The figure is a semantic guidance module of the embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0024] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments, and are not intended to limit the exemplary embodiments according to the present application.

[0025] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0026] Term explanation: Prompt Learning: Prompt learning: a deep learning paradigm that guides the model to complete the specified task by introducing natural language text prompts.

[0027] CLIP: Image-text pre-training model: a jointly trained model that can map images and text to the same feature space, enabling cross-modal understanding.

[0028] MLP, Multi-Layer Perceptron: Multi-layer perceptron: a feedforward neural network structure composed of multiple fully connected layers, commonly used for feature mapping and nonlinear transformation.

[0029] U-Net: A classic fully convolutional neural network for image segmentation tasks, with an encoding-decoding structure.

[0030] Example one Referring to the accompanying Figure 1 The embodiment discloses a medical image segmentation method based on prompt learning, which comprises: Step one: obtain medical images and text prompts; Step two: input the medical images into the teacher model and the student model respectively, and input the text prompts into the teacher model; The teacher model forms semantic guidance based on the text prompts, modulates the features of the medical images through semantic guidance, and outputs the first segmentation result; Step three: the student model processes the received medical images to obtain the second segmentation result; Step four: calculate the hard label loss and soft label distillation loss, and update the student model parameters; Step five: after training, in the deployment stage, only the medical images to be segmented are input into the student model with updated parameters to obtain the generated segmentation result.

[0031] In the technical solution of the embodiment, a teacher-student network architecture is adopted, both of which are based on the encoding-decoding paradigm. The teacher network adopts the UText-teacher model to receive image and text input, and the student network adopts the UText-student model to receive only image input. In the training stage, the student network simulates the semantic perception ability of the teacher network through the semantic distillation mechanism, so as to realize high-precision segmentation only relying on images in the inference stage.

[0032] In one embodiment, the teacher model includes a text encoder, a semantic guidance module, an image encoder, a feature modulation module, and a decoder. For small target lesions with similar morphology or complex morphological structure lesions, the teacher model can load the same or similar modulation parameters through the text encoder and the semantic guidance module, enhancing the model's attention to this part.

[0033] Regarding the image encoder, it is configured to perform feature extraction on the input medical image. The encoder has multiple levels, so features at different levels can be obtained, from low-level edges to high-level semantics. These features are then modulated and fused with information from the text prompt, and together enter the decoder, thereby achieving cross-modal semantic perception and segmentation guidance.

[0034] In an embodiment, a ResNet series network is used as the backbone encoder for visual feature extraction. However, the present application is not limited to this, and any deep neural network capable of extracting multi-level visual features can be substituted to achieve the same technical purpose. For example, a convolutional neural network (CNN) can be substituted, a Transformer can be substituted, a hybrid structure can be substituted, etc., in order to balance the ability to model local textures and the ability to perceive global context, thereby achieving better performance in complex medical scenarios. For different image modalities, such as CT, MRI, and ultrasound, a more suitable backbone network can be selected.

[0035] Through the above-mentioned diversified encoder substitution, the present application can adapt to different medical image types and task requirements, improving the overall universality and promotion value.

[0036] In an embodiment, the text encoder uses a CLIP text encoder, which is configured to encode the input text prompt and generate a text embedding vector representing the semantic information of the text prompt. The text prompt includes disease-related prompt text, such as lesion size, number, shape, etc.

[0037] In the technical solution of the present embodiment, a CLIP text encoder is used to generate semantic embedding. However, CLIP is not the only option, and MedCLIP, BioBERT, and other medical specialized models can be used as alternatives. Through the above-mentioned text model substitution scheme, the semantic modeling capability can be further enhanced, especially in the medical professional context, improving the explainability and accuracy of the system.

[0038] It should be noted that the specific structure, content, and input method of the text prompt can be replaced and extended to adapt to different medical image task requirements.

[0039] For example, a natural language prompt can be a complete natural language sentence, such as "segment the intestinal polyp region" or "detect the left ventricular lesion boundary". This type of prompt allows doctors to directly describe the task goal and has good explainability.

[0040] Keyword prompt: input several keywords to quickly specify the target class or morphological features, reducing the input complexity.

[0041] Structured hints: Input semantic labels in JSON or XML format, such as {“organ”: “liver”,“lesion_type”: “tumor”, “shape”: “irregular”}, to facilitate integration with hospital information systems (HIS) or electronic medical records (EMR).

[0042] Through the replacement and expansion of the above different forms of hints, the present application can adapt to diversified medical application scenarios and improve the flexibility and robustness of semantic guidance.

[0043] In an embodiment, the semantic guidance module is configured to generate affine modulation parameters using text embedding vectors and ; In an embodiment, the feature modulation module is configured to perform semantic modulation on image features of each layer of the image encoder output using affine modulation parameters, achieving cross-modal semantic guidance and better focusing on areas with specific clinical semantics.

[0044] The decoder is configured to decode the semantically modulated features step by step, recover the spatial resolution through skip connection and up-sampling, and obtain the first segmentation result.

[0045] In an embodiment, the student model is similar in structure to the teacher model but does not receive text hints and only inputs the decoder after extracting features through the image encoder to obtain the segmentation result. In the inference stage, the student model is completely independent of the text input and has efficient and practical deployment capabilities.

[0046] Specifically, the student model includes an image encoder and a decoder; the image encoder extracts features from the input medical image; The decoder decodes the extracted features to obtain the second segmentation result.

[0047] If the text hint is missing, a text hint is automatically generated.

[0048] Specifically, the step of automatically generating a text hint is: Input the binary mask image into a processing class, add a text label to each picture according to the number of connected domains, the total pixel area and white pixel area of each connected domain, and the circularity of the connected domain; The image information is combined into a structured text prompt through a text label splicing program. For example, a mask image contains two connected domains, one of which occupies less than 25% of the entire image and has a circularity greater than 0.85, and the other of which occupies more than 25% of the entire image and has a circularity greater than 0.85, and the text prompt is generated as: "Two: small and regular; large and regular." Thus, for data lacking a text prompt, the mask image of a data set (for example, a colon polyp data set) is processed through the above-mentioned automatic analysis text description generation step, and a structured text description is generated by analyzing the target lesions in the mask, i.e., the white connected domain information (such as the number, size, and shape). For example, the colon polyp data set is a black and white binary mask image.

[0049] In an embodiment, the text description is encoded into a fixed-dimension (512-dimension) embedding vector by the text encoder of CLIP, representing the semantic information of the text prompt. These text vectors will be further converted into affine modulation parameters through a designed semantic guidance mechanism, which will act on the image features of each layer of the decoder to realize image-text feature fusion and semantic guidance segmentation.

[0050] In an embodiment, the semantic guidance module adopts a multi-layer perceptron, which generates modulation parameters matching the number of visual feature channels according to the text embedding vector, which are used for affine transformation of feature modulation of feature maps of different scales in the image encoder.

[0051] In the feature modulation module, text prompt information is introduced into the feature space of different scales to adjust the semantic response of the visual feature map and modulate the feature fusion.

[0052] Specifically, the text embedding vector is first passed through a multi-layer perceptron (MLP) to generate modulation parameters matching the number of visual feature channels and Each level corresponds to a set of modulation parameters. The number of channels of each layer of features is different, while the dimension of the text embedding vector is fixed, and variable parameters are needed for adjustment. The modulation parameters are used for affine transformation of feature modulation of feature maps of different scales in the encoder. To realize hierarchical perception, a variable channel dimension is set inside the perceptron to facilitate the mapping of text features to modulation parameters matching the number of channels of the corresponding feature map at different levels.

[0053] The multi-layer perceptron consists of two fully connected layers and an activation function: The first linear layer realizes feature dimension upgrading, which is used to enhance the non-linear mapping ability; The second linear layer outputs the modulation parameters.

[0054] The text prompt information is introduced in the feature space of different scales to adjust the semantic response of the visual feature map. The modulation operation is performed in the form of feature fusion as follows:

[0055] wherein, represents the input image feature, represents the modulated feature map, and are the scale and offset parameters calculated by the text feature of the first m layer, and have the same spatial dimension.

[0056] The mechanism has good modularity and scalability, and can be flexibly embedded in different decoding stages. In the modulation process, the spatial structure information of the original image is retained, and the full participation of the text semantics in each layer of the decoder is ensured, so as to guide the model to focus on the semantic related areas and improve the accuracy and interpretability of the segmentation result.

[0057] During training, a semantic distillation strategy is adopted, and the teacher model directly uses binary cross entropy as the training loss function.

[0058] In order to realize the learning of the semantic ability of the student model to the teacher model, the student model can obtain similar segmentation performance to the teacher model in the inference stage without relying on the text prompt, and a hybrid supervision scheme is designed in the embodiment: Total loss is composed of output distillation loss and feature distillation loss, which jointly optimize the performance and generalization ability of the student model.

[0059] wherein , is a hyperparameter for balancing the influence weight.

[0060]

[0061]

[0062] wherein, is the output prediction of the student model, y is the real label mask, is the output prediction of the teacher model, are the third and fourth layer features of the student model, respectively. , are the third and fourth layer features of the teacher model, respectively.

[0063] Hard label supervision loss The student model is trained by a binary cross-entropy loss, directly uses the artificial annotation mask, and provides basic task supervision; the soft label distillation loss includes a distillation loss between the student output and the teacher output , a feature distillation loss between the student and the teacher in the intermediate layer feature representation , and the student model is encouraged to simulate the behavior of the teacher model in the output layer by measuring the difference by mean square error, so as to realize all-round knowledge transfer of the student from the output and the feature layer.

[0064] In the embodiment, by introducing a semantic guidance module, using a teacher-student structure and a multi-scale distillation loss, deep fusion of image features and clinical semantics is realized, and the segmentation accuracy and the generalization performance of the model are improved; by using a semantic distillation mechanism, the dependence on text prompts in the inference stage is avoided, and the practicability of the method is improved; the compatibility of the text prompts is maintained, diversified clinical descriptions can be introduced in the training, personalized medical treatment is supported, an adjustable and semantically controllable segmentation strategy is realized, and the performance of the model can be kept stable in different scenes; the semantic guidance module and the feature regulation module can be seamlessly embedded in mainstream U-Net, DeepLab and other structures, and have good compatibility and expansibility.

[0065] Table 1 Experimental results of Kvasir-SEG dataset

[0066] As shown in Table 1, compared with the traditional U-Net model, the prompt learning method proposed in the present application shows significant improvement on the public Kvasir colon polyp segmentation dataset: the Dice coefficient is increased by about 3.69%, and the mIoU is increased by about 4.45%.

[0067] The technical scheme of the embodiment uses a teacher-student network structure, combines a semantic guidance module and a semantic distillation strategy, introduces text prompts to realize semantic enhancement in the training process, eliminates the dependence on text in the inference stage through a distillation mechanism, designs an adjustable feature injection mechanism, realizes on-demand control of image segmentation, and solves the problems of insufficient precision, limited semantic expression capability and limited clinical application of the prior art. The on-demand control means that the text prompt structure content corresponding to the true value label of different datasets is different.

[0068] Embodiment two The purpose of the embodiment is to provide a computer device, including a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0069] Embodiment three The purpose of the embodiment is to provide a computer readable storage medium.

[0070] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.

[0071] Embodiment four The purpose of this embodiment is to provide a medical image segmentation system based on prompt learning, comprising: A data acquisition module configured to acquire a medical image and a text prompt; A model processing module configured to input the medical image into a teacher model and a student model respectively, and input the text prompt into the teacher model; The teacher model forms a semantic guide based on the text prompt, modulates the features of the medical image through the semantic guide, and outputs a first segmentation result; The student model processes the received medical image to obtain a second segmentation result; A parameter updating module configured to update the student model parameters using hybrid supervision for the first segmentation result and the second segmentation result; A segmentation module configured to input the medical image to be segmented into the student model after updating the parameters to obtain a generated segmentation result.

[0072] The overall technical solution of the present application can not only be applied to medical image segmentation tasks, but also can be extended to various medical image analysis scenarios, such as lesion detection, organ segmentation, disease classification, multi-task joint learning, etc.

[0073] Embodiment five The purpose of this embodiment is to provide a computer program product containing instructions which, when run on a computer, cause the computer to perform the methods and functions involved in any of the above embodiments. The steps and methods involved in the above embodiments correspond to Embodiment One, and the specific implementation can refer to the relevant description part of Embodiment One. The term "computer readable storage medium" should be understood to include a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium that can store, encode or carry instruction sets for execution by a processor and cause the processor to perform any method in the present application.

[0074] Those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computer device, alternatively, they can be realized by program code executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0075] The above describes the specific embodiments of the present application in conjunction with the drawings, but is not intended to limit the scope of protection of the present application. It should be understood by those skilled in the art that various modifications or variations can be made to the technical solutions of the present application without creative efforts, and still fall within the scope of protection of the present application.

Claims

1. A medical image segmentation method based on cue learning, characterized in that, include: Acquire medical images and text prompts; The medical images are input into the teacher model and the student model respectively, and the text prompts are input into the teacher model; The teacher model forms semantic guidance based on text prompts, modulates the features of the medical image through semantic guidance, and outputs the first segmentation result; The student model processes the received medical images to obtain a second segmentation result; The student model parameters are updated using hybrid supervision based on the first and second segmentation results; The medical image to be segmented is input into the student model with updated parameters to obtain the segmentation result.

2. The medical image segmentation method based on cue learning as described in claim 1, characterized in that, The teacher model includes: a text encoder, a semantic guidance module, an image encoder, a feature modulation module, and a decoder; The image encoder is configured to: extract features from the input medical image to obtain multi-layered features, and gradually abstract from low-level edges to high-level semantics; The text encoder is configured to: encode the input text prompt and generate a text embedding vector representing the semantic information of the text prompt; The semantic guidance module is configured to generate affine modulation parameters using text embedding vectors; The feature modulation module is configured to: for each layer of features output by the image encoder, use affine modulation parameters to perform semantic modulation on the image features of the corresponding layer, realize cross-modal semantic guidance, and focus on regions with specific semantics; The decoder is configured to: progressively decode the semantically modulated features, restore the spatial resolution through skip connections and upsampling, and obtain the first segmentation result.

3. The medical image segmentation method based on cue learning as described in claim 1, characterized in that, If a text hint is missing, it will be automatically generated.

4. The medical image segmentation method based on cue learning as described in claim 3, characterized in that, The steps for automatically generating text prompts are as follows: The binary mask image is input into a processing class, and text labels are added to each image based on the number of connected components, the total pixel area and white pixel area of ​​each connected component, and the circularity of the connected components. The image information is combined into structured text prompts through a text tag splicing program.

5. The medical image segmentation method based on cue learning as described in claim 2, characterized in that, The semantic guidance module employs a multilayer perceptron, which generates modulation parameters that match the number of visual feature channels based on the text embedding vector. These parameters are used to perform affine transformation feature modulation on feature maps of different scales in the image encoder.

6. The medical image segmentation method based on cue learning as described in claim 1, characterized in that, The hybrid supervision package uses a total loss consisting of output distillation loss and feature distillation loss, which together optimize the performance and generalization ability of the student model.

7. A medical image segmentation system based on cue learning, characterized in that, include: The data acquisition module is configured to acquire medical images and text prompts; The model processing module is configured such that the medical images are input to both the teacher model and the student model, and the text prompts are input to the teacher model. The teacher model forms semantic guidance based on text prompts, modulates the features of the medical image through semantic guidance, and outputs the first segmentation result; The student model processes the received medical images to obtain a second segmentation result; The parameter update module is configured to update the student model parameters using hybrid supervision based on the first and second segmentation results. The segmentation module is configured to input the medical image to be segmented into the updated student model to obtain the segmentation result.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-6 above.

Images