Universal medical image segmentation method based on multi-modal collaborative contrast enhancement

Abstract

The invention relates to a universal medical image segmentation method based on multi-modal collaborative contrast enhancement, belongs to the technical field of artificial intelligence, and aims to obtain rich semantic information through a multi-modal collaborative contrast enhancement model, and generate a high-quality segmentation result in combination with test prompt fine tuning, segmentation prompt optimization and fine tuning segmentation model strategies. Through test prompt fine tuning, the attention capability of the model to a target area is significantly enhanced, interference to irrelevant areas is reduced, and a preliminary segmentation prompt is generated. And a high-quality segmentation result is obtained by further combining historical data information to optimize a preliminary segmentation prompt. Meanwhile, the segmentation model is finely adjusted, unsupervised learning is used for carrying out consistency feature analysis on test data, the perception ability of the model on data distribution changes is enhanced, and the generalization performance is improved. According to the method, specific disease type information or dependent label information does not need to be known in advance, the transmitted medical image can be directly and efficiently segmented, and the universality and applicability of the method are greatly improved.

Description

Technical Field

[0001] The invention relates to a universal medical image segmentation method based on multimodal collaborative contrast enhancement, and belongs to the technical field of artificial intelligence. Background Art

[0002] Medical image segmentation has a wide range of applications in the field of medical imaging and is a key step in analyzing and diagnosing specific diseases. Through image segmentation, doctors can accurately identify the region of interest (ROI) in the sample, which is the basis of many medical analysis tasks. Accurate segmentation not only helps doctors better understand the morphology and location of lesions, but also assists in formulating more accurate treatment plans. In practical applications, semantic-based image segmentation can detect and separate biological structures and quantify their morphological characteristics through pixel-level classification tasks. This technology significantly improves the accuracy and speed of disease detection, supports earlier and more accurate medical intervention, and therefore plays a vital role in medical diagnosis and treatment.

[0003] Existing medical image segmentation methods mainly rely on deep learning technology, especially models based on Convolutional Neural Network (CNN) and Transformer architecture. These methods have shown remarkable results in processing two-dimensional and three-dimensional medical imaging data. With its powerful feature extraction capabilities, convolutional neural networks have become a standard tool for medical image segmentation, which can effectively identify and segment various anatomical structures. The introduction of the Transformer model has further enhanced the model's ability to process complex image data, especially in capturing long-distance dependencies and contextual information.

[0004] However, despite the excellent performance of these techniques in specific medical scenarios, their applicability still has obvious limitations. Existing methods are usually designed for specific diseases or specific types of medical images, such as lung cancer detection, brain tumor segmentation, or retinal image analysis. This targeted design makes them less adaptable when facing a wider range of medical scenarios and difficult to meet diverse application needs. Although large models, such as the Contrastive Language-Image Pre-training (CLIP) and the Segment Anything Model (SAM), have brought new hope for general segmentation, these models usually rely on a large amount of domain-specific annotated data. However, in actual clinical applications, annotated data is often scarce, especially when dealing with out-of-distribution (OOD) data that exceeds the distribution of training data, the performance of existing methods is still unsatisfactory. In the real world, the patient's condition and imaging data may be significantly different from the model's training data, and this difference often leads to inaccurate or even invalid model prediction results.

[0005] Therefore, although deep learning-based segmentation methods have made significant progress in specific medical applications, their versatility and adaptability to real-world data are still insufficient. To address these problems, it is urgent to develop a more general and adaptable medical image segmentation method that can perform well in different diseases, different image types, and complex data with OOD characteristics. Summary of the invention

[0006] In view of the deficiencies of the above-mentioned prior art, the present invention provides a universal medical image segmentation method based on multimodal collaborative contrast enhancement, which can effectively cope with a variety of medical segmentation tasks. Specifically, the present invention designs a prompt fine-tuning strategy during testing, using the CLIP model to supplement and capture semantic information for unknown test data. In addition, the present invention also proposes a prompt generation strategy based on historical information optimization, aiming to avoid prediction instability caused by single samples with large distribution differences. Finally, a strategy for fine-tuning the SAM model is designed to further enhance its generalization ability under different data distributions.

[0007] Terminology explanation:

[0008] Language-Image Multimodal Contrast Enhancement Model (CLIP): CLIP is a multimodal collaborative contrast model based on aligned text and images, including a text encoder and a visual encoder. It builds the similarity between the text description (text) and the corresponding image (vision) in a high-dimensional space by simultaneously learning the representation of the two, so as to effectively capture the semantic correspondence between text and image. This model can not only handle traditional visual tasks, but also understand and match images and texts cross-modally, providing support for a variety of medical segmentation application scenarios.

[0009] Segment Everything Model (SAM): SAM is a powerful image segmentation model that includes a visual encoder, a prompt encoder, and a decoder. It performs image segmentation tasks by accepting input segmentation prompts, and can flexibly segment the target area in the image according to different prompts. The SAM model is highly adaptable and flexible, and can handle various types of image segmentation tasks. It is an advanced tool for achieving general image segmentation.

[0010] Test-time Adaptation (TTA): Test-time adaptation is a technique that allows a model to quickly adjust itself when faced with test data. By adapting the model to new data distribution changes without changing the original training model, TTA improves the generalization ability and prediction accuracy of the model in practical applications. This method is particularly suitable for enhancing the performance of the model when there is a difference in the distribution of training data and test data.

[0011] The technical solution of the present invention is as follows:

[0012] A general medical image segmentation method based on multimodal collaborative contrast enhancement comprises the following steps:

[0013] (1) Semantic information generation: Generate corresponding text prompts for the input medical image samples, including positive text t p and negative text t n , these text prompts will serve as guidance information for the medical image segmentation process. The generated text prompts analyze the image content and extract the key features in the image, thereby providing guidance for subsequent segmentation. For example, in a general medical image segmentation task, we can generate positive text prompts for the target area (such as a tumor), such as generating text describing "focusing on abnormal tissue areas (such as tumors, rather than surrounding healthy tissues)" through a large language model (such as the interface of public models such as GPT); while negative text prompts can generate descriptions such as "excluding the target area and focusing on other normal tissues or non-abnormal areas (such as surrounding healthy tissues)", thereby more clearly defining the key and non-key areas of the segmentation task;

[0014] (2) Visual prompt generation: Generate a unique visual prompt v for each input medical image sample p As auxiliary information to guide the segmentation process, visual cues not only rely on the image itself, but also combine the knowledge learned in advance by the model to ensure that the cues can accurately reflect the important features in the image; in the specific implementation, the low-frequency information of the model is extracted through the fast Fourier transform (FFT), and then the visual cues are generated based on the extracted low-frequency information. In addition, the cues are learnable parameters that are adjusted through the loss function. Low-frequency information can effectively reflect the global features of the image. These features are used to guide the model to focus on the key features of the target area when generating visual cues, thereby further improving the segmentation accuracy;

[0015] (3) Test cue fine-tuning: After generating the visual cue, CLIP adjusts the visual cue to make it more consistent with the characteristics of the target text cue by comparing the similarity between the image and the text cue as the loss function (the cue is optimized by the loss function and the cue is a learnable parameter). The purpose of this step is to ensure that the generated visual cue can effectively guide CLIP to capture the key areas in the image;

[0016] (4) Segmentation feature map generation: The corresponding feature map is generated through the forward propagation of CLIP to display the different feature information in the image. At the same time, the gradient map is calculated through back propagation. The gradient map reflects the contribution of each pixel to a specific category. The feature map after forward propagation and the gradient map after back propagation are combined to highlight the area in the image that can best activate a specific category.

[0017] (5) Segmentation region optimization: Conditional Random Fields (CRF) is used to optimize the generated feature map to generate a segmentation map, which is then input into SAM as a segmentation hint. By considering the adjacent relationship between pixels, adjacent pixels are encouraged to have the same label, thereby enhancing spatial consistency.

[0018] (6) Segmentation hint generation and optimization: For a single medical image sample, a preliminary segmentation hint is generated through step (5), and a segmentation hint knowledge base is built based on visual hints. The segmentation hint of each medical image sample is not only derived from itself, but also combined with historical information to ensure the stability and reliability of the segmentation hint and avoid the instability caused by a single OOD sample.

[0019] (7) SAM fine-tuning: Perform data augmentation on the test data to optimize the distance between the original sample features and the enhanced aggregated features. Use unsupervised learning methods to extract robust features so that the model can better adapt to the complexity of the test data. Finally, generate high-precision segmentation results through the fine-tuned SAM model.

[0020] Preferably, in step (2), a visual cue v is generated p The process is:

[0021] For the input medical image sample v, apply the fast Fourier transform (FFT) to obtain its unique amplitude and phase:

[0022] amp,pha = FFT(v) (1)

[0023] In the above formula, amp represents amplitude and pha represents phase;

[0024] The amplitude map is shifted to move the DC component (zero frequency component) of the two-dimensional spectrum map to the center of the image, so that the low-frequency information is concentrated in the central area. Subsequently, an adaptive binary matrix M∈{0,1} is used to control the proportion of low-frequency and high-frequency components. Each element of the binary matrix M is usually between [0,1], indicating the weighting degree of the frequency component at that position: the closer the value is to 1, the more the frequency component is retained, and the closer the value is to 0, the more the frequency component is suppressed. The binary matrix M is continuously adjusted and optimized through back propagation during the SAM model training process to ensure that the visual cues can best capture the characteristics of the target area.

[0025] Then, the amplitude amp is updated through the binary matrix M to obtain amp f , the update process is amp f =amp×M, updated amplitude amp f And the phase pha is recombined according to formula (2) to generate the optimized frequency information:

[0026]

[0027] iFFT() means inverse Fourier transform;

[0028] Finally, the frequency domain information is converted back to the spatial domain (RGB) of the image through inverse Fourier transform to generate the optimized image , as the input of CLIP; through this adaptive visual cue generation method, the high and low frequency components of the frequency domain information can be flexibly adjusted. The learning and optimization process of the matrix M enables the model to adaptively allocate high and low frequency information to capture important features in the image in the best way.

[0029] Visual prompt p It reflects some key frequency domain information of the input image v (such as image contours, details, etc.), and then uses this key information v p (The key information is called visual enhancement) The original image v is "enhanced" (optimized) to obtain the enhanced image (The original image v has no frequency domain information, and these frequency domain information are added through visual enhancement.) Calculate the frequency domain information v of the input image v p , and then the two are fused to get the enhanced image

[0030] Preferably, in step (3), the test prompt fine-tuning is implemented as follows:

[0031] Based on the positive text prompt t generated in step (1) p and negative text t n , via CLIP's text encoder (textencoder) Get the corresponding embedding vectors for positive and negative text prompts and At the same time, the input image passes through the CLIP vision encoder Get the corresponding embedding vector e v :

[0032]

[0033] For each sample vector, calculate the cosine similarity sim(v,t) between it and each text vector:

[0034] sim(v,t)=cosine(e v ,e t ) (4)

[0035] in cosine() represents the cosine distance, and the image embedding vector e is calculated by formula (4) v The cosine distance of the embedding vector of the positive sample and the negative sample; the cosine distance reflects the correlation between the input image and the input text. The greater the similarity, the higher the correlation between the two, and vice versa. In order to enhance CLIP's attention to the target area, the distance to the positive sample is further reduced, while the distance to the negative text is increased:

[0036]

[0037] Where K is the number of positive texts, J is the number of negative samples, is the kth positive textual cue corresponding to image v, is the jth positive text hint corresponding to image v; through the loss function The back-propagation updates the visual cue M, making CLIP more focused on the target area and avoiding misidentification of other irrelevant areas. At the same time, the CLIP model is kept frozen and only the visual cue is updated, so that the original ability of CLIP will not be destroyed due to fine-tuning and rich pre-training knowledge is retained.

[0038] Preferably, in step (4), the implementation process of generating the segmentation feature map is:

[0039] First, the image Input into CLIP to perform forward propagation to obtain the feature map, and then based on the loss function of step (3) Perform back propagation to obtain the gradient map, and use the gradient map to generate the class activation map;

[0040] The feature map is obtained using the Class Activation Map (CAM); the features of the Layer Normalization (LN) of the last layer of the CLIP visual encoder are weighted summed to generate a feature map representing a specific category; by calculating the contribution of each element to the feature category in the image, this feature map can highlight the areas most relevant to the category:

[0041]

[0042] where f k (x,y) is the activation value of the kth channel at position (x,y), w k It is the fully connected weight related to the category; CAM(x,y) represents the activation intensity of a category in the input image corresponding to the position (x,y), reflecting the importance of the position in predicting the category; the larger the CAM(x,y) value, the more the model pays attention to this area and the more important the position is.

[0043] The present invention forward-propagates the input image through the model, calculates the score of the target category, and extracts the feature map of the last convolutional layer; then, the gradient of the target category score to the convolutional feature map is calculated through back-propagation as the gradient map, and the global average value of the gradient in the spatial dimension is extracted to obtain the importance weight of each channel; then, these weights are used to perform weighted summation on each channel of the convolutional feature map to generate an initial category activation map; finally, the generated activation map is upsampled to the same size as the input image, and a color heat map is superimposed for visualization, thereby intuitively displaying the model's focus area on the target category.

[0044] Preferably, in step (5), the implementation process of segmentation region optimization is:

[0045] After obtaining the feature map CAM in step (4), the conditional random field (CRF) is used for optimization to enhance the clarity and spatial consistency of the boundary; the energy function is defined as:

[0046]

[0047] in is the single-point potential energy, representing the pixel x i The ability relationship between the CAM and its category label, the CAM value is used as the confidence of the category to which the pixel belongs, that is, CAM(x i ) is higher, the pixel x i The more likely it is to fall into this particular category; is the dual potential energy, indicating that the adjacent pixels x i and x j This potential energy is calculated by the Gaussian kernel function, which takes into account the spatial distance and color similarity between pixels, thereby encouraging spatially adjacent pixels to have the same label; by minimizing the energy function E(X), the optimized CRF segmentation result can be obtained:

[0048] M crf =argmin X E(X) (8).

[0049] Preferably, in step (6), the implementation process of segmentation hint generation and optimization is:

[0050] In order to further avoid the problem that the generated segmentation area is inaccurate due to a single sample with large distribution difference, which in turn affects the instability of the segmentation prompt, a segmentation prompt knowledge base B is built based on visual prompts to store N key-value pairs. The segmentation hint knowledge base follows the first-in-first-out principle, where k is the low-frequency feature calculated for each sample based on the visual hint M, and q is the CRF segmentation result M calculated for the corresponding sample using formula (8) crf ;k n Represents the low-frequency features of the nth sample, q n Represents the segmentation result corresponding to the nth sample; for each sample v, the similarity is calculated based on its low-frequency vector and k of each sample in the knowledge base:

[0051]

[0052] Where ‖·‖ is the Euclidean norm; the U most similar samples are selected from the similarity calculated by formula (9) as the support set of the current sample, and then the exponential sliding average method is used to update step (5) to obtain the optimized segmentation result M crf :

[0053]

[0054] Where α is a hyperparameter, is the normalized similarity measure, q u represents the segmentation result corresponding to the u-th sample; the final optimized segmentation result M crf It is input into SAM as the segmentation hint for the current sample.

[0055] Preferably, α is set to 0.99.

[0056] Preferably, the implementation process of SAM fine-tuning in step (7) is:

[0057] For each test sample v, A enhanced versions are generated through data augmentation. Data augmentation can generally be achieved by rotating, enlarging, adding color disturbance, etc. The ath image after augmentation is The average feature of all enhanced versions is expressed as: Through optimization, the difference between the test sample characteristics and the more stable average characteristics is minimized, and the optimization goal is:

[0058]

[0059] Where D is the feature dimension of SAM’s visual encoder IE; IE represents SAM’s visual encoder;

[0060] The final segmentation result y is defined as:

[0061]

[0062] SAM() stands for Segmentation Everything Model; this process helps the Segmentation Everything Model (SAM) maintain its characteristics when the input data changes, thereby improving its performance on OOD samples. In particular, this optimization only involves updating the visual encoder of SAM.

[0063] For any details not provided in the present invention, please refer to the prior art.

[0064] The beneficial effects of the present invention are:

[0065] 1. The present invention proposes a complete solution for general medical image segmentation. By introducing the methods of test prompt fine-tuning, segmentation prompt optimization and SAM fine-tuning, the test data can be directly segmented without the need for pre-training with labeled data. Traditional medical image segmentation methods are mostly targeted at specific tasks and can only be applied to specific types of segmentation tasks at a time, such as eye recognition or lung image segmentation. The segmentation knowledge of eye blood vessels cannot be directly applied to the segmentation task of lung images, which means that after completing the eye recognition task, the lung data must be re-trained to adapt to the new segmentation task. On the one hand, it is difficult to obtain medical data, and on the other hand, knowledge between different tasks cannot be shared.

[0066] 2. The present invention effectively solves the problem of poor quality of generated segmentation prompts due to lack of semantic information in test data by fine-tuning test prompts, optimizing segmentation regions, and optimizing segmentation prompts. Specifically, by combining the language-image multimodal contrast enhancement model and introducing visual prompt fine-tuning, CLIP pays more attention to the target area, thereby generating high-quality segmentation prompts. At the same time, with the help of historical information, improvements are made to eventually generate stable segmentation prompts, avoiding the problem of unstable prediction caused by samples with large distribution differences.

[0067] 3. By fine-tuning SAM and using unsupervised learning to calculate consistent features of test data, the ability to perceive changes in data distribution is enhanced, further improving the generalization ability of the model, thereby improving the segmentation results. In addition, by fine-tuning the visual encoder, the pre-training knowledge is retained as much as possible to ensure that the model does not lose the original effective features while adapting to new data. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0069] Figure 1 It is a flowchart of the universal medical image segmentation method based on multimodal collaborative contrast enhancement of the present invention. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings in the implementation of this specification, but are not limited to this. Anything not fully described in the present invention shall be based on the conventional technology in the art.

[0071] Example 1

[0072] A general medical image segmentation method based on multimodal collaborative contrast enhancement is used to segment medical images of glaucoma, comprising the following steps:

[0073] (1) Semantic information generation: Use language models such as GPT to generate semantic information. The input requirement is: "I am implementing a glaucoma segmentation task. Please generate some positive text and negative text. The positive text should emphasize the cup-shaped retina, and the negative text should emphasize the blood vessels rather than the retina." The text generated in this way can provide semantic guidance for subsequent segmentation tasks.

[0074] (2) Visual cue generation: Apply Fourier transform to each eyeball photo and initialize the image into low-frequency components. The low-frequency components reveal the basic shape of the eyeball image and are relatively stable. Define a learnable matrix as a visual cue and optimize the matrix through training so that it can better capture the key areas in the eyeball structure.

[0075] (3) Test Tips Fine-tuning: When using the CLIP model to extract semantic information, it may be inevitable to emphasize irrelevant blood vessels. This step optimizes the similarity between the input image and the positive text and increases its distance from the negative text, so that CLIP pays more attention to the target area, such as the cup-basin area in the retina, rather than irrelevant blood vessels. Retinal images usually contain a large number of blood vessels, so this fine-tuning allows the model to focus on areas related to glaucoma.

[0076] (4) Segmentation feature map generation: The feature map is generated through the forward propagation of CLIP to display the information of different features in the glaucoma image; then, the gradient map is calculated through back propagation to reflect the contribution of each pixel to the target cup-and-basin area in the retina. Finally, the feature map calculated by forward propagation and the gradient map obtained by back propagation are combined to highlight the area in the image that can best activate the specific target cup-and-basin area.

[0077] (5) Segmentation region optimization: For the cup-and-basin region in glaucoma images, the generated cup-and-basin feature map is optimized by applying conditional random fields to generate a more accurate segmentation map. The conditional random field uses the adjacent relationship between pixels and the edge information of the image to adjust the initial segmentation results, encouraging adjacent pixels to have the same label, thereby enhancing spatial consistency and segmentation boundary details. The optimized segmentation map is input into SAM as a segmentation hint to further extract and refine the segmentation results of the target cup-and-basin region in the retinal image, thereby improving the accuracy and reliability of the segmentation of glaucoma-related regions.

[0078] (6) Segmentation hint generation and optimization: When generating preliminary segmentation hints, more accurate segmentation results are obtained based on the optimization results of feature maps and conditional random fields. At the same time, in order to further improve the segmentation hints, an information knowledge base is constructed to store and manage the segmentation information of high-quality samples. Through this knowledge base, samples with similar structures (such as samples with close retinal positions) are analyzed and extracted so that their segmentation hints can remain consistent. This process also comprehensively utilizes the information of high-quality samples to reduce the negative impact on segmentation stability caused by irrelevant blood vessels or large differences in sample distribution, thereby improving the accuracy and robustness of segmentation of glaucoma-related areas.

[0079] (7) SAM fine-tuning: Data augmentation is performed on eye photos, such as adding color perturbations or rotations, to generate augmented data. By optimizing the consistency loss, the output before and after augmentation remains consistent, thereby improving the model's stability and adaptability to distribution changes.

[0080] (8) Segmentation result generation: The generated segmentation hints and test images are input into the SAM model to finally obtain accurate segmentation results.

[0081] Example 2

[0082] A general medical image segmentation method based on multimodal collaborative contrast enhancement is used to segment a medical image of the lung, comprising the following steps:

[0083] (1) Semantic information generation: Use language models such as GPT to generate semantic information. The input requirement is: "We are implementing the lung image segmentation task. Please generate some positive text and negative text. Among them, the positive text should focus on the lung area, and the negative text should focus on parts outside the lungs, such as the heart and ribs." The text generated in this way can provide semantic guidance for subsequent segmentation tasks.

[0084] (2) Visual cue generation: A Fourier transform is performed on each lung image, focusing on extracting low-frequency information to capture the general outline and shape of the lungs. Based on the low-frequency information, a learnable matrix is ​​constructed as a visual cue, and through optimization and adjustment, it can effectively highlight the key feature areas of the lungs while reducing interference from non-lung areas such as the heart and ribs.

[0085] (3) Test cue fine-tuning: When using the CLIP model to process lung images, in order to avoid the model focusing on non-target areas (such as the heart or ribs), this step adjusts the model's attention. Specifically, the matching degree between the input image and the positive semantic cue is optimized, and its correlation with the negative cue is suppressed, thereby ensuring that the model focuses on the lung area and minimizing false detections.

[0086] (4) Segmentation feature map generation: The feature map is generated through the forward propagation of CLIP to display the different feature information of the lungs in the image; then, the gradient map is calculated through back propagation to reflect the contribution of each pixel to the lung area (including the lung lobe and boundary). Finally, the features calculated by the forward propagation are combined.

[0087] (5) Segmentation region optimization: For the lung region in the image, the initial lung feature map generated is optimized by applying conditional random fields to generate more accurate segmentation results. The initial segmentation results are adjusted by using the adjacent relationship between pixels and the edge information of the image to encourage adjacent pixels to have the same label, thereby enhancing spatial consistency and refining the segmentation boundaries. The optimized lung segmentation map is input into SAM as a segmentation hint to further extract and refine the segmentation results of the target lung region in the image, thereby improving the accuracy and reliability of the segmentation.

[0088] (6) Segmentation hint generation and optimization: When generating preliminary lung segmentation hints, more accurate segmentation results are obtained based on the optimization results of feature maps and conditional random fields. At the same time, in order to further improve the segmentation hints, an information knowledge base is constructed to store and manage the segmentation information of high-quality samples. Through this knowledge base, samples with similar structures (such as images with similar lung morphology and position) are analyzed and extracted so that the segmentation hints can remain consistent. This process also uses the information of high-quality samples to reduce the instability of segmentation results caused by irrelevant structures (such as ribs, heart shadows) or large differences in sample distribution, thereby improving the accuracy and robustness of lung area segmentation.

[0089] (7) SAM fine-tuning: Further, by performing data augmentation on lung images (such as introducing different angle rotations, noise perturbations, etc.), the model’s adaptability to diverse inputs is enhanced. The consistency loss function is used for optimization so that the model outputs stable and consistent under different augmentation conditions, thereby improving its generalization ability and sensitivity to lung structure.

[0090] (8) Segmentation result generation: Combining the generated segmentation hints with the enhanced lung image, the system uses the SAM model to generate the final segmentation result. This result has high accuracy and robustness, and can provide a reliable basis for the diagnosis of lung diseases.

[0091] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A general medical image segmentation method based on multimodal collaborative contrast enhancement, characterized in that: The steps include: (1) Semantic information generation: Generate corresponding text prompts for the input medical image samples, including positive text t p and negative text t n ; (2) Visual cue generation: Generate a unique visual cue v for each input medical image sample p As auxiliary information to guide the segmentation process, visual cues not only rely on the image itself, but also combine the knowledge learned in advance by the model to ensure that the cues can accurately reflect the important features in the image; (3) Test cue fine-tuning: After generating the visual cue, CLIP adjusts the visual cue to make it more consistent with the characteristics of the target text cue by comparing the similarity between the image and the text cue as a loss function; (4) Segmentation feature map generation: The corresponding feature map is generated through the forward propagation of CLIP to display the different feature information in the image. At the same time, the gradient map is calculated through back propagation. The gradient map reflects the contribution of each pixel to a specific category. The feature map after forward propagation and the gradient map after back propagation are combined to highlight the area in the image that can best activate a specific category. (5) Segmentation region optimization: Apply conditional random fields to optimize the generated feature map to generate a segmentation map, which is input into SAM as a segmentation hint. By considering the adjacent relationship between pixels, adjacent pixels are encouraged to have the same label, thereby enhancing spatial consistency. (6) Segmentation hint generation and optimization: For a single medical image sample, generate preliminary segmentation hints through step (5), and build a segmentation hint knowledge base based on visual hints. The segmentation hints of each medical image sample are not only derived from itself, but also combined with historical information to ensure the stability and reliability of the segmentation hints. (7) SAM fine-tuning: Perform data augmentation on the test data to optimize the distance between the original sample features and the enhanced aggregated features. Use unsupervised learning methods to extract robust features so that the model can better adapt to the complexity of the test data. Finally, generate high-precision segmentation results through the fine-tuned SAM model.

2. The general medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 1, characterized in that: In step (2), generate visual cues v p The process is: For the input medical image sample v, apply the fast Fourier transform to obtain its unique amplitude and phase: amp,pha = FFT(v) (1) In the above formula, amp represents amplitude and pha represents phase; The amplitude map is shifted to move the DC component of the two-dimensional spectrum map to the center of the image, so that the low-frequency information is concentrated in the central area. Then, an adaptive binary matrix M∈{0,1} is used to control the proportion of low-frequency and high-frequency components. Each element of the binary matrix M is between [0,1], indicating the weighting degree of the frequency component at that position: the closer the value is to 1, the more the frequency component is retained, and the closer the value is to 0, the more the frequency component is suppressed. The binary matrix M is continuously adjusted and optimized through back propagation during the SAM model training process to ensure that the visual cues can best capture the characteristics of the target area. Then, the amplitude amp is updated through the binary matrix M to obtain amp f , the update process is amp f =amp×M, updated amplitude amp f And the phase pha is recombined according to formula (2) to generate the optimized frequency information: iFFT() means inverse Fourier transform; Finally, the frequency domain information is converted back to the spatial domain of the image through inverse Fourier transform to generate the optimized image As input to CLIP.

3. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 2, characterized in that: In step (3), the test prompt fine-tuning is implemented as follows: Based on the positive text prompt t generated in step (1) p and negative text t n , text encoder via CLIP Get the corresponding embedding vectors for positive and negative text prompts and At the same time, the input image passes through the CLIP visual encoder Get the corresponding embedding vector e v : For each sample vector, calculate the cosine similarity sim(v,t) between it and each text vector: sim(v,t)=cosine(e v ,e t ) (4) in cosine() represents the cosine distance, and the image embedding vector e is calculated by formula (4) v The cosine distance of the embedding vector of the positive sample and the negative sample; in order to enhance CLIP's attention to the target area, the distance to the positive sample is further reduced, while the distance to the negative text is increased: Where K is the number of positive texts, J is the number of negative samples, is the kth positive textual cue corresponding to image v, is the jth positive text hint corresponding to image v; through the loss function Back-propagation updates the visual cue M.

4. The general medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 3, characterized in that: In step (4), the implementation process of segmentation feature map generation is: First, the image Input into CLIP to perform forward propagation to obtain the feature map, and then based on the loss function of step (3) Perform back propagation to obtain the gradient map, and use the gradient map to generate the class activation map; Use the class activation map to get the feature map; perform weighted summation on the features of the normalized network of the last layer of the CLIP visual encoder to generate a feature map representing a specific category; calculate the contribution of each element to the feature category in the image: where f k (x,y) is the activation value of the kth channel at position (x,y), w k It is the fully connected weight related to the category; CAM(x,y) represents the activation intensity of a category in the input image corresponding to the position (x,y), reflecting the importance of the position for predicting the category.

5. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 4, characterized in that: In step (5), the implementation process of segmentation region optimization is as follows: After obtaining the feature map CAM in step (4), the conditional random field is used for optimization to enhance the clarity and spatial consistency of the boundary; the energy function is defined as: in is the single-point potential energy, representing the pixel x i The ability relationship between the CAM and its category label is used as the confidence of the category to which the pixel belongs; is the dual potential energy, indicating that the adjacent pixels x i and x j The interaction between them; by minimizing the energy function E(X), the optimized CRF segmentation result can be obtained: M crf =argmin X E(X) (8)。 6. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 5, characterized in that: In step (6), the implementation process of segmentation hint generation and optimization is as follows: Build segmentation hint knowledge base B based on visual hints to store N key-value pairs The segmentation hint knowledge base follows the first-in-first-out principle, where k is the low-frequency feature calculated for each sample based on the visual hint M, and q is the CRF segmentation result M calculated for the corresponding sample using formula (8) crf ;k n Represents the low-frequency features of the nth sample, q n Represents the segmentation result corresponding to the nth sample; for each sample v, the similarity is calculated based on its low-frequency vector and k of each sample in the knowledge base: Where ‖·‖ is the Euclidean norm; the U most similar samples are selected from the similarity calculated by formula (9) as the support set of the current sample, and then the exponential sliding average method is used to update step (5) to obtain the optimized segmentation result M crf : Where α is a hyperparameter, is the normalized similarity measure, q u represents the segmentation result corresponding to the u-th sample; the final optimized segmentation result M crf It is input into SAM as the segmentation hint for the current sample.

7. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 6, characterized in that: α is set to 0.

99.

8. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 6, characterized in that: The implementation process of SAM fine-tuning in step (7) is: For each test sample v, A enhanced versions are generated through data augmentation, and the ath image after augmentation is The average feature of all enhanced versions is expressed as: Through optimization, the difference between the test sample characteristics and the more stable average characteristics is minimized, and the optimization goal is: Where D is the feature dimension of SAM’s visual encoder IE; IE represents SAM’s visual encoder; The final segmentation result y is defined as: SAM() means split all models.

Images