A universal medical image segmentation method based on multimodal collaborative contrast enhancement

Abstract

The present invention relates to a universal medical image segmentation method based on multimodal collaborative contrast enhancement, which belongs to the field of artificial intelligence technology. Rich semantic information is obtained through a multimodal collaborative contrast enhancement model, and high-quality segmentation results are generated by combining test prompt fine-tuning, segmentation prompt optimization and fine-tuning segmentation model strategies. Through test prompt fine-tuning, the model's ability to focus on the target area is significantly enhanced, the interference with irrelevant areas is reduced, and preliminary segmentation prompts are generated. The preliminary segmentation prompts are further optimized in combination with historical data information to obtain high-quality segmentation results. At the same time, by fine-tuning the segmentation model and using unsupervised learning to perform consistency feature analysis on the test data, the model's ability to perceive changes in data distribution is enhanced, and the generalization performance is improved. This method does not require prior knowledge of specific disease type information or reliance on label information, and can directly and efficiently segment the incoming medical images, greatly improving the versatility and applicability of the method.

Description

Technical Field

[0001] The present invention relates to a universal medical image segmentation method based on multimodal collaborative contrast enhancement, and belongs to the technical field of artificial intelligence. Background Art

[0002] Medical image segmentation has a wide range of applications in the field of medical imaging and is a key step in analyzing and diagnosing specific diseases. Image segmentation allows doctors to accurately identify regions of interest (ROIs) within a sample, which is the foundation of many medical analysis tasks. Accurate segmentation not only helps doctors better understand the morphology and location of lesions, but also assists in formulating more precise treatment plans. In practical applications, semantic-based image segmentation can detect and separate biological structures through pixel-level classification tasks, and then quantify their morphological characteristics. This technology significantly improves the accuracy and speed of disease detection, supports earlier and more precise medical intervention, and therefore plays a vital role in medical diagnosis and treatment.

[0003] Existing medical image segmentation methods primarily rely on deep learning techniques, particularly models based on convolutional neural networks (CNNs) and the Transformer architecture. These methods have demonstrated remarkable results when processing two-dimensional and three-dimensional medical imaging data. Convolutional neural networks, with their powerful feature extraction capabilities, have become a standard tool for medical image segmentation, effectively identifying and segmenting various anatomical structures. The introduction of the Transformer model further enhances its ability to process complex image data, particularly in capturing long-range dependencies and contextual information.

[0004] However, despite the impressive performance of these techniques in specific medical scenarios, their applicability remains significantly limited. Existing methods are typically designed for specific diseases or specific types of medical images, such as lung cancer detection, brain tumor segmentation, or retinal image analysis. This targeted design makes them less adaptable to a wider range of medical scenarios and difficult to meet diverse application needs. While large models, such as the Contrastive Language-Image Pre-training (CLIP) and Segment Anything Model (SAM), have brought new hope for general segmentation, these models typically rely on large amounts of domain-specific annotated data. However, in real-world clinical applications, annotated data is often scarce, and existing methods still perform poorly when dealing with out-of-distribution (OOD) data. In the real world, a patient's condition and imaging data may differ significantly from the model's training data, often leading to inaccurate or even invalid model predictions.

[0005] Therefore, although deep learning-based segmentation methods have made significant progress in specific medical applications, their versatility and adaptability to real-world data are still insufficient. To address these problems, it is urgent to develop a more general and adaptable medical image segmentation method that can perform well in different diseases, different image types, and complex data with OOD characteristics. Summary of the Invention

[0006] In response to the above-mentioned deficiencies in the prior art, the present invention provides a universal medical image segmentation method based on multimodal collaborative contrast enhancement, which can effectively cope with a variety of medical segmentation tasks. Specifically, the present invention designs a test-time prompt fine-tuning strategy, which uses the CLIP model to supplement and capture semantic information for unknown test data. In addition, the present invention also proposes a prompt generation strategy based on historical information optimization, aiming to avoid prediction instability caused by samples with large individual distribution differences. Finally, a strategy for fine-tuning the SAM model is designed to further enhance its generalization ability under different data distributions.

[0007] Explanation of terms:

[0008] Language-Image Multimodal Contrast Enhancement (CLIP): CLIP is a multimodal collaborative contrast model based on aligned text and images, consisting of a text encoder and a visual encoder. It simultaneously learns representations of text descriptions (text) and corresponding images (vision), building similarities between them in a high-dimensional space, effectively capturing the semantic correspondence between text and images. This model not only handles traditional vision tasks but also understands and matches images and text across modalities, supporting diverse medical segmentation applications.

[0009] Segment Everything Model (SAM): SAM is a powerful image segmentation model consisting of a visual encoder, a prompt encoder, and a decoder. It performs image segmentation tasks by accepting segmentation prompts as input, flexibly segmenting the target region in the image based on the prompts. The SAM model is highly adaptable and flexible, capable of handling a wide range of image segmentation tasks, making it an advanced tool for general-purpose image segmentation.

[0010] Test-time Adaptation (TTA): Test-time adaptation is a technique that allows a model to quickly adjust itself to new data distributions without changing the original trained model, thereby improving the model's generalization and prediction accuracy in real-world applications. This approach is particularly useful for enhancing model performance when there are differences in the distribution of training and test data.

[0011] The technical solutions of the present invention are as follows:

[0012] A general medical image segmentation method based on multimodal collaborative contrast enhancement comprises the following steps:

[0013] (1) Semantic information generation: Generate corresponding text prompts for the input medical image samples, including positive text t p and negative text t n These text prompts will serve as guidance information for the medical image segmentation process. The generated text prompts analyze the image content and extract the key features in the image, thereby providing guidance for subsequent segmentation. For example, in a general medical image segmentation task, we can generate positive text prompts for the target area (such as a tumor), such as generating text describing "focusing on abnormal tissue areas (such as tumors, rather than surrounding healthy tissues)" through a large language model (such as the interface of public models such as GPT); while negative text prompts can generate descriptions such as "excluding the target area and focusing on other normal tissues or non-abnormal areas (such as surrounding healthy tissues)", thereby more clearly defining the key and non-key areas of the segmentation task;

[0014] (2) Visual prompt generation: Generate a unique visual prompt v for each input medical image sample p As auxiliary information to guide the segmentation process, visual cues not only rely on the image itself, but also combine the knowledge learned in advance by the model to ensure that the cues can accurately reflect the important features in the image. In the specific implementation, the low-frequency information of the model is extracted through the fast Fourier transform (FFT), and then the visual cues are generated based on the extracted low-frequency information. In addition, the cues are learnable parameters and are adjusted through the loss function. Low-frequency information can effectively reflect the global features of the image. These features are used to guide the model to focus on the key features of the target area when generating visual cues, thereby further improving the segmentation accuracy.

[0015] (3) Test cue fine-tuning: After generating the visual cue, CLIP adjusts the visual cue to make it more consistent with the characteristics of the target text cue by comparing the similarity between the image and the text cue as a loss function (the cue is optimized by the loss function and the cue is a learnable parameter). The purpose of this step is to ensure that the generated visual cue can effectively guide CLIP to capture the key areas in the image;

[0016] (4) Segmentation feature map generation: The corresponding feature map is generated through the forward propagation of CLIP to display the different feature information in the image. At the same time, the gradient map is calculated through back propagation. The gradient map reflects the contribution of each pixel to a specific category. Combining the feature map after forward propagation and the gradient map after back propagation, the area in the image that can most activate a specific category is highlighted;

[0017] (5) Segmentation region optimization: Conditional Random Fields (CRF) is applied to optimize the generated feature map to generate a segmentation map, which is then input into SAM as a segmentation hint. By considering the adjacent relationship between pixels, adjacent pixels are encouraged to have the same label, thereby enhancing spatial consistency.

[0018] (6) Segmentation hint generation and optimization: For a single medical image sample, preliminary segmentation hints are generated through step (5), and a segmentation hint knowledge base is built based on visual hints. The segmentation hints of each medical image sample are not only derived from itself, but also combined with historical information to ensure the stability and reliability of the segmentation hints and avoid the instability caused by a single OOD sample;

[0019] (7) SAM fine-tuning: Perform data enhancement on the test data to optimize the distance between the original sample features and the enhanced aggregated features. Use unsupervised learning methods to extract robust features so that the model can better adapt to the complexity of the test data. Finally, a high-precision segmentation result is generated through the fine-tuned SAM model.

[0020] Preferably, in step (2), a visual cue v is generated p The process is:

[0021] For the input medical image sample v, apply the fast Fourier transform (FFT) to obtain its unique amplitude and phase:

[0022] amp,pha = FFT(v) (1)

[0023] In the above formula, amp represents amplitude and pha represents phase;

[0024] The amplitude map is shifted to move the DC component (zero-frequency component) of the two-dimensional spectrum map to the center of the image, so that the low-frequency information is concentrated in the central area. Subsequently, an adaptive binary matrix M∈{0,1} is used to control the proportion of low-frequency and high-frequency components. Each element of the binary matrix M is usually between [0,1], indicating the degree of weighting of the frequency component at that position: the closer the value is to 1, the more the frequency component is retained, and the closer the value is to 0, the more the frequency component is suppressed. The binary matrix M is continuously adjusted and optimized through backpropagation during the SAM model training process to ensure that the visual cue can best capture the characteristics of the target area.

[0025] Then, the amplitude amp is updated by the binary matrix M to obtain amp f , the update process is amp f =amp×M, updated amplitude amp f And the phase pha is recombined according to formula (2) to generate the optimized frequency information:

[0026]

[0027] iFFT() means inverse Fourier transform;

[0028] Finally, the frequency domain information is converted back to the spatial domain (RGB) of the image through inverse Fourier transform to generate the optimized image , as the input of CLIP; through this adaptive visual cue generation method, the high and low frequency components of the frequency domain information can be flexibly adjusted. The learning and optimization process of the matrix M enables the model to adaptively allocate high and low frequency information to capture important features in the image in the best way.

[0029] Visual prompt p It reflects some key frequency domain information of the input image v (such as image contours, details, etc.), and then uses these key information v p (The key information is called visual enhancement) The original image v is "enhanced" (optimized) to obtain the enhanced image (The original image v has no frequency domain information, and these frequency domain information are added through visual enhancement.) Calculate the frequency domain information v for the input image v p , and then the two are fused to get the enhanced image

[0030] Preferably, in step (3), the test prompt fine-tuning is implemented as follows:

[0031] Based on the positive text prompt t generated in step (1) p and negative text t n , text encoder via CLIP Get the corresponding embedding vectors for positive and negative text prompts and At the same time, the input image passes through the CLIP vision encoder Get the corresponding embedding vector e v :

[0032]

[0033] For each sample vector, calculate the cosine similarity sim(v,t) between it and each text vector:

[0034] sim(v,t)=cosine(e v ,e t ) (4)

[0035] in cosine() represents the cosine distance, and the image embedding vector e is calculated by formula (4) v The cosine distance between the embedding vectors of the positive and negative samples reflects the correlation between the input image and the input text. The greater the similarity, the higher the correlation between the two, and vice versa. To enhance CLIP's focus on the target area, the distance to the positive sample is further reduced, while the distance to the negative text is increased:

[0036]

[0037] Where K is the number of positive texts, J is the number of negative samples, is the kth positive textual cue corresponding to image v, is the jth positive text hint corresponding to image v; through the loss function Back-propagation updates the visual cue M, making CLIP more focused on the target area and avoiding misidentification of other irrelevant areas. At the same time, the CLIP model is kept frozen and only the visual cue is updated, so that the original ability of CLIP will not be destroyed by fine-tuning and rich pre-training knowledge is retained.

[0038] Preferably, in step (4), the implementation process of generating the segmentation feature map is:

[0039] First, the image Input into CLIP to perform forward propagation to obtain the feature map, and then based on the loss function of step (3) Perform backpropagation to obtain the gradient map, and use the gradient map to generate the class activation map;

[0040] The class activation map (CAM) is used to obtain the feature map. The features of the layer normalization network (LN) of the last layer of the CLIP visual encoder are weighted and summed to generate a feature map representing a specific category. By calculating the contribution of each element to the feature category in the image, this feature map can highlight the areas most relevant to the category:

[0041]

[0042] where f k (x,y) is the activation value of the kth channel at position (x,y), w k It is the fully connected weight related to the category; CAM(x,y) represents the activation intensity of a category in the input image at position (x,y), reflecting the importance of the position in predicting the category; the larger the CAM(x,y) value, the more the model pays attention to this area and the more important the position is.

[0043] The method forward-propagates the input image through the model, calculates the score of the target category, and extracts the feature map of the last convolutional layer; then, through back-propagation, the gradient of the target category score with respect to the convolutional feature map is calculated as the gradient map, and the global average of the gradient in the spatial dimension is extracted to obtain the importance weight of each channel; then, these weights are used to perform a weighted summation on each channel of the convolutional feature map to generate an initial category activation map; finally, the generated activation map is upsampled to the same size as the input image and overlaid with a color heat map for visualization, thereby intuitively showing the model's focus area on the target category.

[0044] Preferably, in step (5), the implementation process of segmentation region optimization is:

[0045] After obtaining the feature map CAM in step (4), the conditional random field (CRF) is used for optimization to enhance the clarity and spatial consistency of the boundary; the energy function is defined as:

[0046]

[0047] in is the single-point potential energy, representing the pixel x i The ability relationship between the CAM and its category label, the CAM value is used as the confidence of the category to which the pixel belongs, that is, CAM(x i ) is higher, the pixel x i The more likely one is to fall into this particular category; is the dual potential energy, representing the adjacent pixel x i and x j This potential energy is calculated by the Gaussian kernel function, which takes into account the spatial distance and color similarity between pixels, thereby encouraging spatially adjacent pixels to have the same label; by minimizing the energy function E(X), the optimized CRF segmentation result can be obtained:

[0048] M crf =argmin X E(X) (8).

[0049] Preferably, in step (6), the implementation process of segmentation hint generation and optimization is:

[0050] In order to further avoid the problem that the generated segmentation area is inaccurate due to a single sample with large distribution differences, which in turn affects the instability of the segmentation prompt, a segmentation prompt knowledge base B is constructed based on visual prompts to store N key-value pairs. The segmentation hint knowledge base follows the first-in-first-out principle, where k is the low-frequency feature calculated for each sample based on the visual hint M, and q is the CRF segmentation result M calculated for the corresponding sample using formula (8) crf ;k n Represents the low-frequency features of the nth sample, q n Represents the segmentation result corresponding to the nth sample; for each sample v, the similarity is calculated based on its low-frequency vector and k of each sample in the knowledge base:

[0051]

[0052] Where ‖·‖ is the Euclidean norm; the U most similar samples are selected from the similarity calculated by formula (9) as the support set of the current sample, and then the exponential sliding average method is used to update step (5) to obtain the optimized segmentation result M crf :

[0053]

[0054] where α is a hyperparameter, is the normalized similarity measure, q u Represents the segmentation result corresponding to the u-th sample; the final optimized segmentation result M crf It is input into SAM as the segmentation hint of the current sample.

[0055] Preferably, α is set to 0.99.

[0056] Preferably, the implementation process of SAM fine-tuning in step (7) is:

[0057] For each test sample v, A enhanced versions are generated through data enhancement. Data enhancement can generally be achieved by rotating, enlarging, adding color disturbance and other operations on the image. The ath image after enhancement is The average feature of all enhanced versions is expressed as: Through optimization, the difference between the test sample characteristics and the more stable average characteristics is minimized, and the optimization goal is:

[0058]

[0059] Where D is the feature dimension of SAM’s visual encoder IE; IE represents SAM’s visual encoder;

[0060] The final segmentation result y is defined as:

[0061]

[0062] SAM() stands for Segmentation Everything Model; this process helps the Segmentation Everything Model (SAM) maintain its characteristics when the input data changes, thereby improving its performance on OOD samples. In particular, this optimization only involves updating the visual encoder of SAM.

[0063] Where the present invention is not exhaustive, please refer to the prior art.

[0064] The beneficial effects of the present invention are:

[0065] 1. The present invention proposes a complete solution for general medical image segmentation. By introducing the methods of test prompt fine-tuning, segmentation prompt optimization and SAM fine-tuning, it can directly segment the test data without the need for pre-training with labeled data. Traditional medical image segmentation methods are mostly targeted at specific tasks and can only be applied to specific types of segmentation tasks at a time, such as eye recognition or lung image segmentation. The segmentation knowledge of eyeball blood vessels cannot be directly applied to the segmentation task of lung images, which means that after completing the eye recognition task, lung data must be re-trained to adapt to the new segmentation task. On the one hand, it is difficult to obtain medical data, and on the other hand, knowledge between different tasks cannot be shared.

[0066] 2. This invention effectively addresses the problem of poor segmentation hint quality caused by a lack of semantic information in test data through test hint fine-tuning, segmentation region optimization, and segmentation hint optimization. Specifically, by combining a language-image multimodal contrast enhancement model and introducing visual hint fine-tuning, CLIP focuses more on the target region, thereby generating high-quality segmentation hints. Furthermore, by leveraging historical information for improvements, the proposed method ultimately generates stable segmentation hints, avoiding the problem of unstable predictions caused by single samples with large distribution differences.

[0067] 3. By fine-tuning SAM and using unsupervised learning to calculate consistent features on test data, we enhanced our ability to perceive changes in data distribution, further improving the model's generalization and thus improving segmentation results. Furthermore, by fine-tuning the visual encoder, we preserved as much pre-trained knowledge as possible, ensuring that the model adapts to new data without losing its original valid features. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings in the specification, which constitute a part of this application, are used to provide further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute improper limitations on this application.

[0069] Figure 1 This is a flow chart of a general medical image segmentation method based on multimodal collaborative contrast enhancement according to the present invention. DETAILED DESCRIPTION

[0070] In order to enable people in this technical field to better understand the technical solutions in this specification, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings in the implementation of this specification, but are not limited to this. Anything not fully described in the present invention shall be based on the conventional technology in this field.

[0071] Example 1

[0072] A general medical image segmentation method based on multimodal collaborative contrast enhancement is used to segment medical images of glaucoma, comprising the following steps:

[0073] (1) Semantic information generation: Use language models such as GPT to generate semantic information. The input requirement is: "I am implementing a glaucoma segmentation task. Please generate some positive text and negative text. Among them, the positive text should emphasize the cup-basal retina, and the negative text should emphasize blood vessels rather than the retina." The text generated in this way can provide semantic guidance for subsequent segmentation tasks.

[0074] (2) Visual cue generation: A Fourier transform is applied to each eye image to initialize the image into low-frequency components. The low-frequency components reveal the basic shape of the eye image and are relatively stable. A learnable matrix is ​​defined as a visual cue and optimized through training to better capture the key areas of the eye structure.

[0075] (3) Test-cue fine-tuning: When using the CLIP model to extract semantic information, it may inevitably emphasize irrelevant blood vessels. This step optimizes the similarity between the input image and the positive text and increases the distance between it and the negative text, allowing CLIP to focus more on the target area, such as the cup-basin area of ​​the retina, rather than irrelevant blood vessels. Retinal images typically contain a large number of blood vessels, so this fine-tuning allows the model to focus on areas relevant to glaucoma.

[0076] (4) Segmentation feature map generation: CLIP forward propagation generates a feature map that displays information about different features in the glaucoma image. Backpropagation is then used to calculate a gradient map that reflects the contribution of each pixel to the target cup-and-basin region of the retina. Finally, the feature map calculated by forward propagation and the gradient map obtained by backpropagation are combined to highlight the region in the image that most activates the specific target cup-and-basin region.

[0077] (5) Segmentation region optimization: For the cup-and-basin region in glaucoma images, the generated cup-and-basin feature map is optimized by applying conditional random fields to generate a more accurate segmentation map. Conditional random fields use the adjacent relationship between pixels and the edge information of the image to adjust the initial segmentation results, encouraging adjacent pixels to have the same label, thereby enhancing spatial consistency and segmentation boundary details. The optimized segmentation map is input into SAM as a segmentation hint to further extract and refine the segmentation results of the target cup-and-basin region in the retinal image, thereby improving the accuracy and reliability of the segmentation of glaucoma-related regions.

[0078] (6) Segmentation hint generation and optimization: When generating preliminary segmentation hints, more accurate segmentation results are obtained based on the optimization results of feature maps and conditional random fields. At the same time, in order to further improve the segmentation hints, an information knowledge base is constructed to store and manage the segmentation information of high-quality samples. Through this knowledge base, samples with similar structures (such as samples with close retinal positions) are analyzed and extracted so that their segmentation hints can remain consistent. This process also comprehensively utilizes the information of high-quality samples to reduce the negative impact on segmentation stability caused by irrelevant blood vessels or large differences in sample distribution, thereby improving the accuracy and robustness of glaucoma-related area segmentation.

[0079] (7) SAM fine-tuning: Data augmentation is performed on eye photos, such as adding color perturbations or rotations, to generate augmented data. By optimizing the consistency loss, the output before and after augmentation remains consistent, thereby improving the model's stability and adaptability to distribution changes.

[0080] (8) Segmentation result generation: The generated segmentation hints and test images are input into the SAM model to finally obtain accurate segmentation results.

[0081] Example 2

[0082] A general medical image segmentation method based on multimodal collaborative contrast enhancement is used to segment lung medical images, comprising the following steps:

[0083] (1) Semantic information generation: Use language models such as GPT to generate semantic information. The input requirement is: "We are implementing the lung image segmentation task. Please generate some positive text and negative text. Among them, the positive text should focus on the lung area, and the negative text should focus on parts outside the lungs, such as the heart and ribs." The text generated in this way can provide semantic guidance for subsequent segmentation tasks.

[0084] (2) Visual Cue Generation: A Fourier transform is performed on each lung image, focusing on extracting low-frequency information to capture the general outline and shape of the lungs. Based on this low-frequency information, a learnable matrix is ​​constructed as a visual cue. Through optimization and adjustment, it can effectively highlight the key feature areas of the lungs while reducing interference from non-lung areas such as the heart and ribs.

[0085] (3) Test cue fine-tuning: When using the CLIP model to process lung images, in order to prevent the model from focusing on non-target areas (such as the heart or ribs), this step adjusts the model's attention. Specifically, the matching degree between the input image and the positive semantic cue is optimized, and its correlation with the negative cue is suppressed, thereby ensuring that the model focuses on the lung area and minimizing false detections.

[0086] (4) Segmentation feature map generation: The forward propagation of CLIP generates a feature map to display the different feature information of the lungs in the image. Then, the gradient map is calculated through backpropagation to reflect the contribution of each pixel to the lung area (including the lung lobes and boundaries). Finally, the features calculated by the forward propagation are combined.

[0087] (5) Segmentation region optimization: For the lung region in the image, the initial lung feature map generated is optimized by applying conditional random fields to generate more accurate segmentation results. The initial segmentation results are adjusted by using the adjacent relationship between pixels and the edge information of the image, encouraging adjacent pixels to have the same label, thereby enhancing spatial consistency and refining the segmentation boundaries. The optimized lung segmentation map is input into SAM as a segmentation hint to further extract and refine the segmentation results of the target lung region in the image, improving the accuracy and reliability of the segmentation.

[0088] (6) Segmentation hint generation and optimization: When generating preliminary lung segmentation hints, more accurate segmentation results are obtained based on the optimization results of feature maps and conditional random fields. At the same time, in order to further improve the segmentation hints, an information knowledge base is constructed to store and manage the segmentation information of high-quality samples. Through this knowledge base, samples with similar structures (such as images with similar lung morphology and position) are analyzed and extracted to ensure that the segmentation hints remain consistent. This process also utilizes the information of high-quality samples to reduce the instability of segmentation results caused by irrelevant structures (such as ribs, heart shadows) or large differences in sample distribution, thereby improving the accuracy and robustness of lung region segmentation.

[0089] (7) SAM fine-tuning: Further, by performing data augmentation on lung images (e.g., introducing different angle rotations, noise perturbations, etc.), the model’s adaptability to diverse inputs is enhanced. Optimization is performed using a consistency loss function, ensuring that the model’s output is stable and consistent under different augmentation conditions, thereby improving its generalization ability and sensitivity to lung structure.

[0090] (8) Segmentation result generation: Combining the generated segmentation hints with the enhanced lung image, the system uses the SAM model to generate the final segmentation result. This result has high accuracy and robustness, and can provide a reliable basis for the diagnosis of lung diseases.

[0091] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A general medical image segmentation method based on multimodal collaborative contrast enhancement, characterized in that: The steps include: (1) Semantic information generation: Generate corresponding text prompts for the input medical image samples, including positive text t p and negative text t n ; (2) Visual cue generation: Generate a unique visual cue v for each input medical image sample p As auxiliary information to guide the segmentation process, visual cues not only rely on the image itself, but also combine the knowledge learned in advance by the model to ensure that the cues can accurately reflect the important features in the image; (3) Test cue fine-tuning: After generating the visual cue, CLIP adjusts the visual cue to make it more consistent with the characteristics of the target text cue by comparing the similarity between the image and the text cue as a loss function; (4) Segmentation feature map generation: The corresponding feature map is generated through the forward propagation of CLIP to display the different feature information in the image. At the same time, the gradient map is calculated through back propagation. The gradient map reflects the contribution of each pixel to a specific category. Combining the feature map after forward propagation and the gradient map after back propagation, the area in the image that can most activate a specific category is highlighted; (5) Segmentation region optimization: Apply conditional random fields to optimize the generated feature map to generate a segmentation map, which is input into SAM as a segmentation hint. By considering the adjacent relationship between pixels, adjacent pixels are encouraged to have the same label, thereby enhancing spatial consistency. (6) Segmentation hint generation and optimization: For a single medical image sample, preliminary segmentation hints are generated through step (5), and a segmentation hint knowledge base is built based on visual hints. The segmentation hints of each medical image sample are not only derived from the sample itself, but also combined with historical information to ensure the stability and reliability of the segmentation hints. (7) SAM fine-tuning: Perform data enhancement on the test data to optimize the distance between the original sample features and the enhanced aggregated features. Use unsupervised learning methods to extract robust features so that the model can better adapt to the complexity of the test data. Finally, a high-precision segmentation result is generated through the fine-tuned SAM model.

2. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 1, characterized in that: In step (2), generate visual cue v p The process is: For the input medical image sample v, apply the fast Fourier transform to obtain its unique amplitude and phase: amp,pha = FFT(v) (1) In the above formula, amp represents amplitude and pha represents phase; The amplitude map is shifted to move the DC component of the two-dimensional spectrum map to the center of the image, so that the low-frequency information is concentrated in the central area. Subsequently, an adaptive binary matrix M∈{0,1} is used to control the proportion of low-frequency and high-frequency components. Each element of the binary matrix M is between [0,1], indicating the degree of weighting of the frequency component at that position: the closer the value is to 1, the more the frequency component is retained, and the closer the value is to 0, the more the frequency component is suppressed. The binary matrix M is continuously adjusted and optimized through backpropagation during the SAM model training process to ensure that the visual cue can best capture the characteristics of the target area. Then, the amplitude amp is updated by the binary matrix M to obtain amp f , the update process is amp f =amp×M, updated amplitude amp f And the phase pha is recombined according to formula (2) to generate the optimized frequency information: iFFT() means inverse Fourier transform; Finally, the frequency domain information is converted back to the spatial domain of the image through inverse Fourier transform to generate the optimized image As input to CLIP.

3. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 2, characterized in that: In step (3), the test prompt fine-tuning is implemented as follows: Based on the positive text prompt t generated in step (1) p and negative text t n , text encoder via CLIP Get the corresponding embedding vectors for positive and negative text prompts and At the same time, the input image passes through the CLIP visual encoder Get the corresponding embedding vector e v : For each sample vector, calculate the cosine similarity sim(v,t) between it and each text vector: sim(v,t)=cosine(e v ,e t ) (4) in cosine() represents the cosine distance, and the image embedding vector e is calculated by formula (4) v The cosine distance between the embedding vectors of positive and negative samples; in order to enhance CLIP’s focus on the target area, the distance to the positive sample is further reduced, while the distance to the negative text is increased: Where K is the number of positive texts, J is the number of negative samples, is the kth positive textual cue corresponding to image v, is the jth positive text hint corresponding to image v; through the loss function Back-propagation updates the visual cue M.

4. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 3, characterized in that: In step (4), the implementation process of segmentation feature map generation is: First, the image Input into CLIP to perform forward propagation to obtain the feature map, and then based on the loss function of step (3) Perform backpropagation to obtain the gradient map, and use the gradient map to generate the class activation map; Use the class activation map to get the feature map; perform weighted summation on the features of the normalized network of the last layer of the CLIP visual encoder to generate a feature map representing a specific category; calculate the contribution of each element to the feature category in the image: where f k (x,y) is the activation value of the kth channel at position (x,y), w k It is the fully connected weight related to the category; CAM(x,y) represents the activation intensity of a category in the input image at position (x,y), reflecting the importance of the position for predicting the category.

5. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 4, characterized in that: In step (5), the implementation process of segmentation region optimization is as follows: After obtaining the feature map CAM in step (4), the conditional random field is used for optimization to enhance the clarity and spatial consistency of the boundary; the energy function is defined as: in is the single-point potential energy, representing the pixel x i The ability relationship between the CAM and its category label, the value of the CAM is used as the confidence of the category to which the pixel belongs; is the dual potential energy, representing the adjacent pixel x i and x j By minimizing the energy function E(X), the optimized CRF segmentation result can be obtained: M crf =argmin X E(X) (8)。 6. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 5, characterized in that: In step (6), the implementation process of segmentation hint generation and optimization is as follows: Build a segmentation hint knowledge base B based on visual hints to store N key-value pairs The segmentation hint knowledge base follows the first-in-first-out principle, where k is the low-frequency feature calculated for each sample based on the visual hint M, and q is the CRF segmentation result M calculated for the corresponding sample using formula (8) crf ;k n Represents the low-frequency features of the nth sample, q n Represents the segmentation result corresponding to the nth sample; for each sample v, the similarity is calculated based on its low-frequency vector and k of each sample in the knowledge base: Where ‖·‖ is the Euclidean norm; the U most similar samples are selected from the similarity calculated by formula (9) as the support set of the current sample, and then the exponential sliding average method is used to update step (5) to obtain the optimized segmentation result M crf : where α is a hyperparameter, is the normalized similarity measure, q u Represents the segmentation result corresponding to the u-th sample; the final optimized segmentation result M crf It is input into SAM as the segmentation hint of the current sample.

7. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 6, characterized in that: α is set to 0.

99.

8. The universal medical image segmentation method based on multimodal collaborative contrast enhancement according to claim 6, characterized in that: The implementation process of SAM fine-tuning in step (7) is: For each test sample v, A enhanced versions are generated through data enhancement, and the enhanced a-th image is The average feature of all enhanced versions is expressed as: Through optimization, the difference between the test sample characteristics and the more stable average characteristics is minimized, and the optimization goal is: Where D is the feature dimension of SAM’s visual encoder IE; IE represents SAM’s visual encoder; The final segmentation result y is defined as: SAM() means split all models.

Images