Method for applying multi-level fine-tuning visual language model to medical image segmentation

Abstract

The invention discloses a method for applying a multi-level fine-tuning visual language model to medical image segmentation, and relates to the technical field of intelligent image processing. According to the method, a plurality of residual adapters are integrated into a pre-trained visual encoder, visual features are gradually enhanced on different levels, in order to reserve original input information, Convtokenizer is introduced to align an image token of CLIP and original input, and the multi-level adaptation is guided through a multi-level and pixel-level visual-language feature alignment loss function, so that the visual feature alignment accuracy is improved. The focus of the model is recalibrated, object semantics in a natural image is converted into anomaly recognition in a medical image, adaptive features show better generalization ability in various medical data types, and even if the model encounters unseen medical modalities and anatomical regions in the training process, the model can be better recognized. And a good effect can be shown in a zero sample scene.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent image processing, and in particular to a method for using a multi-level fine-tuning visual language model for medical image segmentation. Background Art

[0002] Medical anomaly detection (AD) aims to identify abnormal patterns in medical data and is crucial for preventing misdiagnosis and promoting early intervention. Medical images vary greatly in imaging modalities and anatomical regions, so a general model that can be applied to various data types is needed.

[0003] Few-shot AD methods attempt to achieve model generalization when training data is scarce. Although each new AD task still requires lightweight retraining or distribution adjustment, they represent an initial attempt to build a universal AD model.

[0004] In recent years, large-scale pre-trained vision-language models (VLMs) have provided new ideas for building robust and generalizable anomaly detection models. One important attempt is to directly use CLIP (an open source VLM for natural images) for AD, adapting it only with carefully designed artificial text prompts. Further leveraging labeled training data, additional linear layers are introduced between the joint embedding space of image features and text features for more effective feature matching. Despite the excellent performance of these methods, their application in the medical field remains underexplored.

[0005] Current medical AD methods typically treat anomaly detection as a single-class classification problem and rely on normal images for training. These methods detect anomalies by identifying samples that deviate from the normal distribution, but typically require a large number of normal samples per class, which is unrealistic in actual diagnosis. Furthermore, many techniques target only specific anatomical regions or limit each model to processing a single data type, making them difficult to generalize to different data modalities and anatomical regions.

[0006] Therefore, a new solution to the above problems needs to be proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for fine-tuning a multi-level visual language model for medical image segmentation, aiming to fine-tune the CLIP model to make it suitable for medical anomaly detection, so as to solve the technical problems raised in the background technology.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for using a multi-level fine-tuning visual language model for medical image segmentation, comprising at least the following steps:

[0009] S1: Pre-training on natural images to obtain a natural image visual language model M nat, after the medical image dataset D med The training and fine-tuning generates a medical adaptation model M med , thus used for abnormality detection in medical images, the medical adaptation model M med It is the CLIP model;

[0010] S2: In medical adaptation model M med A multi-level feature adaptation framework designed specifically for AD in medical images is introduced in

[15] . Lightweight feature adapters are applied at multiple feature levels for feature adaptation and additional convolution paths are used to preserve the original input information, ultimately obtaining global and local features.

[0011] S3: Language features are further introduced as a differentiation method in the multi-level feature adaptation framework. The language feature differentiation method is a two-layer text prompt method for an effective anomaly classification and segmentation framework;

[0012] S4: Leveraging multi-level adaptive features, we can effectively identify global anomalies for classification and local anomalies for segmentation through visual-linguistic feature alignment.

[0013] S5: Perform multi-level feature comparison. In order to accurately predict image-level and pixel-level anomalies, a two-branch multi-level feature comparison architecture is adopted. The multi-level feature comparison architecture includes a zero-sampling branch and a few-sample branch to obtain the final prediction result.

[0014] Furthermore, the medical image dataset D med Treated as a pre-training dataset, rather than data in the test sample, it evaluates the model's generalization to unseen scenes;

[0015] The medical image dataset D med Contains medical data from different modalities and anatomical regions, making M nat Can be converted into M med ;

[0016] D med Defined as a set of tuples Where K is the total number of image samples in the dataset;

[0017] Each tuple contains a training image x i , and its corresponding image-level abnormality classification label is the AC label c i ∈{-,+}, and pixel-level anomaly segmentation, namely AS annotation S i ∈{-,+} h×w , for an image of size h×w;

[0018] The label '+' represents an abnormal sample, while '-' represents a normal sample;

[0019] For a given test image X test ,aims to accurately predict anomalies at the image level and pixel level, corresponding to AC and AS, respectively;

[0020] In order to model abnormality detection from invisible imaging patterns and anatomical regions, the natural image visual language model is applied to the background of zero-shot learning or few-shot learning, that is, to use the medical image dataset D med During the training fine-tuning process, considering the practicality of obtaining a limited number of samples from the target scene, the medical image dataset D med The value of K becomes smaller, resulting in zero-shot learning or few-shot learning.

[0021] Furthermore, in S2, a feature adapter is introduced to address the overfitting problem caused by the high number of parameters and limited training data. While maintaining the original CLIP model backbone, a set of learnable fully connected layers is added to the visual branch of CLIP, achieving adaption at multiple feature levels. In order to keep the original image information as a cross-domain reference for the CLIP model and guide the CLIP model for medical image anomaly detection tasks, a ConvTokenizer block is introduced. The original input image is converted into tokens through a layer of convolution and then merged with the image feature tokens of the CLIP module.

[0022] For an image x, the visual encoder in the CLIP model has four consecutive stages, namely S1 to S4;

[0023] Where h is the height; w is the width; 3 means a three-dimensional tensor with three color channels;

[0024] Convert image x into feature space Among them, G represents the grid number, d represents the feature dimension;

[0025] The outputs of the first three visual encoder stages, i.e., S1 to S3, are denoted as l∈{1,2,3}, represents the characteristics of the three intermediate stages;

[0026] Visual feature adaptation involves three different levels of feature adapters A l (·) and a feature projection layer P(·), where l∈{1,2,3};

[0027] At each layer l∈{1,2,3}, a learnable feature adapter A l (·) is integrated into the feature In it, there are two layers of linear transformations;

[0028] This integration transforms the features to adapt, expressed as:

[0029]

[0030] Here, W 1,1 and W 1,2 Represents learnable parameters of the linear transformation;

[0031] Use residual connections in the feature adapter to retain the original knowledge encoded by the pre-trained CLIP model, that is, a constant γ is used as the residual ratio to adjust the degree of retention of the original knowledge to improve performance;

[0032] Therefore, the feature adapter of the l-th feature layer is expressed as:

[0033]

[0034] by As the next encoder stage S l+1 By default, γ is set to 0.1 and α is set to 0.7. In addition, in order to process the global and local features of AC and AS simultaneously, a dual adapter is used to adapt the anomaly segmentation and detection tasks respectively, generating two parallel feature sets at each level. and

[0035] For the final visual feature F generated by the visual encoder in the CLIP model vis , the feature projection layer P(·) uses parameters W cls and W seg The linear layer projects it to obtain global and local features and

[0036] Furthermore, the differentiation method in S3 includes state level and template level;

[0037] At the state level, this includes using simple, general text descriptions of normal and abnormal states, focusing on clarity and avoiding complex details;

[0038] At the template level, multiple language templates are thoroughly examined, and the average of the text features extracted by the text encoder for normal and abnormal states is calculated to obtain a representation as The text features of , where d is the feature dimension.

[0039] Furthermore, the S4 at least includes the following steps:

[0040] For image-level anomaly annotations c i∈{-,+} and the corresponding pixel-level anomaly map S i ∈{-,+} h×w , by aligning the adaptive visual features and textual features, the model is optimized at each feature level l∈{1,2,3,4}, which is achieved through a loss function that combines different components:

[0041]

[0042] Among them, Dice, Foca and BC are dice loss, focal loss and binary cross entropy loss respectively; λ1, λ2 and λ3 are the weights of each loss respectively, and here we set λ1=λ2=λ3=1.0 as the default value; Represents the visual segmentation features on the lth layer; Represents the visual features on the lth layer; represents text features; S is the pixel-level anomaly map; c is the image-level anomaly annotation;

[0043] The overall adaptation loss Ladapt is then calculated as the sum of the losses at each feature level, expressed as

[0044] Furthermore, the zero-sample branch processes the processed test image X test To produce multi-level adaptive features;

[0045] These features are then combined with text features Make comparisons;

[0046] Zero-shot anomaly detection and anomaly segmentation results, denoted as C zero and S zero , the results of anomaly detection and segmentation are calculated using the average softmax score of the four levels

[0047]

[0048] in, Indicates the maximum value operation on the entire feature map; G represents the grid number; BI(·) reshapes the abnormal map into And use bilinear interpolation to restore it to the original input image resolution.

[0049] Furthermore, in the few-sample branch, D med All the multi-level visual features of the few labeled normal images in

[15] contribute to building a multi-level feature memory To facilitate feature comparison, the AC and AS scores are recorded as C few and S few , is derived from the minimum distance between the test feature and the repository feature at each level through a nearest neighbor search process, see the following formula

[0050]

[0051] Here, Dist(·,·) represents the cosine distance, which is calculated as 1-cos(·,·).

[0052] Furthermore, the final predicted AC and AS results of the multi-level feature comparison combine the results of the zero-sampling branch and the few-sample branch, as shown in the following formula:

[0053] C pred =β1C zero +β2C few ,S pred =β1S zero +β2S few (6)

[0054] Among them, β1 and β2 are the weight factors of zero-shot and few-shot branches, respectively, and are set to 0.5 by default.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The method of the present invention integrates multiple residual adapters into a pre-trained visual encoder to achieve progressive enhancement of visual features at different levels. In order to preserve the original input information, the present invention introduces a ConvTokenizer to align the image tokens of CLIP with the original input. This multi-level adaptation is guided by a multi-level, pixel-level visual-language feature alignment loss function, which recalibrates the model's focus from object semantics in natural images to anomaly recognition in medical images. The adapted features show better generalization ability across various medical data types and perform well in zero-shot scenarios even when the model encounters unseen medical modalities and anatomical regions during training.

[0057] 2. This paper proposes a novel multi-level adaptation and comparison framework. This is the first attempt to use pre-trained vision-language models for medical AD tasks in zero-shot / few-shot scenarios. Experimental results on medical anomaly detection benchmarks show that the method of this paper significantly surpasses the current state-of-the-art models. In zero-shot and few-shot settings, the average AUC for anomaly classification is improved by 10.28% and 10.35%, respectively, and the average AUC for anomaly segmentation is improved by 2.24% and 2.41%, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 Schematic diagram of a multi-level adaptive and comparative framework for zero / few-shot medical anomaly detection according to the present invention;

[0060] Figure 2 A comparative schematic diagram of the method of the present invention; DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0062] The method of the present invention aims to develop a general medical image AD model with good generalization ability, which is used to perform segmentation prediction of medical images and enable it to adapt to imaging modalities and anatomical regions that have never been seen before. Building such a model has important practical value, but there are three major challenges in adapting CLIP for medical AD tasks. First, reusing CLIP for AD tasks means a major shift in task requirements. CLIP's visual encoder focuses mainly on the semantic information of the image, while general AD models need to identify abnormal patterns across multiple semantic scenarios. Second, the migration from natural images to medical images involves significant domain conversion issues. Finally, the task of making the AD model applicable to imaging modalities and anatomical regions that have not been seen during the training phase is itself extremely challenging.

[0063] See also Figure 1 The method of multi-level fine-tuning visual language model for medical image segmentation includes at least the following steps:

[0064] S1: Pre-training on natural images to obtain a natural image visual language model M nat , after the medical image dataset D med The training and fine-tuning generates a medical adaptation model M med , which is used for anomaly detection in medical images, medical adaptation model M med It is the CLIP model;

[0065] S2: In medical adaptation model M medA multi-level feature adaptation framework designed specifically for AD in medical images is introduced in

[15] . Lightweight feature adapters are applied at multiple feature levels for feature adaptation and additional convolution paths are used to preserve the original input information, ultimately obtaining global and local features.

[0066] S3: We further introduce language features as a distinguishing method in the multi-level feature adaptation framework. The language feature distinguishing method is a two-layer text prompt method for an effective anomaly classification and segmentation framework.

[0067] S4: Leveraging multi-level adaptive features, we can effectively identify global anomalies for classification and local anomalies for segmentation through visual-linguistic feature alignment.

[0068] S5: Perform multi-level feature comparison. In order to accurately predict image-level and pixel-level anomalies, a two-branch multi-level feature comparison architecture is adopted. The multi-level feature comparison architecture includes a zero-sampling branch and a few-sample branch to obtain the final prediction results.

[0069] Medical Image Dataset D med Treated as a pre-training dataset, rather than data in the test sample, it evaluates the model's generalization to unseen scenes;

[0070] Medical Image Dataset D med Contains medical data from different modalities and anatomical regions, making M nat Can be converted into M med ;

[0071] D med Defined as a set of tuples Where K is the total number of image samples in the dataset;

[0072] Each tuple contains a training image x i , and its corresponding image-level abnormality classification label is the AC label c i ∈{-,+}, and pixel-level anomaly segmentation, namely AS annotation S i ∈{-,+} h×w , for an image of size h×w;

[0073] The label '+' represents an abnormal sample, while '-' represents a normal sample;

[0074] For a given test image X test ,aims to accurately predict anomalies at the image level and pixel level, corresponding to AC and AS, respectively;

[0075] In order to model abnormality detection from invisible imaging patterns and anatomical regions, the natural image visual language model is applied to the background of zero-shot learning or few-shot learning, that is, to use the medical image dataset D med During the training fine-tuning process, considering the practicality of obtaining a limited number of samples from the target scene, the medical image dataset D med The value of K becomes smaller, resulting in zero-shot learning or few-shot learning.

[0076] In S2, a feature adapter is introduced to address the overfitting problem caused by the high number of parameters and limited training data. While maintaining the original CLIP model backbone, a set of learnable fully connected layers is added to the CLIP visual branch, achieving adaption at multiple feature levels. In order to keep the original image information as a cross-domain reference for the CLIP model and guide the CLIP model for medical image anomaly detection tasks, a ConvTokenizer block is introduced. The original input image is converted into tokens through a layer of convolution and then merged with the image feature tokens of the CLIP module.

[0077] For an image x, the visual encoder in the CLIP model has four consecutive stages, namely S1 to S4;

[0078] Where h is the height; w is the width; 3 represents a three-dimensional tensor with three color channels (RGB);

[0079] Convert image x into feature space Among them, G represents the grid number, d represents the feature dimension;

[0080] The outputs of the first three visual encoder stages, i.e., S1 to S3, are denoted as l∈{1,2,3}, represents the characteristics of the three intermediate stages;

[0081] Visual feature adaptation involves three different levels of feature adapters A l (·) and a feature projection layer P(·), where l∈{1,2,3};

[0082] At each layer l∈{1,2,3}, a learnable feature adapter A l (·) is integrated into the feature In it, there are two layers of linear transformations;

[0083] This integration transforms the features to adapt, expressed as:

[0084]

[0085] Here, W 1,1and W 1,2 Represents learnable parameters of the linear transformation;

[0086] Use residual connections in the feature adapter to retain the original knowledge encoded by the pre-trained CLIP model, that is, a constant γ is used as the residual ratio to adjust the degree of retention of the original knowledge to improve performance;

[0087] Therefore, the feature adapter of the l-th feature layer is expressed as:

[0088]

[0089] by As the next encoder stage S l+1 By default, γ is set to 0.1 and α is set to 0.7. In addition, in order to process the global and local features of AC and AS simultaneously, a dual adapter is used to adapt the anomaly segmentation and detection tasks respectively, generating two parallel feature sets at each level. and

[0090] For the final visual feature F generated by the visual encoder in the CLIP model vis , the feature projection layer P(·) uses parameters W cls and W seg The linear layer projects it to obtain global and local features and

[0091] The distinction methods in S3 include state level and template level;

[0092] At the state level, this includes using simple, general text descriptions of normal and abnormal states, focusing on clarity and avoiding complex details;

[0093] At the template level, multiple language templates are thoroughly examined, and the average of the text features extracted by the text encoder for normal and abnormal states is calculated to obtain a representation as The text features of , where d is the feature dimension.

[0094] S4 includes at least the following steps:

[0095] For image-level anomaly annotations c i ∈{-,+} and the corresponding pixel-level anomaly map S i ∈{-,+} h×w , by aligning the adaptive visual features and textual features, the model is optimized at each feature level l∈{1,2,3,4}, which is achieved through a loss function that combines different components:

[0096]

[0097] Among them, Dice, Foca and BC are dice loss, focal loss and binary cross entropy loss respectively; λ1, λ2 and λ3 are the weights of each loss respectively, and here we set λ1=λ2=λ3=1.0 as the default value; Represents the visual segmentation features on the lth layer; Represents the visual features on the lth layer; represents text features; S is the pixel-level anomaly map; c is the image-level anomaly annotation;

[0098] The overall adaptation loss Ladapt is then calculated as the sum of the losses at each feature level, expressed as

[0099] The zero-shot branch processes the test image X test To produce multi-level adaptive features;

[0100] These features are then combined with text features Make comparisons;

[0101] Zero-shot anomaly detection and anomaly segmentation results, denoted as C zero and S zero , the results of anomaly detection and segmentation are calculated using the average softmax score of the four levels

[0102]

[0103] in, Indicates the maximum value operation on the entire feature map; G represents the grid number; BI(·) reshapes the abnormal map into And use bilinear interpolation to restore it to the original input image resolution.

[0104] D in the few-shot branch med All the multi-level visual features of the few labeled normal images in

[15] contribute to building a multi-level feature memory To facilitate feature comparison, the AC and AS scores are recorded as C few and S few , is derived from the minimum distance between the test feature and the repository feature at each level through a nearest neighbor search process, see the following formula

[0105]

[0106] Here, Dist(·,·) represents the cosine distance, which is calculated as 1-cos(·,·).

[0107] The final predicted AC and AS results of the multi-level feature comparison combine the results of the zero-sampling branch and the few-shot branch, as shown in the following formula:

[0108] C pred =β1C zero +β2C few ,S pred =β1S zero +β2S few (6)

[0109] Among them, β1 and β2 are the weight factors of zero-shot and few-shot branches, respectively, and are set to 0.5 by default.

[0110] In summary:

[0111] The method proposed in the present invention introduces a lightweight multi-level adaptation and comparison framework while building the model, so that CLIP can be reused for AD tasks in medical images. The method designs a multi-level visual feature adaptation architecture to make the features extracted by CLIP meet the requirements of medical AD tasks. The visual feature adaptation process combines adapter tuning and multi-level feature adjustment. Specifically, the method integrates multiple residual adapters in the pre-trained visual encoder to gradually enhance the visual features at different levels, and is guided by a multi-level, pixel-by-pixel visual-language feature alignment loss function. These adapters adjust the focus of the model from object semantic information to anomaly detection in medical images, and use text cues to roughly classify images as "normal" or "abnormal". During the testing phase, the adapted visual features are compared with the text cue features (and available reference image features) to generate a multi-level anomaly score map.

[0112] Based on the above content, the following experimental process is proposed. For a comparative schematic diagram, see Figure 2 ,exist Figure 2 The MLCA method of the present invention is visualized on brain MRI, liver CT, and retinal OCT, and compared with the state-of-the-art April-GAN method. The results show better performance than the results of

[15] , demonstrating the effectiveness of the proposed multi-level feature adaptation.

[0113] Determine the dataset

[0114] This paper considers a BMAD-based medical anomaly detection (AD) benchmark, covering five different medical image domains and generating six datasets. These datasets include brain MRI, liver CT, retinal OCT, chest X-ray, and digital histopathology. The BrainMRI, LiverCT, and RESC datasets are used for both anomaly classification (AC) and segmentation (AS), while OCT17, ChestXray, and HIS are used only for AC.

[0115] Perform experimental setup

[0116] In this paper, various state-of-the-art AD methods are considered as competitive methods in different training settings. These settings include (i) using all normal traditional methods (CFlowAD, RD4AD, PatchCore and MKD), (ii) a small number of normal sample methods (CLIP, MedCLIP, WinCLIP), and (iii) few-shot methods (DRA, BGAD and April-GAN). We evaluate these methods in terms of AC and AS, but exclude BGAD, which is specialized for segmentation due to the need for pixel-level annotations during training.

[0117] The area under the receiver operating characteristic curve (AUC) is used to quantify performance. This metric is a standard in AD evaluation, with image-level AUC considered separately in AC and pixel-level AUC considered separately in AS.

[0118] Model configuration and training details

[0119] This paper uses CLIP with the VIT-L / 14 architecture, with an input image resolution of 240×240. A convolutional layer with a stride of 14 and a kernel size of 16×16 is used as the ConvTokenizer, preserving the original input information. The model contains 24 layers, divided into four stages, each containing six layers. The paper uses the Adam optimizer, a constant learning rate of 1e-3, and a batch size of 16, and trains for 50 epochs on a single NVIDIA GeForce RTX 3090 GPU.

[0120] Comparison with other methods

[0121] Few-shot Setting: In Table 1, we compare the performance of MLCA with other state-of-the-art AD methods in the few-shot setting with K = 4. Compared to competing methods such as April-GAN, it is noteworthy that across all datasets, MLCA achieves an average 10.25% higher AUC for AC and 2.24% higher AUC for AS than April-GAN. Compared to BGAD, MLCA achieves an average 13.6% higher AUC for AC and 3.49% higher AUC for AS on datasets with pixel-level annotations.

[0122] MLCA outperforms several CLIP methods based on normal samples, such as CLIP and WinCLIP, which also use a visual-language pre-trained backbone and perform feature comparison on AD. The advantage of MLCA is that it can effectively utilize a small number of abnormal samples, thus achieving better results than these methods.

[0123] For example, for WinCLIP, the average improvement of MLCA is 18.19% AUC for AC and 4.87% AUC for AS across all datasets. While MedCLIP shows superior results on ChestXray because it is trained on the large-scale overlapping ChestXray data in our medical AD benchmark, it lacks broad generalization ability, as can be seen from its performance on other datasets. In addition, MLCA shows substantial improvements over traditional AD methods on all normal samples such as CFlowAD, RD4AD, PatchCore, and MKD, which rely on datasets much larger than the dataset used in our study. This highlights the value of incorporating a small number of abnormal samples as supervision, especially in medical diagnosis, where obtaining a limited amount of abnormal data may be more practical.

[0124] Zero-shot Setting: Zero-shot AD experiments were conducted in a leave-one-out setting. In this configuration, a designated target dataset is selected for testing, while the remaining datasets with different modalities and anatomical regions are used for training. This approach aims to measure its performance when faced with unknown modalities and anatomical regions, thereby evaluating the generalization ability of the model. Table 2 provides a comprehensive overview of the results related to zero-shot medical AC and AS, providing a comparative evaluation with two state-of-the-art methods that leverage the power of the CLIP backbone. MLCA demonstrates significant superiority; for example, it outperforms WinCLIP across all datasets, achieving a 36.24% average AUC improvement for AC and a 6.45% average AUC improvement for AS. Similarly, compared to April-GAN, MLCA achieves a 10.35% average AUC improvement for AC and a 2.41% average AUC improvement for AS across all datasets, demonstrating its effectiveness in the challenging zero-shot medical image AD setting.

[0125] Table 1 Comparison with the state-of-the-art few-shot anomaly detection methods when K=4

[0126]

[0127] The AUCs (in %) for anomaly classification (AC) and anomaly segmentation (AS) are compared. The best results are in bold and the second best results are underlined.

[0128] Table 2 Comparison with state-of-the-art zero-shot anomaly detection methods

[0129]

[0130] Table 2 compares the auc of AC and AS (unit: %), and the best results are shown in bold.

[0131] Conduct ablation experiments

[0132] The present invention uses the average AC and AS indicators of six data sets as the evaluation indicators of the ablation experiment of the present invention. As shown in Table 3, using ConvTokenizer to retain the original input information can significantly improve the performance of anomaly classification. In the zero-sample setting, the effect of anomaly classification is improved by an average of 1.8%, and in the few-sample setting, the effect of anomaly classification is improved by an average of 1.2%, and the effect of anomaly detection is partially improved.

[0133] Table 3 Ablation experiments on whether to use Convtokenizer to retain the original input information

[0134]

[0135] in conclusion:

[0136] The present invention applies a natural domain pre-trained visual language model to medical AD, with cross-domain generalization across different morphological and anatomical regions. This adaptation involves not only the transition from natural to medical domains, but also the transition from high-level semantics to pixel-level segmentation. To achieve such a goal, a collaborative multi-level feature adaptation approach is introduced, where each adaptation is guided by the corresponding visual language alignment, facilitating the segmentation of various forms of abnormalities from medical images. Combined with a comparison-based AD strategy, the method is able to flexibly adapt to datasets with large modality and distribution differences. The proposed method outperforms existing methods on zero / few-shot AC and AS tasks, indicating a promising research avenue for future exploration.

[0137] The proposed method is evaluated on a challenging medical AD benchmark dataset covering five different medical imaging modalities and anatomical regions, including brain MRI, liver CT, retinal OCT, chest X-ray, and digital pathology images. Experimental results show that the proposed method outperforms the state-of-the-art AD methods.

[0138] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A method for medical image segmentation using a multi-level fine-tuned visual language model, characterized by: At least the following steps are included: S1: Pre-training on natural images to obtain a natural image visual language model M nat , after the medical image dataset D med The training and fine-tuning generates a medical adaptation model M med , which is used for anomaly detection in medical images, medical adaptation model M med is the original CLIP model; S2: In medical adaptation model M med A multi-level feature adaptation framework designed specifically for AD in medical images is introduced in [15]. It applies lightweight CLIP adapters for feature adaptation and additional convolutional paths at multiple feature levels to preserve the original input information, ultimately obtaining global and local features. S3: Language features are further introduced as a differentiation method in the multi-level feature adaptation framework. The language feature differentiation method is a two-layer text prompt method for an effective anomaly classification and segmentation framework; S4: Leveraging multi-level adaptive features, we can effectively identify global anomalies for classification and local anomalies for segmentation through visual-linguistic feature alignment. S5: Perform multi-level feature comparison. In order to accurately predict image-level and pixel-level anomalies, a two-branch multi-level feature comparison architecture is adopted. The multi-level feature comparison architecture includes a zero-sampling branch and a few-sample branch to obtain the final prediction result.

2. The method for medical image segmentation using a multi-level fine-tuned visual language model according to claim 1, characterized in that: The medical image dataset D med Treated as a pre-training dataset, rather than data in the test sample, it evaluates the model's generalization to unseen scenes; The medical image dataset D med Contains medical data from different modalities and anatomical regions, making M nat Can be converted into M med ; D med Defined as a set of tuples Where K is the total number of image samples in the dataset; Each tuple contains a training image x i , and its corresponding image-level abnormality classification label is the AC label c i ∈{-,+}, and pixel-level anomaly segmentation, namely AS annotation S i ∈{-,+} h×w , for an image of size h×w; The label '+' represents an abnormal sample, while '-' represents a normal sample; For a given test image X test ,aims to accurately predict anomalies at the image level and pixel level, corresponding to AC and AS, respectively; In order to model abnormality detection from invisible imaging patterns and anatomical regions, the natural image visual language model is applied to the background of zero-shot learning or few-shot learning, that is, to use the medical image dataset D med During the training fine-tuning process, considering the practicality of obtaining a limited number of samples from the target scene, the medical image dataset D med The value of K becomes smaller, resulting in zero-shot learning or few-shot learning.

3. The method for medical image segmentation using a multi-level fine-tuned visual language model according to claim 2, characterized in that: In S2, a feature adapter is introduced to address the overfitting problem caused by the high number of parameters and limited training data. While maintaining the original model backbone, a set of learnable fully connected layers is added to the visual branch of the original CLIP model, achieving adaption at multiple feature levels. In order to keep the original image information as a cross-domain reference for CLIP and guide the CLIP model for medical image anomaly detection tasks, a ConvTokenizer block is introduced. The original input image is converted into tokens through a layer of convolution and then merged with the image feature tokens of the CLIP model. For an image x, the visual encoder in CLIP has four consecutive stages, namely S1 to S4; Where h is the height; w is the width; 3 represents a three-dimensional tensor with three color channels (RGB); Convert image x into feature space Among them, G represents the grid number, d represents the feature dimension; The outputs of the first three visual encoder stages, i.e., S1 to S3, are denoted as Indicates the characteristics of the three intermediate stages; Visual feature adaptation involves three different levels of feature adapters A l (·) and a feature projection layer P(·), where l∈{1,2,3}; At each layer l∈{1,2,3}, a learnable feature adapter A l (·) is integrated into the feature In it, there are two layers of linear transformations; This integration transforms the features to adapt, expressed as: Here, W 1,1 and W 1,2 Represents learnable parameters of the linear transformation; Use residual connections in the feature adapter to retain the original knowledge encoded by the pre-trained CLIP model, that is, a constant γ is used as the residual ratio to adjust the degree of retention of the original knowledge to improve performance; Therefore, the feature adapter of the l-th feature layer is expressed as: by As the next encoder stage S l+1 By default, γ is set to 0.1 and α is set to 0.

7. In addition, in order to process the global and local features of AC and AS simultaneously, a dual adapter is used to adapt the anomaly segmentation and detection tasks respectively, generating two parallel feature sets at each level. and For the final visual feature F generated by the visual encoder in the CLIP model vis , the feature projection layer P(·) uses parameters W cls and W seg The linear layer projects it to obtain global and local features and 4. The method for medical image segmentation using a multi-level fine-tuned visual language model according to claim 3, characterized in that: The differentiation methods in S3 include state level and template level; At the state level, this includes using simple, general text descriptions of normal and abnormal states, focusing on clarity and avoiding complex details; At the template level, multiple language templates are thoroughly examined, and the average of the text features extracted by the text encoder for normal and abnormal states is calculated to obtain a representation as The text features of , where d is the feature dimension.

5. The method for medical image segmentation using a multi-level fine-tuned visual language model according to claim 4, characterized in that: The S4 at least includes the following steps: For image-level anomaly annotations c i ∈{-,+} and the corresponding pixel-level anomaly map S i ∈{-,+} h×w , by aligning the adaptive visual features and textual features, the model is optimized at each feature level l∈{1,2,3,4}, which is achieved through a loss function that combines different components: Among them, Dice, Foca and BC are dice loss, focal loss and binary cross entropy loss respectively; λ1, λ2 and λ3 are the weights of each loss respectively, and here we set λ1=λ2=λ3=1.0 as the default value; Represents the visual segmentation features on the lth layer; Represents the visual features on the lth layer; represents text features; S is the pixel-level anomaly map; c is the image-level anomaly annotation; The overall adaptation loss Ladapt is then calculated as the sum of the losses at each feature level, expressed as 6. The method for medical image segmentation using a multi-level fine-tuned visual language model according to claim 5, characterized in that: The zero-sample branch processes the test image X test To produce multi-level adaptive features; These features are then combined with text features Make comparisons; Zero-shot anomaly detection and anomaly segmentation results, denoted as C zero and S zero , the results of anomaly detection and segmentation are calculated using the average softmax score of the four levels in, Indicates the maximum value operation on the entire feature map; G represents the grid number; BI(·) reshapes the abnormal map into And use bilinear interpolation to restore it to the original input image resolution.

7. The method for medical image segmentation using a multi-level fine-tuned visual language model according to claim 6, characterized in that: In the few-sample branch, D med All the multi-level visual features of the few labeled normal images in [15] contribute to building a multi-level feature memory To facilitate feature comparison, the AC and AS scores are recorded as C few and S few , is derived from the minimum distance between the test feature and the repository feature at each level through a nearest neighbor search process, see the following formula Here, Dist(·,·) represents the cosine distance, which is calculated as 1-cos(·,·).

8. The method for medical image segmentation using a multi-level fine-tuned visual language model according to claim 7, characterized in that: The final predicted AC and AS results of the multi-level feature comparison combine the results of the zero-sampling branch and the few-sample branch, as shown in the following formula: C pred =β1C zero +β2C few ,S pred =β1S zero +β2S few (6) Among them, β1 and β2 are the weight factors of zero-shot and few-shot branches, respectively, and are set to 0.5 by default.

Images