Contrastive language pre-training and test-time adaptation based ct image classification method

Abstract

The application discloses a CT image classification method based on contrast language pre-training and test time adaptation, comprising the following steps: 1, collecting and preprocessing abdominal CT images and category labels from multiple imaging centers to obtain source domain and target domain datasets; 2, pre-training a network architecture based on contrast language pre-training on the source domain dataset to obtain a pre-trained baseline model; 3, performing test time adaptation fine-tuning on the network parameters of the pre-trained baseline model on the target domain dataset to obtain a strengthened model; and 4, predicting CT images in the target domain dataset by using the strengthened model. The pre-training of the baseline model is completed on the source domain dataset, and the test time adaptation fine-tuning is performed on the pre-trained baseline model, so as to alleviate the performance decline of the model caused by the data distribution difference between the source domain dataset and the target domain dataset, thereby improving the prediction accuracy and generalization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology based on medical images, and in particular to a CT image classification method based on contrastive language pre-training and adaptive testing time. Background Technology

[0002] In recent years, deep learning models based on medical imaging have become a research hotspot in the medical-engineering interdisciplinary field due to their superior feature extraction capabilities, and have achieved significant success in disease diagnosis and prognosis prediction. This rapid development has made it possible to obtain microscopic pathological information using radiographic imaging such as CT and MRI, and has already achieved fine-grained predictions, such as lymph node invasion status, in various malignant tumors, including lung cancer.

[0003] However, the limited amount of publicly available clinical imaging data and the complex biological characteristics of tumors make it difficult for existing models in this field to meet medical requirements in terms of predictive accuracy. In addition, the data distribution varies greatly among different imaging centers due to factors such as imaging equipment and examination procedures. How to improve the generalization ability of models on datasets from different imaging centers has also become a major problem that needs to be solved in this field. Summary of the Invention

[0004] To overcome the limitations of existing models, this invention proposes a CT image classification method based on contrastive language pre-training and adaptive testing time. This aims to alleviate the performance degradation caused by the data distribution differences between the source and target domain datasets, thereby improving the accuracy and generalization of CT image classification.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The CT image classification method of this invention, based on contrastive language pre-training and adaptive testing time, is characterized by the following steps:

[0007] Step 1: Collect and preprocess abdominal CT images and category labels from N imaging centers, and randomly select preprocessed abdominal CT images from one imaging center as the source domain dataset. The remaining N-1 preprocessed abdominal CT images from the image centers are used as the target domain dataset. in, Represents the source domain dataset D s The preprocessed abdominal CT image of the i-th source domain user. represent The category label, and Where 0 represents normal and the corresponding abdominal CT image is a negative sample, and 1 represents abnormal and the corresponding abdominal CT image is a positive sample. Represents the target domain dataset D t The preprocessed abdominal CT image of the jth target domain user, n s n t Representing the source domain dataset D respectively s and target domain dataset D t The total number of images;

[0008] Step 2 in the source domain dataset D s The baseline model M, which is pre-trained based on contrastive language, is then pre-trained to obtain the pre-trained baseline model M1.

[0009] Step 3: In the target domain dataset D t The enhanced model M2 is obtained by fine-tuning some parameters of the pre-trained baseline model M1 in an adaptive manner during test time.

[0010] Step 4: Apply the enhanced model M2 to the target domain dataset D. t In Make a prediction and obtain Predicted classification results

[0011] The CT image classification method based on contrastive language pre-training and adaptive testing time described in this invention is characterized in that step 2 is performed as follows:

[0012] Step 2.1 Use the pre-trained segmentation model Q to... Process the data to obtain the region of interest for the i-th source domain user.

[0013] Step 2.2 Select the region of interest Perform classification preprocessing to obtain the region of interest (ROI) for the i-th source domain user after classification preprocessing.

[0014] Step 2.3 will Image encoder Φ in the input baseline model M image The image features of the i-th source domain user are obtained through processing.

[0015] Initialize a set of learning vectors (T) p ,T n ), where T p As a hint for positive samples, T n This serves as a hint for negative samples;

[0016] (T) p ,T n The text encoder Φ in the input baseline model M textThe text features are processed to obtain a set of text features. in, The text features are those of positive samples. Text features for negative samples;

[0017] Step 2.4 Calculation and A set of similarity scores in, express and Similarity score, express and The similarity score;

[0018] Step 2.5 Construction and Cross-entropy loss between, and in the frozen text encoder Φ text Under the given parameters, the baseline model M is pre-trained, and the cross-entropy loss is calculated to update (T). p ,T n ) parameters and image encoder Φ image The parameters are used to obtain the pre-trained learning vectors. Pre-trained image encoder and the frozen text encoder Φ text Together, they form the pre-trained baseline model M1, where, As a hint after positive sample pre-training As a hint after negative sample pre-training;

[0019] Step 2.6 Output using the pre-trained baseline model M1 Predicted classification results:

[0020] when When the time is right, the predicted classification result will be output. Otherwise, output the predicted classification result.

[0021] Step 3 is performed as follows:

[0022] Step 3.1 Use the pre-trained segmentation model Q to... Processing is performed to obtain the region of interest for the j-th target domain user.

[0023] Step 3.2 Select the region of interest Perform classification preprocessing to obtain the region of interest for the j-th target domain user after classification preprocessing.

[0024] Step 3.3 will The image encoder input to the pre-trained baseline model M1 The image features of the j-th target domain user are obtained through processing.

[0025] Will The text encoder Φ is input to the pre-trained baseline model M1. text The pre-trained text features are then processed to obtain them. in, Text features used as positive samples after pre-training Text features used as negative samples after pre-training;

[0026] Step 3.4 Constructs a test-time adaptive loss consisting of prototype constraints and nearest neighbor constraints, and freezes the text encoder Φ. text Parameters and Under the given parameters, the image encoder in the pre-trained baseline model M1... Fine-tuning was performed, and the image encoder was updated by calculating the test-time adaptive loss. The parameters are used to obtain the fine-tuned image encoder. and frozen and text encoder Φ text Together they form the enhanced model M2.

[0027] Step 4 is performed as follows:

[0028] Step 4.1 Select the region of interest Image encoder for the enhanced model M2 The enhanced image features of the j-th target domain user are obtained through processing.

[0029] Step 4.2 Calculation and similarity score in, express and Similarity score, express and The similarity score;

[0030] like Then output Predicted classification results Otherwise, output Predicted classification results

[0031] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the CT image classification method, and the processor is configured to execute the program stored in the memory.

[0032] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the CT image classification method.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1. The method of the present invention adjusts the window width and window level of the preprocessed region of interest to improve the contrast of abdominal tissues and organs in CT images; and performs data enhancement on the adjusted region of interest to increase the diversity of image data, thereby alleviating the limitation caused by insufficient clinical image data.

[0035] 2. The baseline model in the method of this invention adopts a network framework based on contrastive language pre-training CLIP. This network architecture consists of an image encoder and a text encoder, which are used to extract image features and text features respectively. By minimizing the similarity score between image features and text features, the distance between them in the common latent feature space is narrowed, and more discriminative positive and negative class prompts are obtained, thereby improving the prediction accuracy of the baseline model on the source domain dataset.

[0036] 3. The enhanced model in the method of this invention constructs a test-time adaptive loss consisting of prototype constraints and nearest neighbor constraints to update some parameters of the pre-trained baseline model, making it better adaptable to target domain data outside the source domain data distribution, thereby improving the generalization of the enhanced model on the target domain dataset. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the overall technical process of the method of the present invention.

[0038] Figure 2 This is a flowchart illustrating the acquisition of the source and target domain datasets in this embodiment;

[0039] Figure 3 This is a schematic diagram of the segmentation network structure in this embodiment;

[0040] Figure 4 This is a flowchart illustrating the operation of region of interest classification preprocessing in this embodiment;

[0041] Figure 5 This is a schematic diagram of the base version and the enhanced version of the model in this embodiment. Detailed Implementation

[0042] In this embodiment, a CT image classification method based on contrastive language pre-training and adaptive testing time involves pre-training a baseline model on the source domain dataset and then fine-tuning the pre-trained baseline model using adaptive testing time. This mitigates the performance degradation caused by data distribution differences between the source and target domain datasets, thereby improving the accuracy and generalization of CT image classification. Specifically, as shown... Figure 1 As shown, the method mainly includes the following specific steps:

[0043] Step 1: Collect and preprocess abdominal CT images and category labels from N imaging centers, such as... Figure 2 As shown, preprocessed abdominal CT images from one image center were randomly selected as the source domain dataset. The remaining N-1 preprocessed abdominal CT images from the image centers are used as the target domain dataset. in, Represents the source domain dataset D s The preprocessed abdominal CT image of the i-th source domain user. represent The category label, and Where 0 represents normal and the corresponding abdominal CT image is a negative sample, and 1 represents abnormal and the corresponding abdominal CT image is a positive sample. Represents the target domain dataset D t The preprocessed abdominal CT image of the jth target domain user, n s n t Representing the source domain dataset D respectively s and target domain dataset D t The total number of images.

[0044] Step 2 in the source domain dataset D s The baseline model M, which is pre-trained based on contrastive language, is then pre-trained to obtain the pre-trained baseline model M1.

[0045] Step 2.1 Use the pre-trained segmentation model Q to... Process the data to obtain the region of interest for the i-th source domain user.

[0046] Step 2.1.1 The method of this invention pre-trains the segmentation model based on the 3D U-Net network architecture to obtain the pre-trained segmentation model Q. For example... Figure 3As shown, this 3D U-Net mainly includes downsampling and upsampling stages. The basic convolutional modules used in both stages consist of a convolutional layer, a normalization layer, and an activation layer. The 3D U-Net-based network architecture was pre-trained using two publicly available abdominal CT datasets (LiTs dataset and TotalSegmentator dataset) with duodenal region annotations to obtain the pre-trained segmentation model Q.

[0047] Step 2.1.2 The image of the i-th source domain user... Input the pre-trained segmentation model Q to obtain the region of interest.

[0048] Step 2.2 Select the region of interest Perform classification preprocessing to obtain the region of interest (ROI) for the i-th source domain user after classification preprocessing.

[0049] like Figure 4 As shown, the classification preprocessing operation in step 2.2 mainly includes two methods: adjusting window width and window level, and data augmentation.

[0050] Step 2.2.1 Adjust the region of interest The window width and window level are used to obtain the adjusted region of interest, which is denoted as .

[0051] The window width determines the contrast range of a CT image, while the window level determines the central grayscale value displayed in the CT image. In this invention, the window width WW is adjusted to 350, and the window level WL is adjusted to 35 to more clearly display abdominal tissues and organs. First, the adjusted region of interest is calculated based on WW = 350 and WL = 35. The upper limit (Up) and lower limit (Down);

[0052]

[0053] Then, the region of interest before adjustment Each pixel value r i s Perform mapping, and record the mapped pixel values ​​as... Specifically: If r i s If r > 210, then r i s =210; if r i s If r < -140, then i s = -140; if -140 ≤ r i s If ≤210, then calculate according to formula (3).

[0054]

[0055] Therefore, the adjusted region of interest is obtained. Where, n i express The total number of pixels in the image.

[0056] Step 2.2.2 Adjust the region of interest. Perform data augmentation to obtain the augmented region of interest. Specifically, data augmentation methods include horizontal flipping, vertical flipping, cropping, and rotation, thereby increasing data diversity and alleviating, to some extent, the limitations caused by insufficient medical imaging data.

[0057] Step 2.3 as follows Figure 5 As shown, Image encoder Φ in the input baseline model M image The image features of the i-th source domain user are obtained through processing.

[0058] Initialize a set of learning vectors (T) p ,T n ), where T p As a hint for positive samples, T n This serves as a hint for negative samples;

[0059] like Figure 5 The baseline model shown here uses a set of vectors to be learned (T) p ,T n The text encoder Φ in the input baseline model M text The text features are processed to obtain a set of text features. in, The text features are those of positive samples. Text features for negative samples.

[0060] Step 2.4 Calculation and A set of similarity scores in, express and Similarity score, express and The similarity score is calculated using the formulas (4) and (5):

[0061]

[0062] Step 2.5 Construction and Cross-entropy loss between, and in the frozen text encoder Φ text Under the given parameters, the baseline model M is pre-trained, and the cross-entropy loss is calculated to update (T). p ,T n ) parameters and image encoder Φ image The parameters are used to obtain the pre-trained learning vectors. Pre-trained image encoder and the frozen text encoder Φ text Together, they form the pre-trained baseline model M1, where, As a hint after positive sample pre-training As a hint after negative sample pre-training. Figure 5 The snowflakes shown represent the freezing parameters, and the flames represent the updating parameters.

[0063] Step 2.6 Output using the pre-trained baseline model M1 Predicted classification results:

[0064] when When the time is right, the predicted classification result will be output. Otherwise, output the predicted classification result.

[0065] Step 3: In the target domain dataset D t The baseline model M1, after pre-training, undergoes adaptive fine-tuning of some parameters to obtain the enhanced model M2.

[0066] Step 3.1 Use the pre-trained segmentation model Q-pair Processing is performed to obtain the region of interest for the j-th target domain user.

[0067] Step 3.2 Select the region of interest Perform classification preprocessing to obtain the region of interest for the j-th target domain user after classification preprocessing.

[0068] Step 3.2.1 Adjust the region of interest The window width and window level are used to obtain the adjusted region of interest, which is denoted as . The principle is the same as step 2.2.1.

[0069] Step 3.2.2 Adjust the region of interest. Perform data augmentation to obtain the augmented region of interest. The principle is the same as step 2.2.2.

[0070] Step 3.3 will The image encoder input to the pre-trained baseline model M1 The image features of the j-th target domain user are obtained through processing.

[0071] Will The text encoder Φ is input to the pre-trained baseline model M1. text The pre-trained text features are then processed to obtain them. in, Text features used as positive samples after pre-training Text features used as negative samples after pre-training;

[0072] Step 3.4 as follows Figure 5 As shown, a test-time adaptive loss consisting of prototype constraints and nearest neighbor constraints is constructed, and the text encoder Φ is frozen. text Parameters and Under the given parameters, the image encoder in the pre-trained baseline model M1... Fine-tuning was performed, and the image encoder was updated by calculating the test-time adaptive loss. The parameters are used to obtain the fine-tuned image encoder. and frozen and text encoder Φ text Together they form the enhanced model M2.

[0073] The test time adaptive loss consists of two parts: prototype constraint loss and nearest neighbor constraint loss.

[0074] Prototype constraint loss: based on the image features obtained in step 3.3 and text features Calculate using equations (6) and (7) in, express and Similarity score, express and Let the similarity score be the similarity score of the j-th target domain user image. logit value Pseudo-labels

[0075]

[0076] Use a repository B to store image features and image logit value Right now Sort the samples according to their Shannon entropy S, and remove the top s samples with the largest Shannon entropy from B. Among them, for The formula for calculating Shannon entropy is shown in equation (8):

[0077]

[0078] In the selected repository B, obtain the prototype for each category, and the prototype c for category k. k Defined as in equation (9):

[0079]

[0080] Here, 'I' represents an indicator function that outputs 1 if the input equation is true, and 0 otherwise. Based on C... k Classification output The definition is as shown in equation (10):

[0081]

[0082] Then, based on prototype c k Classification output results and logit value Calculate the cross-entropy loss L j To ensure that the same input yields the desired result. and They have similar distributions. The loss L of the prototype constraint part pc Defined as in equations (11)-(13):

[0083]

[0084] Nearest neighbor constraint loss: In the selected repository, embodiments of the invention minimize image features. K nearest neighbor image features The cosine similarity is used to align the logit values ​​of the sample. K nearest neighbors The loss L of the nearest neighbor constraint part nnc The definition is as shown in equation (14):

[0085]

[0086] In this embodiment, L is balanced using α and β. pc ,L nnc The loss, weighted summation, is the overall loss L of the fine-tuning process. ft As in equation (15):

[0087] L ft =α*L pc +β*L nnc (15)

[0088] Step 4: Apply the enhanced model M2 to the target domain dataset D. t In Make predictions;

[0089] Step 4.1 Select the region of interest Image encoder for the enhanced model M2 The enhanced image features of the j-th target domain user are obtained through processing.

[0090] Step 4.2 Calculation and similarity score in, express and Similarity score, express and The similarity score;

[0091] like Then output Predicted classification results Otherwise, output Predicted classification results

[0092] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the methods described above, and the processor is configured to execute the program stored in the memory.

[0093] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0094] To quantitatively evaluate the classification performance of the method of this invention in clinical practice, a multicenter, retrospective study was conducted based on this method. The classification performance evaluation metrics for the method of this invention are: accuracy (Acc), area under the ROC curve (AUC), and sensitivity (Sen).

[0095] Acc refers to the proportion of samples correctly predicted by the model out of the total number of samples. It measures the overall classification accuracy of the model. The higher the value, the better. Its definition is as shown in equation (16):

[0096]

[0097] TP: True Positive, which is the number of samples correctly predicted as positive; TN: True Negative, which is the number of samples correctly predicted as negative; FP: False Positive, which is the number of samples incorrectly predicted as positive; FN: False Negative, which is the number of samples incorrectly predicted as negative.

[0098] AUC refers to the area under the receiver operating characteristic (ROC) curve. The ROC curve is plotted using the true positive rate and false positive rate at different thresholds. The AUC value is between 0 and 1, with a higher value indicating better model performance.

[0099] Sen (also known as recall) refers to the model's ability to correctly identify positive examples; that is, the proportion of samples that are actually positive that are correctly predicted as positive. A high Sen means that the model can identify more actual positive examples. Its definition is as shown in equation (17):

[0100]

[0101]

[0102] Table 1 shows the classification performance results of the deep learning model-baseline version and the deep learning model-enhanced version used in this invention for preoperative prediction of duodenal invasion (MDI) status in pancreatic cancer patients from the training set, internal test set and external test set.

[0103] In Table 1, the training set and the internal validation set are derived from the same abdominal CT dataset by a random partitioning ratio of 4:1. They can be considered as source domain datasets and used for pre-training and validation of the baseline model (the DPDI-base). The external test sets 1 and 2 are independently composed of image center data from different sources than the aforementioned abdominal CT dataset. They can be considered as target domain datasets and used for testing the enhanced model (the DPDI-enhanced).

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0105] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A CT image classification method based on contrastive language pre-training and test-time adaptation, characterized in that, The procedure is as follows: Step 1: Collect and preprocess data from... Abdominal CT images and category labels from several imaging centers were randomly selected. Preprocessed abdominal CT images from several imaging centers were used as the source domain dataset. The rest Preprocessed abdominal CT images from several imaging centers were used as the target domain dataset. ,in, Representative source domain dataset The Middle Example: Abdominal CT images preprocessed by the source domain user. represent The category label, and In this context, 0 represents normal and the corresponding abdominal CT image is used as a negative sample, while 1 represents abnormal and the corresponding abdominal CT image is used as a positive sample. Represents the target domain dataset The Middle Example: Preprocessed abdominal CT image of the target domain user. , Representing the source domain dataset respectively and target domain dataset The total number of images; Step 2 In the source domain dataset Above, the baseline model based on contrastive language pre-training. Pre-training is performed to obtain a pre-trained baseline model. ; Step 2.1 Utilize the pre-trained segmentation model right Process it to obtain the first Example: Region of Interest for Source Domain Users ; Step 2.2 Select the region of interest. Perform classification preprocessing to obtain the first... Example: Region of Interest after preprocessing for user classification in the source domain ; Step 2.3 Input baseline model Image encoder in Processing yields the first... Image features of source domain users ; Initialize a set of vectors to be learned ,in, As a hint for positive samples, This serves as a hint for negative samples; Will Input baseline model Text encoder in The text features are processed to obtain a set of text features. ;in, The text features are those of positive samples. Text features for negative samples; Step 2.4 Calculation and A set of similarity scores ;in, express and Similarity score, express and The similarity score; Step 2.5 Construction and Cross-entropy loss between, and in the frozen text encoder Under the given parameters, for the baseline model Pre-training is performed, and the cross-entropy loss is calculated to update... Parameters and image encoder The parameters are used to obtain the pre-trained learning vectors. Pre-trained image encoder and the frozen text encoder Together they form the pre-trained baseline model. ,in, As a hint after positive sample pre-training As a hint after negative sample pre-training; Step 2.6 Utilize the pre-trained baseline model Output Predicted classification results: when When the time is right, the predicted classification result will be output. =1, otherwise, output the predicted classification result. =0; Step 3: In the target domain dataset Above, the baseline version of the pre-trained model Some parameters were fine-tuned adaptively during testing to obtain an enhanced model. ; Step 4: Utilize the enhanced model For the target domain dataset In Make a prediction and obtain Predicted classification results .

2. The CT image classification method based on contrastive language pre-training and adaptive testing time according to claim 1, characterized in that, Step 3 is performed as follows: Step 3.1 Utilize the pre-trained segmentation model right Processing is performed to obtain the region of interest for the j-th target domain user. ; Step 3.2 Select the region of interest. Perform classification preprocessing to obtain the region of interest for the j-th target domain user after classification preprocessing. ; Step 3.3 will Input the pre-trained baseline model Image encoder The image features of the j-th target domain user are obtained through processing. ; Will Input the pre-trained baseline model Text encoder The pre-trained text features are then processed to obtain them. ;in, Text features used as positive samples after pre-training Text features used as negative samples after pre-training; Step 3.4 Construct a test-time adaptive loss consisting of prototype constraints and nearest neighbor constraints, and freeze the text encoder. Parameters and Under the given parameters, the pre-trained baseline model Image encoder in Fine-tuning was performed, and the image encoder was updated by calculating the test-time adaptive loss. The parameters are used to obtain the fine-tuned image encoder. and frozen and text encoder Together they form an enhanced model .

3. The CT image classification method based on contrastive language pre-training and adaptive testing time according to claim 2, characterized in that, Step 4 is performed as follows: Step 4.1 Select the region of interest Input Enhanced Model Image encoder The enhanced image features of the j-th target domain user are obtained through processing. ; Step 4.2 Calculation and similarity score ,in, express and Similarity score, express and The similarity score; like Then output Predicted classification results =1, otherwise, output Predicted classification results =0.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the CT image classification methods of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by the processor, performs the steps of any of the CT image classification methods described in claims 1-3.

Images