Intestinal polyp classification method based on improved CLIP model

Abstract

The invention belongs to the technical field of medical image processing, and discloses an intestinal polyp classification method based on an improved CLIP model. Comprising the steps of collecting a data set and screening and cutting collected images; an SE attention mechanism is added in the Vision Transform block, and polyp image features are learned; the method comprises the following steps: introducing a pathological description text labeled by a professional doctor into Adpater-BERT, and learning pathological character features; and the two features are fused by adopting adaptive weight, and finally classification is carried out. Through the optimization strategy, the improved CLIP model can improve the accuracy of classification of adenomatous polyps and proliferative polyps while keeping relatively low calculation overhead.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and particularly relates to a method for classifying intestinal polyps based on an improved CLIP model. Background Art

[0002] The incidence of intestinal diseases has been increasing year by year, mainly due to the increasing stress level and irregular work-rest patterns. Intestinal polyps are a common intestinal lesion. If not detected and treated in time, they may develop into colon cancer, becoming the main cause of cancer-related deaths globally. Research shows that most colon cancers originate from adenomatous polyps. Therefore, early detection and removal of adenomatous polyps are crucial for reducing mortality. In contrast, hyperplastic polyps are generally considered benign lesions. Therefore, accurately distinguishing adenomatous polyps from hyperplastic polyps during colonoscopy has important clinical significance.

[0003] Traditional colonoscopy mainly relies on doctors' experience and visual observation, suffering from problems such as insufficient detection accuracy, low efficiency, and inconsistent diagnostic criteria. The morphology of intestinal polyps is diverse, which easily leads to misdiagnosis, and image processing diagnosis usually requires a large amount of time and human resources. To solve these problems and improve the speed and accuracy of polyp classification, computer-aided diagnosis systems (CAD) have gradually been widely used. With the continuous development of machine learning and deep learning algorithms in the field of image classification, the application of computer technology in medical image analysis has gradually received high attention from academia and clinical practice.

[0004] Although deep learning methods have made significant progress in some aspects, they still face two problems that need to be solved urgently: one is that the classification accuracy has not yet reached the requirements of clinical applications; the other is that the unbalanced distribution of the dataset has a negative impact on the model performance. Summary of the Invention

[0005] To overcome the problems that the existing manual polyp classification requires high professional ability and has insufficient judgment accuracy, and that the existing classification models and methods are difficult to meet the requirements of clinical applications and have high complexity, the present invention provides a method for classifying intestinal polyps based on an improved CLIP model, aiming to combine the pathological text information annotated by professional doctors with the polyp image data to improve the accuracy and reliability of the classification model, so as to provide more accurate decision-making support for clinical diagnosis. In addition, a high-quality polyp image dataset is constructed, and the model performance is further optimized through sample balancing technology, in order to provide more effective technical support for the early detection and clinical treatment of intestinal polyps.

[0006] The above object of the present invention is achieved by the following technical solutions: A method for classifying intestinal polyps based on an improved CLIP model, comprising the following steps:

[0007] S1: Collect the dataset and screen and crop the collected images;

[0008] S2: Add the SE attention mechanism to the Vision Transformer block to learn the polyp image features;

[0009] S3: Introduce the pathological description text annotated by professional doctors into Adpater-BERT to learn the pathological text features;

[0010] S4: Use adaptive weights to fuse the two types of features and finally perform classification.

[0011] Furthermore, the specific content of S1 is as follows:

[0012] S1.1 Dataset collection: The self-collected dataset of adenomatous polyps and hyperplastic polyps is used as the dataset.

[0013] S1.2 Screening: For the collected polyp data, filter out the incomplete and unclear polyp parts in the images and retain the complete and clear polyp data.

[0014] S1.3 Cropping: For the polyp images retained after screening, crop them into images of appropriate sizes. For the processed dataset, expand the dataset through data augmentation methods - color transformation, rotation, and adding Gaussian noise.

[0015] Furthermore, the specific content of S2 is as follows:

[0016] S2.1 Feature extraction and preliminary processing: Use the standard Vision Transformer structure. First, perform preliminary feature extraction on the input polyp images to generate a preliminary representation of the images. The Vision Transformer first divides the input polyp images into multiple small patches and flattens each small patch into a one-dimensional vector. Then, these small patches are embedded into a high-dimensional space through a linear transformation, and position encoding is added to retain the spatial information of each image patch. Next, the features of the image patches are processed through the Transformer encoder layer to capture the dependencies between the various patches in the image, and finally, a global representation of the image is generated.

[0017] S2.2 Introduce the SE attention mechanism: In each encoder layer of the Vision Transformer, add the SE attention mechanism to enhance the model's ability to focus on key features. Apply global average pooling to each feature map to obtain the global information of each channel, and calculate the importance weights of each channel through a fully connected layer and an activation function. Finally, multiply the calculated weights by the original feature map to achieve channel-level feature recalibration.

[0018] S2.3 Feature Weighting and Optimization: Apply the channel weighting information generated by the SE attention mechanism to the original feature map, enhance the feature responses in key regions, and suppress the influence of irrelevant regions. Amplify the channel features with higher weights, and the SE attention mechanism increases their expressiveness in the feature map, enabling the subsequent classification model to better focus on this region. For regions with irrelevant or large interference, the SE attention mechanism assigns lower weights to them.

[0019] S2.4 Position Encoding and Multi-Head Self-Attention Mechanism: In the fused feature map, add position encoding to retain spatial position information. Then input it into the multi-head self-attention mechanism to capture global dependencies. The feature map processed by the self-attention mechanism is input into a feed-forward neural network for non-linear transformation and normalized by Layer Normalization. Assume the input feature map is X = [x1, x2,......, x d , where d is the dimension of the feature, and the layer normalization operation is as follows: First, calculate the mean μ and variance σ 2 , Then perform a normalization operation on each feature so that their mean is 0 and variance is 1. Finally, apply the scaling parameter γ and offset parameter β to the normalized output to enable the model to recover some important feature scales.

[0020] S2.5 Residual Connection and Output: Add a residual connection between the input and output of each encoder layer to ensure the effective transmission of information. Finally, the feature map processed by multiple encoder layers is used for subsequent classification tasks.

[0021] Further, the specific content of S3 is as follows:

[0022] S3.1 Preprocessing and Encoding of Pathological Texts: Preprocess the pathological description texts annotated by doctors - perform word segmentation, remove noise characters, and generalize common descriptions, and input the processed texts into a pre-trained BERT model for encoding to extract the semantic features of the texts.

[0023] S3.2 Text Feature Extraction Using BERT: Input the processed pathological description texts into the BERT model, use the BERT model to extract the semantic features of the texts, capture the context information in the texts, and generate high-dimensional text representations.

[0024] S3.3 Fine-tune BERT: Use the Adapter technique to fine-tune the BERT model to learn and adapt to task-specific knowledge. The fine-tuning specifically enhances the correlation between pathological descriptions and image features and improves the classification accuracy. Through fine-grained parameter adjustment, BERT enhances its understanding of professional terms while retaining its original general pre-training ability.

[0025] Further, the specific content of S4 is as follows:

[0026] S4.1 Adaptive weight mechanism: According to the contributions of image and text features to the final classification result during the training process, evaluate the importance of each feature through the attention mechanism or gradient information during the training process. Use the obtained weights to weight the image and text features. The image features and text features are multiplied by their respective weights and then fused, enabling the model to dynamically adjust the contributions of the image and text to the classification task. During the evaluation process, the importance of the features is measured by the gradient information in the loss function, and the gradient backpropagation helps the model identify the impact magnitude of each feature on the classification result. At this time, the weights of the image and text are further optimized through the adaptive learning rate to ensure that each feature makes a reasonable contribution in the model training. The evaluation benchmarks are measured by metrics such as accuracy, F1-score, and AUC curve to ensure that the weighted features can effectively improve the classification performance of the model.

[0027] S4.2 Feature fusion: Multiply the weighted image and text features element by element to highlight the mutual dependence between the two - F fusion = F image ⊙ F text , capturing the interrelationship between the image and the text.

[0028] S4.3 Feed the processed features into the Softmax classifier to output the final class prediction: The Softmax layer maps the final feature vector to the confidence levels of different classes by calculating the probability distributions of each class, thereby making a classification decision.

[0029] The beneficial effects of the present invention compared with the prior art are as follows: The CLIP model is introduced into the improved architecture. By combining image features and text features, an improved CLIP model is constructed for polyp classification. The polyp image is first input into the network, and the image features are extracted layer by layer through the Vision Transformer (ViT) block and the SE attention mechanism module. Then, the pathological description text is encoded by the BERT model to extract text features. Next, an adaptive weight mechanism is used to weight and fuse the image and text features, enabling the model to dynamically adjust the contributions of the two in the classification task. Finally, through feature fusion and the Softmax classifier, the final classification result is output. Through this optimization strategy, the improved CLIP model can improve the accuracy of adenomatous polyp and hyperplastic polyp classification while maintaining a low computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described below in conjunction with the drawings and specific embodiments.

[0031] Figure 1 It is the overall architecture diagram for polyp classification. Among them, Image Encoder is Vision Transformer, Text Encoder is BERT, and the processed features are fused through adaptive weights.

[0032] Figure 2 It is the specific structure of Image Encoder. The SE attention mechanism module is inserted before the multi-layer perceptron of the Vision Transformer block to make the model pay more attention to the polyp lesion area.

[0033] Figure 3 It is the structure diagram of the SE attention module. An adaptive channel weighting mechanism is introduced into the network to help the model dynamically adjust and select important feature channels, thereby improving performance. SPECIFIC EMBODIMENTS

[0034] The present invention will be described in detail below through specific embodiments, but the protection scope of the present invention is not limited. Unless otherwise specified, the experimental methods used in the present invention are all conventional methods, and the experimental equipment, materials, reagents, etc. used can all be obtained from commercial channels.

[0035] Example 1

[0036] This example proposes a method for classifying intestinal polyps based on the CLIP model, applying the SE attention module, and fine-tuning the CLIP model.

[0037] This research is divided into two parts: feature extraction and classification processing. In the data preprocessing stage, Vision Transformer is used to extract the features of polyp images, and data augmentation methods are used to expand the dataset. The Vision Transformer block is redesigned according to the idea of convolutional neural network, and the SE attention module is added before the multi-layer perceptron to make the model pay more attention to the lesion area. Finally, the linear classifier Softmax is used to classify the features and output the results.

[0038] S1: Collect the dataset and screen and crop the image data;

[0039] S1.1 Dataset self-collection.

[0040] The dataset used is a self-collected dataset. Images are collected from patients visiting Xinhua Hospital Affiliated to Dalian University using the Olympus colonoscope PCF-H290, with a resolution of 1080P. All patient data have passed the ethical review and desensitization processing by the hospital's information department. The initial dataset is as shown in the appendix, Figure 1 divided into adenomatous polyps and hyperplastic polyps, Figure 1 shows some images of hyperplastic polyps and adenomatous polyps taken with the Olympus intestinal endoscope PCF-H290. These images are the main data sources in the research. Taken with high-resolution endoscope equipment, they can provide delicate images of the lesion areas of intestinal polyps, ensuring the authenticity and clinical relevance of the data. The description of each image is annotated by professional doctors, explaining information such as its location and size, as shown in the appendix Figure 2 shown. Figure 2 is the pathological text description annotated by professional doctors, which details the morphological characteristics, location, size, and other important pathological information of different types of polyps. Each pathological description corresponds to an image, providing the pathological information behind the image data. These pathological descriptions are generated by doctors' professional knowledge and clinical experience in interpreting polyp images, greatly enhancing the model's understanding of the lesion area.

[0041] S1.2 Screening.

[0042] For the collected polyp data, the parts of polyps with incomplete shooting and unclear images are screened out, and the complete and clear polyp data are retained.

[0043] S1.3 Cropping. For the polyp images retained after screening, considering comprehensively, they are cropped into images of appropriate size, and the processed dataset is expanded through data augmentation methods.

[0044] S2: Add the SE attention mechanism to the Vision Transformer block to learn the features of polyp images;

[0045] Considering the problems of low distinguishability between the two types of polyp images and few effective features, the SE attention module is introduced into the Vision Transformer, and the relevant definitions are as follows:

[0046] S2.1 Vision Transformer

[0047] Vision Transformer is an image classification method based on the Transformer architecture, which mainly processes images through the following steps: The image is segmented into patches of a fixed size, and each patch is flattened and embedded into a vector space of a fixed dimension; A standard Transformer module is used for feature extraction, and the relationship between each patch in the image is modeled through the self-attention mechanism; Finally, the extracted features are classified through a classification head.

[0048] S2.2 Squeeze operation of the SE module

[0049] First, given the input feature map X ∈ R H×W×C , where H is the height of the image, W is the width, and C is the number of channels. Usually, the spatial information (height H and width W) of the image contains detailed features, while the channel information C corresponds to different feature abstraction levels.

[0050] The goal of the Squeeze operation is to aggregate the information of these spatial dimensions into a channel-level description, that is, to compress the spatial information to focus on the overall features of each channel.

[0051] The specific operation is completed through global average pooling (GAP). It calculates the average value for each channel to form a vector z ∈ R C :

[0052] where x i,j,c is the feature map value at position (i, j), and z c is the global information of channel c, and the spatial features of this channel are averaged and compressed through the entire space H×W.

[0053] Therefore, the result of the Squeeze operation is to generate a vector of length C, and each element represents the global information at the channel level.

[0054] S2.3 Excitation operation of the SE module

[0055] The Excitation operation processes the compressed information through a fully connected layer to generate the weights for each channel, indicating the importance of that channel in the final classification. The process of Excitation includes two fully connected layers and the ReLU activation function, plus the Sigmoid activation function to normalize the output weights.

[0056] First, the compressed feature vector z ∈ R C is projected onto a smaller dimension C / r through a fully connected layer, where r is a scaling factor, usually between 4 and 16. This smaller-dimensional space allows the network to learn more compact feature representations: z' = W1·z, where W1 ∈ R (C / r)×C is the weight matrix of the fully connected layer. Then, the activation function ReLU is applied to z' to introduce non-linear features, enabling the model to capture more complex relationships: z' ReLu = ReLu(z'). Subsequently, the vector z' ReLu after ReLu activation is fed into the second fully connected layer, which projects it back to the original channel dimension C, generating the weight coefficients s ∈ R C for each channel: s = σ(W2·z' ReLu ), where W2 ∈ R C×(C / r) is the weight matrix of the second fully connected layer, and σ represents the Sigmoid activation function which restricts the output value within the range [0,1]. The output s is the importance weight for each channel. Through the Excitation operation, the finally obtained weight vector s assigns a weight coefficient to each channel, indicating the contribution of that channel to the final task: where is the weighted feature map, and s c is the weight of channel c.

[0057] S2.4 Feature recalibration of the SE module

[0058] By multiplying the feature map of each channel with the corresponding weight s c , the weighted feature map is obtained. This process is called feature recalibration. It optimizes the representation ability of the network by dynamically assigning weights to each channel, enabling important features to be strengthened while irrelevant features are suppressed: This step helps the network learn more relevant features and improve the robustness of the model by weighting each channel.

[0059] By adding the SE module to the Vision Transformer, the network can automatically select and weight important features. By learning the "importance" of each channel, the SE module enhances the model's ability to focus on key regions, especially in polyp images with low distinctiveness.

[0060] S3: Introduce the pathological description text annotated by professional doctors into Adapter-BERT to learn pathological text features;

[0061] Before learning text features, it is first necessary to preprocess the pathological description text annotated by doctors. Decompose the pathological description text into word or sub-word units, as shown Figure 3 below. Figure 3 The figure shows the flowchart of the BERT architecture for processing pathological text descriptions. In this process, the pathological text is first preprocessed, including operations such as word segmentation and removal of noise characters, and then the processed text is input into the pre-trained BERT model. The BERT model extracts semantic information from the pathological description through its powerful context understanding ability and captures the key context relationships in the text. After preprocessing the pathological text, we input it into Adapter-BERT for feature extraction. BERT is a language model based on the Transformer architecture that can understand the semantics of words through context relationships. Its input is the text after Tokenization and word embedding processing, and the goal is to generate the semantic representation of the text through multiple layers of Transformer encoders. The input of the BERT model is a sequence composed of multiple Tokens, and each Token is represented by its corresponding word embedding vector. Suppose the input text Token sequence is X = [x1, x2,..., x T , where T is the text length, and x i ∈ R d is the word vector of each Token (the dimension d is the BERT embedding dimension, usually 768).

[0062] Each layer of the Transformer encoder in BERT mainly includes: the self-attention mechanism - this mechanism enables the model to learn the dependencies between Tokens, thereby enhancing context information. The representation h i of each Token is calculated through the formula where Q is the Query matrix, K is the Key matrix, V is the Value matrix, and d kis the dimension of the key; the feedforward neural network - the output after being processed by the attention mechanism, passes through a non-linear transformation layer to further extract the high-dimensional semantic features of the pathological text; positional encoding - since Transformer itself does not have the ability to perceive positions, BERT uses positional encoding to introduce the sequential information of the tokens. After being processed by multiple layers of Transformer, each token output by BERT will have a corresponding context-related representation, forming an embedded representation of the text.

[0063] To make BERT more adaptable to the intestinal polyp classification task, we adopted the Adapter technique. In each layer of BERT, a small adapter network is inserted, enabling the model to adapt to task-specific knowledge (such as the pathological terms of intestinal polyps) by fine-tuning a small number of parameters. First, the output of BERT is reduced in dimension through a fully connected layer - h i ' = W adapter h i + b adapter , then activation functions such as ReLU are applied to enhance the expressive power of the network - Finally, the representation after being processed by the adapter network is mapped back to the output dimension of BERT to combine with the high-level features output by BERT.

[0064] Through Adapter-BERT, the network will learn how to generate features related to polyp images based on the semantic information in the pathological description text (such as medical terms like "Tubular adenoma", "low-grade intraepithelial neoplasia", etc.). These text features will help the model understand the types of lesions or polyp classifications present in the image.

[0065] Specifically, Adapter-BERT captures the semantic relationships and context information in the pathological text through the self-attention mechanism, learns the importance of each token in the context, and generates a feature vector z with rich semantic information: z = Adapter-BERT(X), where z ∈ R C is the feature representation learned from the text, and C is the feature dimension.

[0066] In summary, by introducing pathological description text into Adapter-BERT, the model can extract deep semantic information from medical texts and combine it with image features, thereby effectively improving the accuracy of polyp classification. Through adapter fine-tuning, the model can quickly adapt to the characteristics of the intestinal polyp classification task with limited task-related parameters, further enhancing the ability of multimodal learning.

[0067] The above-described embodiments are only the preferred embodiments of the present invention, and not all the feasible embodiments of the present invention. For those of ordinary skill in the art, any obvious changes made without departing from the principles and spirit of the present invention should be considered to be included within the scope of the claims of the present invention.

Claims

1. A method for classifying intestinal polyps based on an improved CLIP model, characterized in that: The following steps are involved: S1: Collect data sets and screen and crop collected images; S2: Add SE attention mechanism in Vision Transformer block to learn polyp image features; S3: Introduce the pathology description text annotated by professional doctors into Adpater-BERT to learn the pathology text features; S4: Adaptive weights are used to fuse the two features and finally classify them.

2. The intestinal polyp classification method based on the improved CLIP model according to claim 1, characterized in that: The step S1 is specifically as follows: S1.1 Dataset collection: The dataset used is a self-collected dataset of adenomatous polyps and hyperplastic polyps; S1.2 Screening: For the collected polyp data, the polyp parts with incomplete images and unclear images are screened out, and the complete and clear polyp data are retained; S1.3 Cropping: For the polyp images retained after screening, they are cropped into images of appropriate size. For the processed data set, the data set is expanded through data enhancement methods such as color transformation, rotation, and adding Gaussian noise.

3. The intestinal polyp classification method based on the improved CLIP model according to claim 1, characterized in that: The step S2 is specifically as follows: S2.1 Feature extraction and preliminary processing: Using the standard VisionTransformer structure, we first perform preliminary feature extraction on the input polyp image to generate a preliminary representation of the image. The Vision Transformer first divides the input polyp image into multiple small blocks and flattens each small block into a one-dimensional vector. Then, these small blocks are embedded into a high-dimensional space through linear transformation, and position encoding is added to retain the spatial information of each image block. Next, the features of the image block are processed through the Transformer encoder layer to capture the dependencies between the blocks in the image, and finally generate a global representation of the image. S2.2 introduces SE attention mechanism: In each encoder layer of VisionTransformer, SE attention mechanism is added to enhance the model's ability to focus on key features, global average pooling is applied to each feature map, global information of each channel is obtained, and the importance weight of each channel is calculated through the fully connected layer and activation function; finally, the calculated weight is multiplied with the original feature map to achieve channel-level feature recalibration; S2.3 Feature weighting and optimization: Apply the channel weighting information generated by the SE attention mechanism to the original feature map to strengthen the feature response of the key area and suppress the influence of irrelevant areas; amplify the channel features with higher weights, and the SE attention mechanism increases their expressiveness in the feature map, so that the subsequent classification model can better pay attention to this area; for irrelevant or highly interfering areas, the SE attention mechanism will assign lower weights to them; S2.4 Position encoding and multi-head self-attention mechanism: In the fused feature map, position encoding is added to retain the spatial position information; then it is input into the multi-head self-attention mechanism to capture global dependencies; The feature map processed by the self-attention mechanism is input into the feedforward neural network for nonlinear transformation and standardized by layer normalization. Assuming that the input feature map is X = [x1, x2, ..., x d ], where d is the dimension of the feature, the layer normalization operation is as follows: first calculate the mean μ and variance σ 2 , Then, each feature is standardized so that its mean is 0 and its variance is 1. Finally, the scaling parameter γ and the offset parameter β are applied to the standardized output so that the model can recover some important feature scales. S2.5 Residual connection and output: A residual connection is added between the input and output of each encoder layer to ensure the effective transmission of information; finally, the feature map processed by multiple encoder layers is used for subsequent classification tasks.

4. The intestinal polyp classification method based on the improved CLIP model according to claim 1, characterized in that: The step S3 is specifically as follows: S3.1 Pathology text preprocessing and encoding: Preprocess the pathology description text annotated by doctors - word segmentation, noise character removal, and summarization of common descriptions. Input the processed text into the pre-trained BERT model for encoding and extracting the semantic features of the text. S3.2 Use BERT for text feature extraction: Input the processed pathology description text into the BERT model, use the BERT model to extract the semantic features of the text, capture the contextual information in the text, and generate a high-dimensional text representation; S3.3 Fine-tune BERT: Use Adapter technology to fine-tune the BERT model to learn and adapt task-specific knowledge; the fine-tuning is to enhance the correlation between pathological descriptions and image features and improve classification accuracy; through fine-grained parameter adjustment, BERT enhances its understanding of professional terms while retaining its original general pre-training capabilities.

5. The intestinal polyp classification method based on the improved CLIP model according to claim 1, characterized in that: The step S4 is specifically as follows: S4.1 Adaptive weight mechanism: According to the contribution of image and text features to the final classification results during training, the importance of each feature is evaluated through the attention mechanism or the gradient information during training; Weighting the image and text features using the obtained weights; Image features and text features are multiplied by corresponding weights and then fused, so that the model can dynamically adjust the contribution of images and texts to the classification task. During the evaluation process, the importance of features is measured by the gradient information in the loss function, and gradient backpropagation helps the model identify the impact of each feature on the classification result. At this time, the weights of images and texts are further optimized through adaptive learning rates to ensure that each feature makes a reasonable contribution in model training. The evaluation benchmark is measured by indicators such as accuracy, F1-score, and AUC curve to ensure that weighted features can effectively improve the classification performance of the model. S4.2 Feature Fusion: Multiply the weighted image and text features element by element to highlight the interdependence between the two - F fusion =F image ⊙F text ,capturing the interrelationship between images and text; S4.3 sends the operated features to the Softmax classifier and outputs the final category prediction: The Softmax layer calculates the probability distribution of each category and maps the final feature vector to the confidence of different categories, thereby making a classification decision.

Images