A method for identifying a lesion region in a digestive tract image based on a visual language model

Abstract

This invention discloses a method for identifying lesion regions in gastrointestinal images based on a visual language model. It combines the cross-modal feature alignment capability of the visual language model, a medical semantic guidance mechanism, and a data imbalance handling strategy. By constructing a matching relationship between images and corresponding text categories, it achieves the identification of gastrointestinal lesions. This method introduces a bidirectional contrastive learning loss, enhancing the discriminative ability of different lesion categories while maintaining consistency between image features and text semantics. Furthermore, to address the uneven distribution of lesion categories in the training data, this invention introduces a category weighting mechanism and sampling strategy, allowing minority class samples to obtain higher weights or higher frequencies during training, effectively improving the identification accuracy and model stability of minority class lesions. During the inference stage, classification is completed by calculating the similarity between the image and all categories of text features, achieving the identification and classification of various gastrointestinal endoscopic images without the need for additional labeled data or complex training.

Description

Technical Field

[0001] This invention relates to a method for identifying lesion areas in digestive tract images based on a visual language model. It can be applied to the automatic analysis and assisted diagnosis of gastroscopy, colonoscopy and capsule endoscopy images, and belongs to the application of artificial intelligence technology in the field of medical imaging. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, deep learning-based medical image analysis methods have been widely used in the auxiliary diagnosis of digestive tract diseases. Endoscopic images of the digestive tract (such as gastroscopy, colonoscopy, and capsule endoscopy) can directly reflect lesion areas and are important evidence for the clinical diagnosis of polyps, bleeding, inflammation, and other diseases. Traditional image interpretation mainly relies on physician experience, which suffers from high workload, strong subjectivity, and limited diagnostic efficiency. To improve diagnostic efficiency and accuracy, researchers have proposed various automatic recognition methods, especially models based on convolutional neural networks, which have made some progress in feature extraction and classification tasks. In recent years, visual language models, by jointly modeling image and text information, have achieved cross-modal semantic alignment, demonstrating strong capabilities in weakly supervised learning and zero-shot recognition tasks, providing a new technical path for intelligent medical image analysis.

[0003] However, existing technologies still have many shortcomings. On the one hand, traditional deep learning methods typically rely on large-scale, high-quality labeled data, but medical image labeling is costly and difficult to obtain, limiting further improvements in model performance. On the other hand, the application of visual language models in the medical field is still in the exploratory stage. Due to the high complexity and specialization of medical images, general visual language models are prone to semantic bias and recognition instability, making it difficult to directly meet the needs of clinical applications. Furthermore, in real-world data, the distribution of various lesion samples is often significantly imbalanced, leading to insufficient model recognition ability for a minority of lesions, further affecting overall performance. Therefore, there is an urgent need for a digestive tract lesion recognition method that combines visual language models and is optimized for medical scenarios, in order to reduce the dependence on labeling while improving the model's recognition accuracy and stability under imbalanced data conditions. Summary of the Invention

[0004] The purpose of this invention is to provide a method for identifying lesion regions in gastrointestinal images based on a visual language model. By introducing a cross-modal contrastive learning mechanism and an imbalanced data processing strategy, the method reduces the dependence on large-scale labeled data while improving the accuracy and stability of the visual language model in the task of identifying lesions in gastrointestinal images, especially enhancing the ability to identify rare lesions.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for identifying lesion regions in gastrointestinal images based on a visual language model is proposed. This method utilizes a visual language model to jointly model images and medical semantic text, achieves cross-modal feature alignment by optimizing the image encoder, and improves model performance by incorporating an imbalanced data processing strategy. The method specifically includes the following steps:

[0007] S1. Obtain gastrointestinal imaging data: Obtain gastrointestinal endoscopic images of each lesion category as samples for each lesion category, and count the number of samples for each lesion category; classify each lesion category into rare levels according to the number of samples, and set a minimum value for the number of text descriptions for each rare level.

[0008] The digestive tract endoscopy images include raw data images from gastroscopy, colonoscopy, capsule endoscopy, etc. The raw data images undergo uniform preprocessing, including size normalization, brightness and contrast enhancement, noise reduction, and standardization, so that the images meet the input requirements of the visual language model. In order to verify the recognition performance and robustness of the model under different image modalities, an additional grayscale preprocessing verification branch can be set up to convert the original RGB color endoscopy images into single-channel grayscale images, while keeping the size, normalization, enhancement, and other preprocessing steps completely consistent, forming a grayscale sample set.

[0009] The sample set is traversed and the number of samples in each lesion category is counted to obtain data distribution information, thereby evaluating the rarity level of each lesion category; each sample is divided into an independent folder of its respective lesion category to ensure that the samples contained in each lesion category are stored independently, easy to load and divide; through image filtering, format unification and quality verification, the integrity and standardization of the sample set are guaranteed, providing a reliable data foundation for model training and inference.

[0010] S2. Constructing textual descriptions of lesion categories: for the first... A set of standardized medical semantic text descriptions is constructed for each disease category. This forms a set of text descriptions. ;in: , This indicates the total number of disease categories; Indicates the first The first lesion category Standardized medical semantic text description, Indicates the first The total number of standardized medical semantic text descriptions for each lesion class;

[0011] The standardized medical semantic text descriptions should align with clinical expressions and be used to represent the semantic features of different disease categories, enabling the visual language model to understand the meaning of disease categories. The standardized medical semantic text descriptions should adopt a unified sentence structure to ensure semantic consistency and provide stable supervision signals for subsequent image-text comparison learning.

[0012] S3. Feature extraction using a visual language model: [The sample...] and text description set The input is fed into a pre-trained visual language model, and samples are extracted through an image encoder. Image features Extracting text description sets using a text encoder Text features During training, all parameters of the text encoder are frozen, and only the image encoder performs parameter updates and gradient backpropagation. While maintaining the stability of the text semantic space, this significantly reduces the number of training parameters and computational cost, improves training efficiency, and reduces the risk of overfitting.

[0013] S4. Constructing image-text matching relationships: For ease of description, samples... Corresponding text description set Recorded as Corresponding text features Recorded as Construct positive and negative sample pairs. Positive sample pairs are built from samples and their corresponding text descriptions, while negative sample pairs are built from samples and other text descriptions, forming complete cross-modal contrastive learning training data. For samples... Positive sample pairs are Negative samples are ;

[0014] By clearly defining positive and negative samples, a clear alignment supervision signal is provided to the model, so that image features tend to be closer to similar text features and farther away from dissimilar text features during training, thereby improving the distinguishability of different lesion categories in the feature space.

[0015] S5. Cross-modal alignment and imbalance handling during training: First, normalize the image features and text features separately, and then calculate the sample using the normalized inner product. With text description set similarity Then, a similarity matrix is ​​constructed based on the similarity, and the bidirectional contrast loss between text and image is calculated. Total loss Finally, based on the total loss Perform backpropagation optimization on the image encoder;

[0016] By jointly optimizing the loss in both the image-to-text and text-to-image directions, image features and corresponding text features are closely aligned in a unified feature space, while the feature distance between different categories is increased, which can significantly improve the discrimination ability of lesion identification.

[0017] S6. Lesion area identification during the inference phase: For a given image First, extract the given image using an image encoder. Image features Then calculate its relationship with each set of text descriptions. The normalized similarity of each standardized medical semantic text description in the text. ,in: Then, for each lesion category, the average similarity of all standardized medical semantic text descriptions is taken, and the lesion category with the highest average similarity is selected as the prediction result. The predicted lesion category is output as follows: It enables fully automated, high-precision, and rapid identification and classification of digestive tract lesions.

[0018] Specifically, in step S5, similarity Two-way comparison loss Total loss The calculation formula is:

[0019]

[0020]

[0021]

[0022] in: Indicates the first The weight of each lesion category, , This indicates the total number of samples.

[0023] The bidirectional contrast loss includes loss in the image-to-text direction and loss in the text-to-image direction. By jointly optimizing the losses in the two directions, the cross-modal consistency between image features and text features can be enhanced, and the discrimination ability between different lesion categories can also be improved.

[0024] Specifically, in step S2, the lesion is classified according to its rarity level as follows: A set of standardized medical semantic text descriptions is constructed for each disease category. , , A positive integer, representing the first... The minimum number of standardized medical semantic text descriptions for the rarity level of a disease category; the higher the rarity level, the lower the rarity level. The larger the number of samples, the rarer the lesion type. In this case, by constructing multiple semantically equivalent or similar standardized medical semantic text descriptions, the semantic expressive power of rare lesion types in cross-modal space can be enhanced, thereby improving the alignment effect of cross-modal features.

[0025] Specifically, in step S1, based on the rarity level of the lesion category, an undersampling strategy is adopted for common lesion categories, and an oversampling strategy is adopted for rare lesion categories.

[0026] This invention addresses the problem of imbalanced sample sizes for different lesion categories in training data by introducing a multi-level imbalanced data processing strategy, including: 1. Dynamically constructing weights for each lesion category based on the sample size of each lesion category. The strategy employs several methods: 1) increasing the contribution of rare lesion categories to loss calculations and enhancing the model's focus on rare lesions; 2) oversampling of rare lesion samples and undersampling of common lesion samples during sample set construction to balance the distribution of training data; and 3) combining text enhancement strategies to construct multiple text descriptions for rare lesions, strengthening their semantic expression and alignment. Through the synergistic effect of these strategies, the model bias problem caused by data imbalance can be effectively alleviated, significantly improving the recognition accuracy of minority lesions and the overall stability of the model.

[0027] Specifically, in step S3, the sample and text description set Input to a pre-trained visual language model, samples Image features are obtained after L2 normalization following image encoder processing. Text description set After text encoding and L2 normalization, text features are obtained. .

[0028] Multimodal large models (VLMs) are an important research direction in the current intersection of artificial intelligence and medical imaging. In step S3 of this case, the visual language model adopts a typical VLM structure, including two core modules: a visual encoder and a text encoder. The visual encoder is specifically a medical image encoder, which can use a pre-trained model based on convolutional neural networks or visual Transformers, such as the visual part of the CLIP model (ViT series), to extract image features. The text encoder is specifically a biomedical text encoder, which can use a language model based on the Transformer architecture to extract text features. Through shared embedding space or cross-modal alignment mechanism, image features and text features are mapped to a unified feature space to achieve semantic-level matching.

[0029] Specifically, in step S3, the visual encoder uses a pre-trained model based on the Transformer architecture (such as the ViT series) to extract deep visual features from the gastrointestinal endoscopy images; the text encoder uses a language model optimized for the biomedical field (such as PubMedBERT) and has professional medical text understanding and feature encoding capabilities; in order to achieve cross-modal alignment between endoscopy images and medical semantic text, a lightweight feature mapping method is used to embed visual features into a feature space that is unified with text features, so that image features and text features can directly perform similarity calculation and comparative learning, thereby achieving cross-modal alignment between endoscopy images and medical semantic text.

[0030] In terms of training strategy, the visual language model employs an efficient optimization strategy of freezing the text encoder and fine-tuning only the image encoder. This achieves accurate alignment between image features and medical semantics without updating the text encoder parameters, significantly reducing computational overhead and training costs. During training, a contrastive learning strategy is used to optimize the model, combined with a class imbalance handling mechanism, ensuring good performance even with medical data exhibiting uneven class distribution. Furthermore, by freezing the text encoder and optimizing only the visual encoder parameters, training complexity is reduced while maintaining semantic stability, thus improving training efficiency.

[0031] The model uses gastrointestinal endoscopy images and lesion category text as training objects. It completes cross-modal feature fusion based on bidirectional image-text comparison learning. At the same time, it introduces a class imbalance handling mechanism to alleviate the problem of scarce minority class lesion samples in clinical data, so that the model can maintain stable recognition performance under the distribution of real clinical data. The BiomedCLIP structure used in this invention is designed specifically for medical vision-language tasks and can be directly adapted to clinical scenarios such as endoscopy image description, lesion recognition, and assisted diagnosis. It has strong generalization and clinical application potential.

[0032] Specifically, the gastrointestinal endoscopy images used in this invention are generally in color RGB format, but grayscale format is also possible. To eliminate the interference of color information on the model's prediction results and to verify that the model's ability to identify lesions comes from core features such as morphology, texture, and contour rather than color information, this invention additionally performs grayscale processing on the color images, converting the three-channel color RGB images into single-channel grayscale images. This simplifies the image information without changing the key morphological features of the lesions. Then, robustness verification experiments are conducted based on the grayscale images. The verification results show that the model can still achieve high-precision lesion identification even when color information is missing. All original gastrointestinal endoscopy images must undergo uniform format verification, size normalization, grayscale conversion, and standardization processing to ensure consistent input distribution. Grayscale processing does not change the texture, contour, and structural features of the lesion area, so it will not affect the model's feature learning and classification judgment of lesions. At the same time, it can be used to verify the stability and compatibility of the model under different imaging device output formats.

[0033] Specifically, the gastrointestinal endoscopic images are mainly derived from three public datasets. Two of these are standard gastrointestinal endoscopic datasets: the kvasir-capsule-labeled-images dataset and the kvasir-v2-data dataset for white light endoscopy. The third is a grayscale version of the kvasir-capsule-labeled-images dataset. These datasets cover a variety of typical clinical gastrointestinal lesions, including normal mucosa, polyps, inflammation, bleeding, erosion, and tumors, and include various imaging methods such as gastroscopy, colonoscopy, and capsule endoscopy. The datasets are hierarchically divided into training, validation, and testing subsets according to lesion categories to ensure a balanced distribution of each lesion category across different subsets. Simultaneously, the number of samples in each lesion category is statistically analyzed to identify sample imbalance characteristics, facilitating the construction of weights and sampling strategies for different lesion categories. This makes the training process more closely reflect the scarcity of real clinical data, ensuring the effectiveness and reliability of the model in practical applications.

[0034] A system for identifying lesion regions in gastrointestinal images based on a visual language model includes an image acquisition and preprocessing module, a text construction module, a feature extraction module, a cross-modal alignment module, a model optimization module, an imbalance processing module, and a lesion identification module.

[0035] The image acquisition and preprocessing module is used to acquire gastrointestinal endoscopy images and construct samples. The gastrointestinal endoscopy images are derived from the kvasir-capsule-labeled-images capsule endoscopy annotation dataset, the kvasir-v2-data white light endoscopy standard dataset, and the grayscale kvasir-capsule-labeled-images capsule endoscopy annotation dataset. After performing format verification, size normalization, grayscale conversion, and standardization on the gastrointestinal endoscopy images, samples are obtained. The number of samples contained in each lesion category is counted, and each lesion category is classified into rare levels according to the number of samples. A minimum value for the number of text descriptions for each rare level is set.

[0036] The text construction module is used to generate a set of standardized medical semantic text descriptions for each disease category, forming a text description set; for rare disease categories, more standardized medical semantic text descriptions need to be constructed to improve the richness of semantic expression;

[0037] The feature extraction module is used to extract image features of samples and text features of text description sets through a visual language model, freeze the text encoder and optimize only the image encoder;

[0038] The cross-modal alignment module is used to construct positive class text matching relationships and negative class text matching relationships for each sample, forming contrastive learning sample pairs;

[0039] The model optimization module is used to calculate the image-text normalized similarity, construct the bidirectional contrast loss, and update the model parameters.

[0040] The imbalance processing module is used to set the weight of each lesion category according to the rarity of the lesion category, and execute the oversampling strategy for rare categories and the undersampling strategy for common categories to improve the identification ability of rare lesion categories.

[0041] The lesion recognition module is used to calculate the similarity between the image and the entire set of text descriptions during the inference stage, output the lesion category recognition result through the maximum average similarity matching, and generate classification evaluation indicators.

[0042] Specifically, the bidirectional contrast loss optimization module uses image-to-text loss and text-to-image loss together to optimize the model; by minimizing the feature distance of positive sample pairs, it aligns images of the same type of lesion with the corresponding text description set; at the same time, it maximizes the feature distance of negative sample pairs to enhance the feature discrimination between different lesion categories; and by introducing lesion category weights into the loss, it enables samples of rare lesion categories to obtain higher gradient contributions during training, further improving the model's ability to identify small sample lesion categories.

[0043] Specifically, the lesion identification and output module adopts a zero-sample classification mode without labels during the inference stage. By calculating the feature similarity between the image to be identified and the entire set of text descriptions, the lesion category with the highest similarity is selected as the prediction result. A unified inference process is used for grayscale images and color RGB images, so stable identification can be completed without retraining. In the end, efficient, accurate and fully automatic auxiliary diagnosis of digestive tract lesions is achieved.

[0044] The advantages of this invention are as follows: By introducing a visual language model, this invention achieves cross-modal alignment between images and medical semantic text, reducing dependence on large-scale labeled data while improving the accuracy and stability of gastrointestinal lesion identification; by optimizing only the image encoder, the model training complexity is reduced and training efficiency is improved; at the same time, the combination of a bidirectional contrastive learning mechanism enhances the model's feature discrimination ability, making different lesion categories more distinguishable in the feature space; by introducing class weighting and sampling strategies, the common class imbalance problem in medical data is effectively alleviated, significantly improving the model's ability to identify minority classes of lesions; and the use of medical semantic text descriptions enhances the model's understanding of complex lesions, thus enabling this method to have good generalization performance and application value in different gastrointestinal endoscopy scenarios. Attached Figure Description

[0045] Figure 1 This is a flowchart of the overall process of the method of the present invention.

[0046] Figure 2 This is a schematic diagram of the training structure for a visual language model.

[0047] Figure 3 This is a schematic diagram illustrating the calculation of feature similarity between images and text. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0049] Traditional methods for identifying gastrointestinal lesions often rely on pure visual models for feature extraction and classification, failing to fully utilize the semantic information of medical text. This results in weak model generalization ability and low accuracy in identifying lesions in small samples. Addressing the real-world needs of uneven distribution of clinical endoscopic image categories, complex lesion features, and the reliance on specialized semantics for diagnosis, this paper proposes a method for identifying lesion regions in gastrointestinal images based on a visual language model. Through cross-modal alignment learning between images and medical text, this method achieves efficient, accurate, and stable identification of various gastrointestinal lesions, including normal mucosa, polyps, inflammation, erosion, bleeding, and tumors. It effectively improves the performance of lesion category identification in small samples, providing efficient and reliable auxiliary diagnostic support for clinical endoscopy, and possesses strong practicality and clinical application value.

[0050] This invention achieves intelligent lesion type identification through cross-modal feature fusion of endoscopic images and medical semantic text. The system for implementing this method mainly includes: a gastrointestinal endoscopic image acquisition module, a medical semantic text construction module, a cross-modal feature extraction module, an image-text comparison learning module, a class imbalance handling module, and a lesion identification output module. The gastrointestinal endoscopic image acquisition module is used to acquire a standardized gastrointestinal endoscopic image dataset and classify it according to lesion categories. The medical semantic text construction module is used to generate standardized medical semantic description text for each type of gastrointestinal lesion. The cross-modal feature extraction module is used to extract image visual features and text semantic features through a visual encoder and a text encoder, respectively. The image-text comparison learning module is used to construct image-text positive and negative sample pairs and optimize the model using bidirectional contrastive loss. The class imbalance handling module is used to mitigate the identification bias caused by the uneven distribution of lesion samples in clinical data. The lesion identification output module is used to predict the lesion category based on the similarity between image and text features and output the diagnostic result.

[0051] like Figure 1 , Figure 2 , Figure 3 The diagram illustrates the complete process for recognizing digestive tract lesions based on the aforementioned system. By employing an efficient training strategy of freezing the text encoder and fine-tuning only the visual encoder, combined with image-text bidirectional contrast learning and class imbalance processing mechanisms, visual features and medical text features are mapped to a unified feature space, achieving high-precision recognition of digestive tract lesions. The following detailed description of the invention is provided in conjunction with specific implementation steps.

[0052] PART 1: Data Preparation Stage

[0053] S1. Construct a dataset of digestive tract endoscopy images.

[0054] A raw dataset was constructed by collecting gastrointestinal endoscopic images and categorizing them according to clinical lesion types, including typical lesion categories such as normal mucosa, polyps, inflammation, bleeding, erosion, and tumors. The gastrointestinal endoscopic images were preprocessed, including format validation, size normalization, standardization, grayscale conversion, and standardization, resulting in samples. The sample set was then hierarchically divided into training subsets, validation subsets, and test subsets according to proportions to ensure that each lesion category maintains the same distribution across different subsets, thereby improving the stability and generalization ability of the model training.

[0055] S2. Constructing a medical semantic text set

[0056] For each disease category, a set of standardized medical semantic text descriptions is constructed to form a text set. The standardized medical semantic text description of each lesion category is denoted as: , The standardized medical semantic text description should be able to accurately describe the clinical characteristics of the corresponding lesion category, for example: "amedical endoscopic image showing polyp in gastrointestinal tract"; the set of text descriptions for all lesion categories together constitutes the medical semantic text set, which is used to provide professional supervision signals for the model and realize cross-modal alignment between images and text.

[0057] S3. Construct positive and negative sample pairs

[0058] During the training process, for ease of description, the samples are... Corresponding text description set Recorded as ,sample Only the corresponding set of text descriptions Constitute positive sample pairs , along with the rest of the text description set Negative sample pairs are formed; by constructing comparative supervision signals, the model's ability to distinguish different lesion features is enhanced, making the features of similar images and text closer together and the features of different categories farther apart.

[0059] PART 2, Model Training Phase

[0060] S4. Initialize the visual language model

[0061] Load a pre-trained visual language model suitable for the medical field, including two core modules: a visual encoder and a text encoder, and extract samples through the image encoder. Image features Extracting text description sets using a text encoder Text features .

[0062] S5. Model Parameter Settings

[0063] The model training parameters were set as follows: 1. Freeze all parameters of the text encoder to maintain the stability and professionalism of the semantic features of medical text; 2. Set only the parameters of the visual encoder to be updatable to reduce training complexity and improve training efficiency; 3. Use an adaptive optimizer and learning rate scheduling strategy to ensure stable convergence of the model.

[0064] S6, Cross-modal feature extraction

[0065] In each training batch, the following operations are performed:

[0066] Sample The input image encoder obtains image features:

[0067]

[0068] Text description set The input text encoder obtains text features:

[0069]

[0070] Image features and text features are L2 normalized separately and mapped to a unified feature space.

[0071] S7. Calculate the similarity matrix

[0072] Matrix operations are performed on the normalized image features and text features to calculate the similarity matrix between the image features and text features, which is used to measure the similarity of the samples. The degree of semantic matching between different sets of text descriptions:

[0073]

[0074]

[0075] S8. Calculate the bidirectional contrast loss.

[0076] Constructing image-to-text contrast loss and text-to-image contrast loss:

[0077]

[0078] The average of the two losses is taken as the final joint loss:

[0079]

[0080] By using joint loss, bidirectional alignment of images and text is achieved, enhancing the discriminativeness and robustness of cross-modal features.

[0081] S9, Model Optimization and Update

[0082] Backpropagation is performed based on the joint loss, updating only the parameters of the visual encoder to continuously optimize the visual feature extraction capability. Through iterative training, the model can accurately distinguish different types of gastrointestinal lesions, while mitigating the recognition bias caused by class imbalance.

[0083] PART 3: Model Reasoning Stage

[0084] S10, Preprocessed Medical Text Features

[0085] All text descriptions Input text encoder, compute and cache text features It is used for rapid reasoning.

[0086] S11. Extract features from the image to be identified.

[0087] The digestive tract endoscopy images to be detected are preprocessed and then input into a visual encoder to obtain the corresponding image visual features. And perform normalization operations.

[0088] S12. Calculate cross-modal similarity

[0089] Calculate the visual features of the image to be recognized With text description set Normalized similarity of each standardized medical semantic text description .

[0090] S13, Complete lesion identification

[0091] For each lesion category, the average similarity of all standardized medical semantic text descriptions is taken, and the lesion category with the highest average similarity is selected as the prediction result. The predicted lesion category is output as follows: .

[0092] PART 4 ​​Hardware Environment Configuration

[0093] The hardware environment used in this case consisted of an NVIDIA A40 GPU with 48GB of video memory and 16GB of RAM. The CPU was a 20vCPU Intel(R) Xeon(R) Platinum 8470Q. The operating system was Ubuntu 20.04 with CUDA version 12.2. The software environment consisted of Python 3.8 and torch 2.1.2.

[0094] PART 5, Evaluation Indicators

[0095] This study employs multi-dimensional evaluation metrics to comprehensively assess model performance, including accuracy, precision, recall, F1 score, and confusion matrix. Accuracy measures the overall recognition effect, precision and recall evaluate the model's ability to recognize various lesions, F1 score comprehensively measures model stability, and confusion matrix visually displays the recognition of easily confused lesions. These metrics comprehensively and objectively reflect the model's lesion recognition performance on clinical endoscopic images.

[0096] Combining the cross-modal feature alignment capability of visual language models, medical semantic guidance mechanisms, and data imbalance handling strategies, this invention achieves efficient identification of gastrointestinal lesions by freezing the text encoder during the training phase, optimizing only the image encoder, and constructing a one-to-one matching relationship between images and corresponding category texts. Experimental results show that this invention exhibits excellent recognition performance and stable semantic preservation capability on different gastrointestinal imaging datasets, while significantly reducing training costs.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way. All technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for identifying lesion regions in gastrointestinal images based on a visual language model, characterized in that: The visual language model includes an image encoder and a text encoder, and the method includes the following steps: S1. Obtain gastrointestinal imaging data: Obtain gastrointestinal endoscopic images of each lesion category as samples for each lesion category, and count the number of samples for each lesion category; classify each lesion category into rare levels according to the number of samples, and set a minimum value for the number of text descriptions for each rare level. S2. Constructing textual descriptions of lesion categories: For the first... A set of standardized medical semantic text descriptions is constructed for each disease category. This forms a set of text descriptions. ;in: , This indicates the total number of disease categories; Indicates the first The first lesion category Standardized medical semantic text description, Indicates the first The total number of standardized medical semantic text descriptions for each lesion class; S3. Feature extraction using a visual language model: [The sample...] and text description set The input is fed into a pre-trained visual language model, and samples are extracted through an image encoder. Image features Extracting text description sets using a text encoder Text features Freeze the text encoder and only optimize the image encoder. S4. Constructing image-text matching relationships: For ease of description, samples... Corresponding text description set Recorded as Corresponding text features Recorded as Construct positive and negative sample pairs. Positive sample pairs are constructed from samples and their corresponding text description sets, while negative sample pairs are constructed from samples and other text description sets. For a sample... Positive sample pairs are Negative samples are ; S5. Cross-modal alignment and imbalance handling during training: First calculate the samples With text description set similarity Then calculate the bidirectional contrast loss between text and image. Total loss Finally, based on the total loss Backpropagation optimization is performed on the image encoder; S6. Lesion area identification during the inference phase: For a given image First, extract the given image using an image encoder. Image features Then calculate its relationship with each set of text descriptions. Similarity of each standardized medical semantic text description ,in: Then, for each lesion category, the average similarity of all standardized medical semantic text descriptions is taken, and the lesion category with the highest average similarity is selected as the prediction result, outputting the predicted lesion category. .

2. The method for identifying lesion areas in digestive tract images based on a visual language model according to claim 1, characterized in that: In step S5, similarity Two-way comparison loss Total loss The calculation formula is: ， ， ， in: Indicates the first The weight of each lesion category, , This indicates the total number of samples.

3. The method for identifying lesion areas in digestive tract images based on a visual language model according to claim 1, characterized in that: In step S2, the lesion is classified into the rarest level as follows: A set of standardized medical semantic text descriptions is constructed for each disease category. , , A positive integer, representing the first... The minimum number of standardized medical semantic text descriptions for the rarity level of a disease category; the higher the rarity level, the lower the rarity level. The larger.

4. The method for identifying lesion areas in digestive tract images based on a visual language model according to claim 1, characterized in that: In step S1, based on the rarity level of the lesion category, an undersampling strategy is used for common lesion categories, and an oversampling strategy is used for rare lesion categories.

5. The method for identifying lesion areas in digestive tract images based on a visual language model according to claim 1, characterized in that: In step S3, the sample and text description set Input to a pre-trained visual language model, samples Image features are obtained after L2 normalization following image encoder processing. Text description set After text encoding and L2 normalization, text features are obtained. .

6. The method for identifying lesion areas in gastrointestinal images based on a visual language model according to claim 1, characterized in that: In step S3, the visual language model adopts a typical cross-modal structure, including two core modules: a visual encoder and a text encoder. The visual encoder uses a pre-trained model based on a convolutional neural network or a visual Transformer to extract image features. The text encoder uses a language model based on a Transformer architecture to extract text features. By sharing the embedding space or using a cross-modal alignment mechanism, image features and text features are mapped to a unified feature space to achieve semantic-level matching.

7. The method for identifying lesion areas in gastrointestinal images based on a visual language model according to claim 6, characterized in that: In step S3, the visual encoder uses a pre-trained model based on the Transformer architecture to extract deep visual features from the gastrointestinal endoscopy images, and the text encoder uses a language model optimized for the biomedical field. The visual features are embedded into a feature space that is unified with the text features through a lightweight feature mapping method, so that the image features and text features can be directly calculated and compared to learn, thereby achieving cross-modal alignment between endoscopy images and medical semantic text.

Images