Endoscopic cross-modal retrieval method and system based on prompt learning and attention

Abstract

This invention discloses a method and system for cross-modal endoscopic image retrieval based on cue learning and attention, belonging to the field of cross-modal retrieval technology for medical images. The method includes: acquiring gastrointestinal endoscopy images and corresponding text descriptions; inputting the text descriptions into a text encoder; providing domain semantic guidance through a medical domain adaptive cue learning module to output text features; initializing cue vectors using medical keywords in the medical domain adaptive cue learning module; extracting image features from gastrointestinal endoscopy images using an image encoder; inputting the image features and text features into a category-aware attention module; enhancing fine-grained category features through category gating and feature recalibration; training a model using a joint loss function, which includes a basic cross-modal contrastive loss and a scarce category-supervised contrastive loss; and performing cross-modal retrieval using the trained model. This invention effectively solves the problems of domain semantic shift, insufficient fine-grained discrimination, and class imbalance, improving retrieval accuracy.

Description

Technical Field

[0001] This invention belongs to the field of medical image cross-modal retrieval technology, specifically relating to an endoscopy cross-modal retrieval method and system based on cue learning and attention. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Gastrointestinal endoscopy is an important tool for diagnosing digestive tract diseases. However, some lesions have high visual similarity in endoscopic images, leading to subjective differences in physician interpretation and necessitating the use of artificial intelligence for diagnosis. In recent years, vision-language pre-trained models, such as CLIP, have demonstrated powerful capabilities in general cross-modal retrieval, providing new ideas for constructing gastrointestinal endoscopy image-text retrieval systems.

[0004] However, the inventors discovered that general vision-language models still face multiple bottlenecks when transferred to cross-modal retrieval tasks in gastrointestinal endoscopy: First, general models are pre-trained on natural images and general text, resulting in significant domain semantic shifts compared to the gastrointestinal endoscopy scenario. Existing cue learning methods use randomly initialized cue vectors, lacking medical semantic priors. Second, normal mucosa, Paris IIa lesions, and esophagitis show highly similar colors and textures under endoscopy, making it difficult for general models to capture fine-grained lesion differences. Third, some disease categories have scarce samples, and standard contrastive learning frameworks treat most and scarce categories equally, easily submerging scarce category features and significantly reducing retrieval accuracy. These bottlenecks severely restrict the practical application of cross-modal retrieval technology in clinical auxiliary diagnosis. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a cross-modal endoscopic retrieval method and system based on cue learning and attention.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the technical solution of the present invention provides an endoscopy cross-modal retrieval method based on cue learning and attention, including: Acquire gastrointestinal endoscopic images and corresponding text descriptions; The text description is input into the text encoder, and the text encoder is guided by domain semantics through the medical domain adaptive prompting learning module to output text features; the medical domain adaptive prompting learning module initializes the prompt vector with medical keywords; The gastrointestinal endoscopy images are input into an image encoder to extract image features; Image and text features are input into the category-aware attention module. The fine-grained category features are enhanced through a category gating mechanism and a feature recalibration strategy, and the enhanced image and text features are output. Training is performed based on enhanced image features and enhanced text features, using a joint loss function that includes basic cross-modal contrastive loss and scarce class supervised contrastive loss. The trained model was used to perform cross-modal retrieval between gastrointestinal endoscopy images and text.

[0007] In at least one embodiment, the medical domain adaptive prompting learning module initializes the prompting vector with medical keywords, specifically including: A medical keyword library was constructed by extracting high-frequency medical terms and commonly used diagnostic terms in clinical gastrointestinal endoscopy from the gastrointestinal endoscopy dataset. A pre-trained word embedding model was used to encode each medical keyword in the medical keyword library to obtain the semantic embedding vector of each keyword. The word embedding vectors of all keywords were aggregated to generate an initial prompt vector. The text description is converted into a text word sequence, and the start word and end word are marked. The cue vector is concatenated with the text word sequence between the start word and end word to construct a complete text encoder input sequence. The concatenated text encoder input sequence is fed into the text encoder for forward computation, and the feature vector corresponding to the end word in the text encoder output is extracted as the global representation of the entire text.

[0008] In at least one embodiment, medical keywords include three types of medical semantic information: anatomical structure keywords, lesion type keywords, and pathological feature keywords.

[0009] In at least one embodiment, the training process of the medical adaptive prompting learning module adopts a differential parameter update strategy, freezing all parameters of the backbone network of the text encoder and updating only the prompting vector and text projection layer parameters.

[0010] In at least one embodiment, the category-aware attention module dynamically adjusts features through category conditional modulation, feature dimension calibration, and category adaptive scaling; Among them, category conditional modulation uses label-guided conditional modulation to calculate dimensional gating weights, feature dimension calibration uses a feature recalibration mechanism based on attention weights to generate features after dimensional attention recalibration, and category adaptive scaling uses a category variance adaptive scaling mechanism to calculate variance adaptive scaling coefficients. Based on variance adaptive scaling coefficients, features recalibrated by dimensional attention, and dimensional gating weights, multi-mechanism fusion is achieved by combining class-learnable scaling coefficients to generate enhanced image features and enhanced text features.

[0011] In at least one embodiment, tag-guided condition modulation specifically includes: The semantic categories of gastrointestinal endoscopy VQA are mapped to low-dimensional learnable category embedding vectors. Using the original features and the learnable category embedding vectors as input, a gating network outputs dimensional category-aware gating weights ranging from [0,1], expressed as:

[0012] In the formula, This represents the input feature of the i-th sample in modality m. Indicates the modal type, where, Represents image modality, Represents text modality; , For learnable mapping matrix, For category The corresponding learnable category embedding vector, For bias terms, This represents the dimension-level gating weights guided by category conditions; During the training phase, the category-based conditional gating weights are generated using the VQA text extracted from gastrointestinal endoscopy, and the prototype vectors of each category are updated using exponential moving average. During the inference phase, soft-class prediction is completed based on the category prototypes to ensure that the training and inference logics are consistent.

[0013] In at least one embodiment, the feature dimension calibration adopts a two-branch independent structure. Each branch consists of two layers of linear transformation, layer normalization, GELU activation function and dropout regularization, which perform dimension-aware adaptive recalibration on image features and text features respectively, generating features after dimension attention recalibration.

[0014] In at least one embodiment, the adaptive scaling mechanism for class variance specifically involves: calculating the variance of the features of the same type of samples within each mini-batch, updating the variance of the class features using an exponential moving average, and calculating the adaptive scaling coefficient of variance based on the class variance.

[0015] In at least one embodiment, the basic cross-modal contrast loss is specifically expressed as:

[0016] In the formula, This represents the contrast loss from image to text; This represents the contrast loss from text to image; The scarce class supervised contrast loss is activated only when there are scarce class samples in the current batch, specifically as follows:

[0017] In the formula, This represents the set of anchor samples that belong to the scarce sample category in the current batch. This represents the normalized feature representation of the i-th sample. This represents the normalized feature representation of a positive sample p. This represents the normalized feature representation of other samples a in the batch. This indicates the similarity of the inner product of features between sample i and positive sample p. This represents the temperature coefficient of loss for a small sample size. The joint loss function is specifically expressed as:

[0018] In the formula, For the total joint loss; The weighting coefficients for the comparative loss of the scarce category supervision.

[0019] Secondly, the technical solution of the present invention also provides an endoscopy cross-modal retrieval system based on cue learning and attention, comprising: The data acquisition module is configured to acquire gastrointestinal endoscopic images and corresponding text descriptions; The text encoding module is configured to: input text descriptions into the text encoder, and guide the text encoder with domain semantics through the medical domain adaptive prompting learning module to output text features; wherein, the medical domain adaptive prompting learning module initializes the prompt vector with medical keywords; The image encoding module is configured to input gastrointestinal endoscopy images into the image encoder and extract image features; The category-aware enhancement module is configured to: input image features and text features into the category-aware attention module, enhance the features of fine-grained categories through a category gating mechanism and a feature recalibration strategy, and output enhanced image features and enhanced text features; The training module is configured to train based on enhanced image features and enhanced text features using a joint loss function, which includes a basic cross-modal contrastive loss and a scarce class supervised contrastive loss. The cross-modal retrieval module is configured to perform cross-modal retrieval between gastrointestinal endoscopy images and text using a trained model.

[0020] The beneficial effects of the above-described technical solution of the present invention are as follows: 1) This invention's cross-modal endoscopic retrieval method based on cue learning and attention guides the model to learn the professional semantics in gastrointestinal endoscopy question-and-answer text by initializing cue vectors with medical keywords, reducing domain semantic bias. Based on a category-aware attention mechanism, combined with category gating and adaptive feature scaling, it enhances features of fine-grained and scarce sample categories, improving category discrimination ability and retrieval accuracy. During the training phase, a joint contrastive loss is used for end-to-end optimization, achieving image-text alignment while enhancing the feature aggregation ability of scarce categories, thereby improving retrieval performance under class imbalance conditions. Through the synergistic innovation of medical domain adaptive cue learning, category-aware attention, and joint loss function, this invention effectively solves the three major bottleneck problems faced when transferring general vision-language models to cross-modal endoscopic retrieval: domain semantic bias, insufficient fine-grained lesion discrimination, and class imbalance.

[0021] 2) The medical domain adaptive prompt learning module of the present invention uses medical keywords to initialize learnable prompt vectors and combines them with a training strategy of freezing the backbone network. This enables the text encoder to focus on lesion types, anatomical structures and mucosal morphological features in the endoscopic scene while retaining general language capabilities, thereby fundamentally alleviating domain semantic shift.

[0022] 3) The category-aware attention module of the present invention achieves feature dimension selection for category differentiation through label-guided conditional modulation, strengthens the discriminative dimension through feature recalibration strategy, and suppresses the distribution noise of scarce categories through variance adaptive scaling. The three progressively and synergistically enhance the discriminative power of fine-grained features and the stability of scarce categories.

[0023] 4) This invention introduces a scarce category supervised contrast loss on the basis of standard contrast loss. The supervised contrast loss is calculated only for scarce categories. By narrowing the feature distance within scarce categories, the intra-class compactness and retrieval accuracy of scarce categories are improved.

[0024] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0026] Figure 1 This is a schematic diagram of the endoscopy cross-modal retrieval method based on cue learning and attention disclosed in Embodiment 1 of the present invention; Figure 2 This is a framework diagram of the endoscopy cross-modal retrieval method based on cue learning and attention disclosed in Embodiment 1 of the present invention; Figure 3This is a structural diagram of the adaptive prompting learning module in the medical field disclosed in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the category-aware gating mechanism disclosed in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the feature dimension calibration and class variance adaptive scaling mechanism disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0027] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] Digestive system diseases, especially malignant tumors such as esophageal cancer, gastric cancer, and colorectal cancer, have always been a serious global public health problem. Statistics show that the number of new cancer cases and deaths worldwide remains at a high level. Therefore, improving the ability to screen for and accurately diagnose digestive tract diseases in the early stages is crucial for improving patient prognosis. In recent years, advancements in visual-language pre-training technology have promoted the application of general visual-language models in biomedical problems and provided a new approach to the intelligent analysis of medical images.

[0029] Gastrointestinal endoscopy allows direct observation of the morphology, color, and lesion characteristics of the digestive tract mucosa, making it a crucial tool for diagnosing esophagitis, gastrointestinal polyps, and early-stage digestive tract tumors. In recent years, research into artificial intelligence in gastrointestinal endoscopy has deepened, with applications expanding from traditional lesion detection to tasks such as endoscopic quality assessment, anatomical structure recognition, and assisted diagnosis. However, some gastrointestinal lesions still exhibit high visual similarity in endoscopic images, leading to subjective differences among physicians in their judgment of fine-grained lesion boundaries and tissue morphology. Current algorithmic research in gastrointestinal endoscopy primarily focuses on image classification, object detection, and region segmentation. However, these tasks typically only output fixed predictions, failing to meet the needs of similar case retrieval and image-text association analysis in clinical settings. In contrast, cross-modal image-text retrieval for gastrointestinal endoscopy focuses more on the semantic connections between images and text, relying on images or text to retrieve corresponding case and treatment descriptions, providing effective support for clinical diagnosis and case verification.

[0030] For medical image analysis tasks, some studies have begun to attempt to jointly model medical images and text. For example, Zhang et al.'s ConVIRT, for the first time, uses medical image-report samples to construct a contrastive learning system, jointly learning medical visual features and textual semantics; Huang et al.'s GLoRIA, through joint learning of global image features and local region features, uses cross-modal attention to achieve fine-grained semantic alignment, enhancing the model's ability to express medical structures; Wang et al.'s MedCLIP, using unpaired medical image-text data to alleviate the problem of insufficient medical annotation data, improves the robustness of cross-modal alignment. However, most of these existing studies focus on general medical scenarios such as radiological imaging, and there is still a lack of sufficient modeling for the relationship between mucosal texture, anatomical structures, and lesion descriptions specific to gastrointestinal endoscopy images.

[0031] The inventors discovered that existing general-purpose visual-language models still face multiple bottlenecks when transferred to cross-modal retrieval tasks in gastrointestinal endoscopy: First, general-purpose models are pre-trained on natural images and general text, resulting in significant domain semantic shifts between them and the image-text descriptions in gastrointestinal endoscopy scenarios. Existing cue learning methods use randomly initialized cue vectors, lacking medical semantic priors. Second, normal mucosa, early superficial lesions of Paris IIa, and esophagitis exhibit highly similar colors, textures, and morphologies under endoscopy, making it difficult for general-purpose models to capture fine-grained lesion differences. Third, some disease categories have scarce samples, and standard contrastive learning frameworks treat most categories and scarce categories equally, easily obscuring the features of scarce categories. General-purpose models cannot effectively capture their unique visual representations, exacerbating the category confusion problem and leading to a significant decrease in retrieval accuracy.

[0032] To overcome the shortcomings of the existing technologies, a typical embodiment of this invention proposes a CLIP cross-modal retrieval framework for gastrointestinal endoscopy images based on medical domain adaptive prompt learning and category-aware attention. This framework is designed based on a semantically guided adaptive alignment framework. A medical domain adaptive prompt learning module is introduced into the text branch, initializing the prompt vector with medical keywords to guide the model in learning the professional semantics of the gastrointestinal endoscopy question-and-answer text, thus mitigating domain semantic bias. Simultaneously, a category-aware attention mechanism is introduced, combining category gating and adaptive feature scaling to enhance features of fine-grained and scarce sample categories, improving category discrimination and retrieval accuracy. Furthermore, a supervised contrastive loss function for scarce categories is constructed to strengthen the intra-class feature distribution of scarce samples within the traditional cross-modal contrastive learning framework, improving the stability of image-text retrieval under data imbalance. This achieves image-text alignment while enhancing the feature aggregation capability of scarce categories, thereby improving retrieval performance under class imbalance conditions.

[0033] Example 1 In a typical embodiment of the present invention, such as Figures 1 to 5 As shown, this embodiment discloses an endoscopy cross-modal retrieval method based on cue learning and attention. It employs the CLIP gastrointestinal endoscopy image cross-modal retrieval framework, which is based on adaptive cue learning in the medical field and category-aware attention. Specifically, it includes the following steps: S1. Obtain gastrointestinal endoscopy images and corresponding text descriptions; S2. Input the text description into the text encoder, and guide the text encoder with domain semantics through the medical domain adaptive prompting learning module to output text features; wherein, the medical domain adaptive prompting learning module initializes the prompt vector with medical keywords; S3. Input the gastrointestinal endoscopy images into the image encoder to extract image features; S4. Input image features and text features into the category-aware attention module, and enhance the fine-grained category features through the category gating mechanism and feature recalibration strategy, and output enhanced image features and enhanced text features; S5. Based on enhanced image features and enhanced text features, a joint loss function is used for training. The joint loss function includes basic cross-modal contrastive loss and scarce class supervised contrastive loss. S6. Use the trained model to perform cross-modal retrieval between gastrointestinal endoscopy images and text.

[0034] The following describes in detail the above-mentioned cross-modal endoscopic retrieval method based on cue learning and attention, with specific implementation methods.

[0035] S1. Obtain gastrointestinal endoscopy images and corresponding text descriptions.

[0036] In this step, the gastrointestinal endoscopic images are derived from raw image data collected during clinical gastrointestinal endoscopy examinations, including but not limited to at least one of white light endoscopy images, narrow band imaging images, magnified endoscopy images, and chromoendoscopic images. These raw image data record the morphology, color, vascular texture, and visual information of lesion areas of the digestive tract mucosa.

[0037] The text descriptions corresponding to gastrointestinal endoscopic images are specifically medical text data associated with each gastrointestinal endoscopic image, derived from at least one of clinical endoscopy reports, physician diagnostic records, or structured question-and-answer data. The text descriptions contain multi-dimensional semantic annotation information on the image content, including descriptions of anatomical locations, classifications of lesion morphology and pathological types, quantitative descriptions of lesion size and extent, and descriptions of functional metabolic or hemodynamic characteristics.

[0038] As a further implementation, a data reading interface is used to batch load paired data of gastrointestinal endoscopy images and descriptive texts from a local database, network storage, or medical information system. To ensure semantic consistency between the image and text pairs, the gastrointestinal endoscopy images and text descriptions maintain a one-to-one or many-to-one association at the data level, meaning that each gastrointestinal endoscopy image is associated with at least one corresponding text description for subsequent cross-modal semantic alignment and feature learning.

[0039] As a further implementation method, the acquired gastrointestinal endoscopy images and text descriptions are preprocessed. Specifically, the original gastrointestinal endoscopy images are normalized in size and uniformly scaled to 224×224 pixels; pixel value standardization is performed, mapping the RGB channel pixel values ​​to the [0,1] interval and then normalizing them according to the mean and standard deviation of the pre-trained model; at the same time, low-quality images with blurry images and no effective lesion areas are filtered out, and duplicate samples are removed.

[0040] For text descriptions, the process begins with basic text cleaning, removing special characters and standardizing medical terminology. Next, sample quality checks are performed, eliminating meaningless questions and answers, filtering duplicate question-answer pairs, and concatenating the questions and answers into a complete semantic description. Finally, CLIP's built-in BPE tokenizer is used to encode the concatenated text. The segmented sequence has a fixed length of 77 tokens (including start and end markers). Tokens exceeding the length are truncated at the end; those insufficient are padded with padding to ensure consistent input format. The processed text, along with the corresponding image and weakly supervised category, constitutes an "image-text-category" triple.

[0041] S2. Input the text description into the text encoder, and guide the text encoder with domain semantics through the medical domain adaptive prompt learning module to output text features; wherein, the medical domain adaptive prompt learning module initializes the prompt vector with medical keywords.

[0042] The core objective of the semantic layer is to guide the text distribution in the semantic space towards convergence with the medical domain. Existing research largely follows common cue learning approaches, such as CoOp and MapLe, which rely on introducing learnable soft cue vectors to improve the model's downstream adaptability. However, these methods primarily initialize cue vectors randomly and do not incorporate specialized semantic priors related to gastrointestinal endoscopy. Therefore, it is difficult to address the domain semantic shift problem between general pre-trained models and gastrointestinal endoscopy question-answering text.

[0043] Considering the strong medical specialization of question-and-answer (VQA) text in gastrointestinal endoscopy, this embodiment adopts the domain-adaptive prompt learning framework and designs a medical domain-adaptive prompt learning module for the characteristics of gastrointestinal endoscopy scenarios. By introducing the medical domain-adaptive prompt learning module into the text branch, medical domain semantic guidance is carried out, and the prompt vector is optimized with the help of medical semantic priors, so that the text features are adapted to the scenario of gastrointestinal endoscopy applications.

[0044] Specifically, high-frequency medical terms and commonly used clinical diagnostic terms in gastrointestinal endoscopy are first extracted from the gastrointestinal endoscopy dataset to construct a medical keyword library. This library contains three categories of medical semantic information: anatomical structure keywords, lesion type keywords, and pathological feature keywords. A Contrastive Language-Image Pre-training (CLIP) word embedding model is used to encode each of the medical keywords, obtaining a semantic embedding vector for each keyword. The word embedding vectors of all keywords are then aggregated to generate a dimensionality of [missing information]. The initial cue vector, where, To indicate the length of the vector, The feature dimension is defined by medical keywords. This ensures that the prompt vectors possess prior medical semantic knowledge of lesion types, anatomical structures, and pathological morphologies in gastrointestinal endoscopy scenarios from the initial training stage, avoiding slow training convergence or getting trapped in local optima due to random initialization.

[0045] As a further implementation, since the length of the question-and-answer text in gastrointestinal endoscopy varies, the length of the prompt vector will be adjusted accordingly. Setting it to 20 ensures semantic guidance while adapting to CLIP's input length constraint.

[0046] For each input text description, the CLIP-compatible word segmenter is used to convert it into a text word sequence, with the first word of the sequence labeled as the start word [SOS] and the last word labeled as the end word [EOS]. Between the start word [SOS] and the end word [EOS], a learnable continuous cue vector is concatenated with the text word sequence to construct a complete text encoder input sequence.

[0047] The concatenated text encoder input sequence is fed into the CLIP text encoder for forward computation. The feature vector corresponding to the end word [EOS] in the CLIP text encoder output is extracted. The output feature of the end word [EOS] incorporates the semantic information of the entire sequence. Therefore, the feature vector at this position is extracted as the global representation of the entire input text, specifically as follows:

[0048] In the formula, Indicates CLIP text encoder; This represents a sequence concatenation operation. Represents an adaptive cue vector in the medical field; This represents the multiple text sequences corresponding to the i-th gastrointestinal endoscopy image. In the scenario of one image and multiple texts, a single image corresponds to multiple text descriptions. The feature vector corresponding to the end word [EOS] in the CLIP text encoder output.

[0049] In this step, a differential parameter update strategy is used during training. All parameters of the CLIP text encoder's backbone network are frozen, ensuring the encoder remains unchanged during training and preserving CLIP's generalization ability of language representations learned on general corpora. Only the cue vectors are updated. The text projection layer parameters are set to a trainable state and updated via backpropagation. This differentiated parameter update strategy incorporates medical domain knowledge while preserving the generalization ability of a general language model, effectively bridging the semantic gap between the general CLIP model and gastrointestinal endoscopy scenarios. Furthermore, the training process updates only a small number of cue vectors and projection layer parameters, significantly reducing the number of parameters and training costs required for medical domain adaptation.

[0050] S3. Input the gastrointestinal endoscopy images into the image encoder to extract image features.

[0051] In this step, the gastrointestinal endoscopy images are input into the CLIP image encoder to extract image features. Specifically, the preprocessed gastrointestinal endoscopy images are first segmented and embedded, and then positionally encoded to convert the two-dimensional images into a sequential image block embedding sequence. This sequence is then input into the CLIP visual Transformer image encoder, and after forward propagation through multiple layers of self-attention and feedforward networks, an image encoding sequence is obtained. Finally, the feature vectors corresponding to the global classification terms in the encoding sequence are extracted as the initial image features of the image.

[0052] S4. Input image features and text features into the category-aware attention module, and enhance the fine-grained category features through the category gating mechanism and feature recalibration strategy, and output enhanced image features and enhanced text features.

[0053] Most existing cross-modal methods employ a uniform attention mechanism or fixed weights for feature modulation, making it difficult to truly distinguish subtle lesion features in endoscopic images. To enhance the class discrimination capability of image-text cross-modal features and address the issues of uneven class distribution and unclear fine-grained lesion features in gastrointestinal endoscopy data, this embodiment introduces a category-aware attention (CAA) module in the feature representation layer. Image and text features are input into the CAA module, and the features are dynamically adjusted through category conditional modulation, feature dimension calibration, and category adaptive scaling, enabling the model to learn the feature differences and distribution characteristics of different categories. In this step, category-aware attention (CAA) can be formally represented as a conditional feature transformation function, specifically as follows:

[0054] In the formula, This represents the input feature of the i-th sample in modality m; The VQA semantic category of the i-th sample is determined by the content of gastrointestinal endoscopy questions and answers. It is labeled by coarse keyword matching, clinical rule denoising, and cross-verification of official attributes of the dataset. This represents the category-aware attention feature modulation function; This represents the modal characteristics enhanced by the CAA module; This represents the modal type, where, Represents image modality, Represents the text modality.

[0055] To balance model generalization and class discrimination capabilities, label-guided conditional modulation is employed for class conditional modulation. This maps the semantic categories of gastrointestinal endoscopy (VQA) to low-dimensional learnable class embedding vectors, which serve as conditional information in the gating weight calculation of the class-aware attention module. Figure 4 The category-aware gating mechanism is shown. The gating network takes initial image features, initial text features, and learnable category embedding vectors as input, and outputs category-aware gating weights in the range [0,1], specifically represented as follows:

[0056] In the formula, This represents the input feature of the i-th sample in mode m; , For learnable mapping matrix, For category The corresponding learnable category embedding vector, For bias terms, This represents the dimension-level gating weights guided by category conditions.

[0057] Input features under mode m To query features, first calculate The cosine similarity with the initial bimodal features, through Weighted fusion yields global gating weights, enabling adaptive feature enhancement without explicit classification. The gating calculation during inference is specifically represented as follows:

[0058] In the formula, The category prototype vector is maintained and updated for each category using an exponential moving average (EMA). For cross-modal cosine similarity between input features and category prototypes; It represents The weighted temperature hyperparameter is used to control the smoothness of the similarity distribution; c is the category index for traversing the entire lesion category set during the inference phase. The feature gating weights corresponding to category c are represented.

[0059] During the training phase, category-based conditional gating weights are generated using text extracted from gastrointestinal endoscopy (VQA), and the prototype vectors for each category are updated using exponential moving average. In the inference phase, soft-class prediction is performed based on the category prototypes. First, the cosine similarity between the input query features and all category prototypes is calculated. After temperature scaling and Softmax normalization, the soft weight distribution of the sample's category is obtained. Based on this soft weight, all category standard gating templates are weighted and fused to generate a global dimensional gating adapted to the current sample. This eliminates the need for deterministic single-class determination, thus enhancing category-aware adaptive features and ensuring consistency between training and inference logic. The gating network, CLIP image encoder, and text encoder are jointly trained end-to-end, learning the feature differences of different lesions under category supervision, thereby enhancing the ability to distinguish fine-grained categories. This category-aware attention module only operates on the feature representation layer; category information does not participate in cross-modal similarity calculations, and category prototypes are only updated during training. No additional data is introduced during the inference phase, ensuring the model's generalization ability.

[0060] To enable the model to better learn discriminative semantic representations between categories, feature dimension calibration employs an attention-weight-based feature recalibration mechanism. This mechanism adaptively learns the importance weights of different feature dimensions through an additional attention branch and dynamically strengthens the original features, thereby highlighting important semantic representations while weakening unnecessary information, such as... Figure 5 As shown.

[0061] Specifically, the feature dimension attention adopts a two-branch independent structure. Each branch consists of two layers of linear transformation, layer normalization, GELU activation function, and dropout regularization, performing dimension-aware adaptive recalibration on image features and text features respectively. Taking image features as an example, the attention weights are first calculated, specifically as follows:

[0062] In the formula, For the image features of the i-th sample, This represents the attention network corresponding to the feature dimension of modality m. The output is a dimension-level attention weight vector (dimension D=512). Next, through attention weights The original features are relabeled dimensionally to enhance the contribution of discriminative feature dimensions and suppress the interference of irrelevant feature dimensions. Specifically, this is expressed as follows:

[0063] In the formula, The features are those after dimensionality attention recalibration. This indicates element-wise multiplication. Represents the original input features. It is a dimensional attention weight. This is the attention modulation coefficient.

[0064] To alleviate the problems of discrete distribution and statistical instability of rare disease category features, the category adaptive scaling adopts a category variance adaptive scaling mechanism. This mechanism draws on the idea of ​​feature normalization based on statistics, dynamically adjusts the feature response intensity based on the statistical results of the feature distribution of each category, and adjusts the feature scale through variance adaptive strategy to reduce the learning bias caused by the imbalance of category distribution and improve the stability of rare category feature expression.

[0065] During training, the variance of features of similar samples within each mini-batch is first calculated and continuously adjusted using an exponential moving average. Then, an adaptive scaling factor for variance is calculated based on the class variance, specifically expressed as follows:

[0066] In the formula, The feature variance of class c is updated by the training moving average; For class c, the variance adaptive scaling factor is used. and These represent the lower and upper bounds of the scaling factor, respectively. This is the clipping function used to limit the range of scaling factors; To prevent the denominator from being a minimal stable smoothing constant of 0, the feature scale is dynamically adjusted based on the variance of the class features, as shown in the equation, to suppress high-variance class noise and enhance the discriminative power of low-variance classes.

[0067] Based on variance adaptive scaling coefficients, features recalibrated with dimensional attention, and dimensional gating weights, a class-learnable scaling coefficient is introduced to achieve multi-mechanism fusion, generating the final enhanced features of the model, specifically as follows:

[0068] In the formula, For category The corresponding learnable scaling coefficient; Enhance the final output features of the i-th sample in mode m. This represents the modal type, where, Represents image modality, Representing text modality, when hour, For the enhanced image features of the i-th sample, when hour, Enhanced text features for the i-th sample; This indicates a feature normalization operation; For category The corresponding variance adaptive scaling factor.

[0069] In this step, the category-aware attention module adopts a cross-modal feature-sharing architecture through the aforementioned category gating and feature recalibration strategies. Image features and text features are weighted through independent attention branches, and then share category gating and variance adaptive scaling units. This design preserves the original characteristics of each modality while avoiding semantic misalignment, adapting to medical retrieval scenarios with multiple images and text, and effectively improving retrieval capabilities. This category-aware attention module also complements the medical domain adaptive prompting learning module. The category-aware attention module strengthens image and text features, while the medical domain adaptive prompting learning module guides text semantics, jointly optimizing cross-modal alignment from two dimensions and improving fine-grained retrieval capabilities.

[0070] S5. Based on enhanced image features and enhanced text features, a joint loss function is used to train the model. The joint loss function includes basic cross-modal contrastive loss and scarce class supervised contrastive loss.

[0071] To achieve collaborative training of cue learning and category-aware attention, and to improve the fine-grained cross-modal alignment performance of scarce samples, this step is based on the basic contrastive loss and introduces supervised contrastive learning to enhance the compactness of category features of scarce samples, so as to achieve global alignment and local compactness of cross-modal features, and thus achieve end-to-end optimization of the entire framework.

[0072] Specifically, this step constructs a joint loss function based on the basic cross-modal contrastive loss and the scarce class supervised contrastive loss, which is expressed as follows:

[0073] In the formula, This represents the total joint loss during end-to-end training of the model. Based on cross-modal contrastive loss; For scarce categories, supervise the contrast loss; The weighting coefficients for the comparative loss of the scarce category supervision.

[0074] Among them, the basic cross-modal contrast loss Based on the CLIP standard contrastive loss design, for image-text pairing scenarios, it maximizes the feature similarity between a single image and its multiple corresponding texts, and minimizes the similarity with non-matching texts, achieving global alignment of cross-modal features. Specifically, it is expressed as follows:

[0075] In the formula, The image-to-text contrast loss is expressed as:

[0076] The contrast loss from text to image is represented as:

[0077] In the formula, Represents the i-th enhanced image feature With the j-th enhanced text feature The similarity between them is expressed as: ; Let cosine similarity represent the i-th image and its k-th paired text. Let represent the cosine similarity between any image j and the current text k; The standard contrast loss temperature coefficient is used to control the smoothness of the similarity distribution; N is the batch size, and M is the total number of texts in the batch. Let be the set of multiple matching text descriptions corresponding to the i-th image.

[0078] Scarcity Category Supervision and Comparison Loss For categories with scarce samples, the supervised contrast loss is calculated only for the Paris IIa, Paris Is, and Oesophagitis sample categories, specifically expressed as follows:

[0079] In the formula, This represents the set of anchor samples that belong to the scarce sample category in the current batch. This set of anchor samples contains both enhanced image features and enhanced text features of the scarce category. This represents the normalized feature representation of the i-th sample, including normalized enhanced image features and normalized enhanced text features. This represents the normalized feature representation of a positive sample p. This represents the normalized feature representation of other samples a in the batch. This indicates the similarity of the inner product of features between sample i and positive sample p. Temperature coefficient for small sample loss. Rare category supervised contrast loss. It is activated only when there are scarce category samples in the current batch. It improves the intra-class compactness and fine-grained inter-class separability of scarce categories by bringing the features within scarce categories and paired text features closer together.

[0080] In this step, the joint loss is performed on a batch basis. During each training iteration, the gastrointestinal endoscopy images and corresponding text descriptions in the batch are first fed into the image encoder and text encoder, respectively. The outputs of the two encoders are then fed into the category-aware attention module to calculate the enhanced image features and enhanced text features. Then, the basic cross-modal contrastive loss is calculated based on the enhanced image features and enhanced text features. Compared with the loss of supervision of scarce categories And then, by weighted summation, the total joint loss is obtained. Among them, the supervised contrast loss of scarce categories Activate only if there are scarce class samples in the current batch; calculate the total joint loss. Gradients are calculated for all trainable parameters of the image encoder, learnable cue vectors, text projection layer, and category-aware attention module. The trainable parameters are then updated based on these gradients to gradually bring the model to converge.

[0081] S6. Use the trained model to perform cross-modal retrieval between gastrointestinal endoscopy images and text.

[0082] This step is divided into two stages: offline feature base library construction and online retrieval execution. L2 normalization is completed based on enhanced image features and enhanced text features. The semantic matching degree is measured by cosine similarity, and the final result is output using the Top-K strategy.

[0083] Specifically, in the preprocessing stage, the size of all images to be retrieved is normalized to 224×224 pixels, and RGB channel standardization is completed, with the rules being completely consistent with those in the training stage. In the initial feature extraction stage, the trained CLIP image encoder is input to obtain initial image features. In the adaptive feature enhancement stage, the category-aware attention module performs soft category prediction through the pre-stored EMA category prototype, generates dimensionality gating and variance coefficients, and outputs enhanced image features. Finally, normalization and storage are performed. The enhanced features are L2 normalized and stored in the base database, synchronously associated with metadata such as the original image, lesion category, and number, to complete the construction of the image feature base database.

[0084] The entire text to be retrieved undergoes the same preprocessing and word segmentation as during the training phase, and medical adaptive prompt vectors are concatenated to construct a complete input sequence. Initial features are extracted by the CLIP text encoder, and feature enhancement is performed through a category-aware attention module. After L2 normalization, the enhanced features are stored in the text feature base library, and metadata such as the original text and question-and-answer diagnostic information are synchronously associated to complete the construction of the text feature base library.

[0085] It supports bidirectional retrieval from image to text and from text to image, with consistent execution logic for both types of tasks: In the query sample preprocessing stage, it receives user input and performs modality-specific preprocessing that is completely consistent with the base database construction; following the same process as the base database construction, it completes encoding, feature enhancement, and L2 normalization to obtain query features; it calculates the inner product of the query features and the full set of features in the target modality base database to obtain an equivalent set of cosine similarity scores; it sorts the results in descending order of similarity, selects the Top-K results for output, and simultaneously displays the corresponding original data and metadata.

[0086] Example 2 In a typical embodiment of the present invention, this embodiment discloses an endoscopy cross-modal retrieval system based on cue learning and attention, comprising: The data acquisition module is configured to acquire gastrointestinal endoscopic images and corresponding text descriptions; The text encoding module is configured to: input text descriptions into the text encoder, and guide the text encoder with domain semantics through the medical domain adaptive prompting learning module to output text features; wherein, the medical domain adaptive prompting learning module initializes the prompt vector with medical keywords; The image encoding module is configured to input gastrointestinal endoscopy images into the image encoder and extract image features; The category-aware enhancement module is configured to: input image features and text features into the category-aware attention module, enhance the features of fine-grained categories through a category gating mechanism and a feature recalibration strategy, and output enhanced image features and enhanced text features; The training module is configured to train based on enhanced image features and enhanced text features using a joint loss function, which includes a basic cross-modal contrastive loss and a scarce class supervised contrastive loss. The cross-modal retrieval module is configured to perform cross-modal retrieval between gastrointestinal endoscopy images and text using a trained model.

[0087] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A cross-modal endoscopic retrieval method based on cueing learning and attention, characterized in that, include: Acquire gastrointestinal endoscopic images and corresponding text descriptions; The text description is input into the text encoder, and the text encoder is guided by the domain semantics through the medical domain adaptive prompting learning module to output text features. Among them, the adaptive prompting learning module in the medical field uses medical keywords to initialize the prompt vector; The gastrointestinal endoscopy images are input into an image encoder to extract image features; Image and text features are input into the category-aware attention module. The fine-grained category features are enhanced through a category gating mechanism and a feature recalibration strategy, and the enhanced image and text features are output. Training is performed based on enhanced image features and enhanced text features, using a joint loss function that includes basic cross-modal contrastive loss and scarce class supervised contrastive loss. The trained model was used to perform cross-modal retrieval between gastrointestinal endoscopy images and text.

2. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 1, characterized in that, The adaptive prompting learning module in the medical field initializes the prompt vector using medical keywords, specifically including: A medical keyword library was constructed by extracting high-frequency medical terms and commonly used diagnostic terms in clinical gastrointestinal endoscopy from the gastrointestinal endoscopy dataset. A pre-trained word embedding model was used to encode each medical keyword in the medical keyword library to obtain the semantic embedding vector of each keyword. The word embedding vectors of all keywords were aggregated to generate an initial prompt vector. The text description is converted into a text word sequence, and the start word and end word are marked. The cue vector is concatenated with the text word sequence between the start word and end word to construct a complete text encoder input sequence. The concatenated text encoder input sequence is fed into the text encoder for forward computation, and the feature vector corresponding to the end word in the text encoder output is extracted as the global representation of the entire text.

3. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 2, characterized in that, Medical keywords include three categories of medical semantic information: anatomical structure keywords, lesion type keywords, and pathological feature keywords.

4. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 2, characterized in that, The training process of the adaptive prompting learning module in the medical field adopts a differential parameter update strategy, freezing all parameters of the backbone network of the text encoder and updating only the prompting vector and the text projection layer parameters.

5. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 1, characterized in that, The category-aware attention module dynamically adjusts features through category-conditional modulation, feature dimension calibration, and category-adaptive scaling; Among them, category conditional modulation uses label-guided conditional modulation to calculate dimensional gating weights, feature dimension calibration uses a feature recalibration mechanism based on attention weights to generate features after dimensional attention recalibration, and category adaptive scaling uses a category variance adaptive scaling mechanism to calculate variance adaptive scaling coefficients. Based on variance adaptive scaling coefficients, features recalibrated by dimensional attention, and dimensional gating weights, multi-mechanism fusion is achieved by combining class-learnable scaling coefficients to generate enhanced image features and enhanced text features.

6. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 5, characterized in that, Tag-guided conditional modulation specifically refers to: The semantic categories of gastrointestinal endoscopy VQA are mapped to low-dimensional learnable category embedding vectors. Using the original features and the learnable category embedding vectors as input, a gating network outputs dimensional category-aware gating weights ranging from [0,1], expressed as: In the formula, This represents the input feature of the i-th sample in modality m. Indicates the modal type, where, Represents image modality, Represents text modality; , For learnable mapping matrix, For category The corresponding learnable category embedding vector, For bias terms, This represents the dimension-level gating weights guided by category conditions; During the training phase, the category-based conditional gating weights are generated using the VQA text extracted from gastrointestinal endoscopy, and the prototype vectors of each category are updated using exponential moving average. During the inference phase, soft-class prediction is completed based on the category prototypes to ensure that the training and inference logics are consistent.

7. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 5, characterized in that, Feature dimension calibration adopts a dual-branch independent structure. Each branch consists of two layers of linear transformation, layer normalization, GELU activation function and dropout regularization, which perform dimension-aware adaptive recalibration on image features and text features respectively, generating features after dimension attention recalibration.

8. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 5, characterized in that, The adaptive scaling mechanism for class variance is as follows: calculate the variance of the features of the same type of samples within each mini-batch, update the variance of the class features using an exponential moving average, and calculate the adaptive scaling coefficient of variance based on the class variance.

9. The endoscopic cross-modal retrieval method based on cue learning and attention as described in claim 1, characterized in that, The basic cross-modal contrastive loss is specifically expressed as follows: In the formula, This represents the contrast loss from image to text; This represents the contrast loss from text to image; The scarce class supervised contrast loss is activated only when there are scarce class samples in the current batch, specifically as follows: In the formula, This represents the set of anchor samples that belong to the scarce sample category in the current batch. This represents the normalized feature representation of the i-th sample. This represents the normalized feature representation of a positive sample p. This represents the normalized feature representation of other samples a in the batch. This indicates the similarity of the inner product of features between sample i and positive sample p. This represents the temperature coefficient of loss for a small sample size. The joint loss function is specifically expressed as: In the formula, For the total joint loss; The weighting coefficients for the comparative loss of the scarce category supervision.

10. An endoscopy cross-modal retrieval system based on cueing learning and attention, characterized in that, include: The data acquisition module is configured to acquire gastrointestinal endoscopic images and corresponding text descriptions; The text encoding module is configured to: input text descriptions into the text encoder, and guide the text encoder with domain semantics through the medical domain adaptive prompting learning module, and output text features; Among them, the adaptive prompting learning module in the medical field uses medical keywords to initialize the prompt vector; The image encoding module is configured to input gastrointestinal endoscopy images into the image encoder and extract image features; The category-aware enhancement module is configured to: input image features and text features into the category-aware attention module, enhance the features of fine-grained categories through a category gating mechanism and a feature recalibration strategy, and output enhanced image features and enhanced text features; The training module is configured to train based on enhanced image features and enhanced text features using a joint loss function, which includes a basic cross-modal contrastive loss and a scarce class supervised contrastive loss. The cross-modal retrieval module is configured to perform cross-modal retrieval between gastrointestinal endoscopy images and text using a trained model.

Images