An image classification method, system, terminal, and storage medium based on a fine-tuned biomedical visual language model.

Abstract

This invention discloses an image classification method, system, terminal, and storage medium based on a fine-tuned biomedical visual language model. The method includes: an adaptive prototype visual locator extracting an adaptive visual prototype from the multi-scale representation of the input image and correcting text prompts to obtain target text prompts; a co-causal decoupling module decoupling the target text prompts into causal and non-causal prompts, controlling the learning of the biomedical visual language model through an overall loss function, suppressing pseudo-related information associated with the training category, and obtaining a fine-tuned biomedical visual language model; applying the fine-tuned biomedical visual language model to an automatic classification and recognition task of biomedical images, processing the input biomedical images, and outputting category labels or identification information. This invention uses a multi-scale adaptive visual prototype for prompt generation, which can retain multi-scale information from coarse-grained anatomical structures to fine-grained pathological textures.

Description

Technical Field

[0001] This invention relates to the field of biomedical engineering technology, and in particular to an image classification method, system, terminal, and computer-readable storage medium based on a fine-tuned biomedical visual language model. Background Technology

[0002] In recent years, visual language models (VLMs) have demonstrated tremendous potential in the biomedical field. For example, pre-trained visual language models, such as CLIP, trained on large-scale image-text pairs, have achieved strong zero-shot transfer capabilities. For the biomedical field, researchers have further developed specialized models such as BiomedCLIP, which, pre-trained on approximately 15 million biomedical image-text pairs, has achieved excellent performance in various biomedical vision tasks. However, how to efficiently adapt these general-purpose or domain-specific large-scale visual language models to downstream biomedical tasks remains a pressing problem to be solved.

[0003] The aim of adapting general or domain-specific large-scale visual language models to downstream biomedical tasks is to enable pre-trained large-scale visual language models to serve specific downstream biomedical tasks more effectively under limited labeled sample conditions, thereby improving the recognition accuracy, data utilization efficiency, and generalization ability to unseen categories.

[0004] In practical clinical applications, biomedical image annotation requires the participation of medical experts with specialized knowledge, and the annotation cost is high and time-consuming. Therefore, few-shot adaptation based on a small number of annotated samples becomes a practically feasible solution. At the same time, due to the long-tail distribution characteristics of medical data, some rare disease categories may only have a small number of samples or even none available, making them unsuitable for effective model training. This necessitates that the model also possess a certain ability to recognize these categories that have not been encountered during training.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide an image classification method, system, terminal, and computer-readable storage medium based on a fine-tuned biomedical visual language model, aiming to solve the problem that existing visual language models cannot be adapted to downstream biomedical tasks, thus making it impossible to automatically classify and recognize biomedical images.

[0007] To achieve the above objectives, this invention provides an image classification method based on a fine-tuned biomedical visual language model, wherein the biomedical visual language model includes an adaptive prototype visual locator and a co-causal decoupling module; the image classification method based on the fine-tuned biomedical visual language model includes the following steps: The adaptive prototype visual locator extracts an adaptive visual prototype from the multi-scale representation of the input image, and uses the adaptive visual prototype to correct the text prompt to obtain the target text prompt. The collaborative causal decoupling module decouples the target text prompt into causal and non-causal prompts, controls the learning of the biomedical visual language model through the overall loss function, suppresses pseudo-related information associated with the training category, and obtains a fine-tuned biomedical visual language model. The finely tuned biomedical visual language model is applied to the automatic classification and recognition task of biomedical images. After processing the input biomedical images, it outputs category labels or identification information.

[0008] Optionally, the image classification method based on a fine-tuned biomedical visual language model includes an image encoder and a text encoder.

[0009] Optionally, the image classification method based on the fine-tuned biomedical visual language model, wherein the adaptive prototype visual locator extracts an adaptive visual prototype from the multi-scale representation of the input image, and uses the adaptive visual prototype to correct the text prompt to obtain the target text prompt, specifically includes: Image After being input into the image encoder, in the image encoder's... The set of image block feature vectors output by the block is , Indicates the number of image patches. Indicates the first Feature vectors of image patches; The adaptive prototype visual locator uses a contribution matrix , For the scope, Aggregate into An adaptive visual prototype: ; in, Represents the first feature projection matrix. , This represents the second characteristic projection matrix. , It is the dimension of the image patch feature vector. It is the dimension of the intermediate feature vector. Representation layer normalization, This represents the activation function. Represents the probability distribution function; Contribution Matrix Each element in Quantitative First The feature vector of the i-th image patch is paired with the i-th image patch feature vector. The contribution level of each adaptive visual prototype, the set of adaptive visual prototypes The calculation method is as follows: ; in, Indicates transpose; The first After inputting the text prompts corresponding to the category labels into the text encoder, the text encoder will then... Category labeling features of layer output As a query feature vector, the adaptive visual prototype set As key and value feature vectors, visual information related to category label features is obtained from the adaptive visual prototype through a cross-attention mechanism, using the query projection matrix. Key projection matrix Value projection matrix This maps the query feature vector, key feature vector, and value feature vector to the same dimensional space. The resulting features are the query-mapped features. Key mapping features Sum value mapping features : ; in, , and Both represent the learnable projection matrix in cross-attention; Query mapping features based on the same feature dimension Key mapping features Sum value mapping features The attention mechanism is used to compute the visual feature representations related to the category label features in the adaptive visual prototype. : ; in, It is the dimension size of the query mapping features; Visual features are represented using a momentum update mechanism. Integrating into category labeling features : ; in, It is a category label feature that incorporates visual information. It is a hyperparameter that controls the degree of fusion of visual information.

[0010] Optionally, in the image classification method based on the fine-tuned biomedical visual language model, the adaptive prototype visual locator extracts image patch feature vectors at the shallow, middle, and deep layers of the image encoder.

[0011] Optionally, in the image classification method based on the fine-tuned biomedical visual language model, the overall loss function includes: cross-entropy loss, non-causal entropy maximization loss, orthogonal constraint loss, and semantic anchor regularization loss.

[0012] Optionally, the image classification method based on the fine-tuned biomedical visual language model, wherein the co-causal decoupling module decouples the target text prompt into causal and non-causal prompts, controls the learning of the biomedical visual language model through an overall loss function, suppresses pseudo-related information associated with the training category, and obtains the fine-tuned biomedical visual language model, specifically includes: The target text prompt is decoupled into causal prompts and non-causal prompts, for the first... The categories and causal clues are: Non-causal indication ,in, The first in the causal prompt Each word element, Indicates the first Category words corresponding to each category This represents an empty string or a placeholder. The first in the non-causal prompt Each word element; Calculate the causal image-text matching score matrix between the causal text cue features output by the text encoder and the image features output by the image encoder. Cross-entropy loss Detection information used to drive causal branch learning discriminative ability: ; ; in, Representing image features, Text features representing the causal text prompts output by the text encoder. Indicates transpose. Indicates temperature parameter, Indicates the number of input images. Indicates the true category label; Calculate the noncausal image-text matching score matrix between the noncausal text cue features output by the text encoder and the image features output by the image encoder. Non-causal entropy maximization loss Used to control non-causal branches to produce a uniform probability distribution: ; ; in, Text features representing non-causal text prompts output by the text encoder. Denotes KL divergence, Indicates a discrete uniform distribution. Indicates the number of categories. Indicates the first The text features of the first category of non-causal text prompts are similar to those of the second category. The inner product between the image features of the images. Indicates the first The probability of each category, ; Orthogonal constraint loss Used to ensure that the representation spaces of causal and non-causal prompts are separated from each other: ; in, This indicates retrieving the elements from the main diagonal of the matrix; Semantic anchor regularization loss Use manual tips Features used as anchor points: ; in, This represents the computation of the trace of a matrix. Text features representing the hand-crafted prompts output by the text encoder; The final overall loss function is the weighted sum of the individual loss terms. for: ; in, and It is a balancing hyperparameter; The overall loss function is used to control the learning of the biomedical visual language model. During the inference phase, the output of the causal branch is used for prediction, the results of the non-causal branch are discarded, and pseudo-related information associated with the training category is suppressed to obtain the fine-tuned biomedical visual language model.

[0013] Optionally, the image classification method based on the fine-tuned biomedical visual language model further includes the application of the fine-tuned biomedical visual language model in the following aspects: pre-annotation of medical image data, automatic archiving, sample sorting, data management and retrieval assistance.

[0014] Furthermore, to achieve the above objectives, the present invention also provides an image classification system based on a fine-tuned biomedical visual language model, wherein the image classification system based on the fine-tuned biomedical visual language model includes: The text prompt correction module is used to extract an adaptive visual prototype from the multi-scale representation of the input image through an adaptive prototype visual locator, and use the adaptive visual prototype to correct the text prompt to obtain the target text prompt. The visual language model fine-tuning module is used to decouple the target text prompt into causal and non-causal prompts through the co-causal decoupling module, control the learning of the biomedical visual language model through the overall loss function, suppress pseudo-related information associated with the training category, and obtain the fine-tuned biomedical visual language model. The biomedical image classification module is used to apply the fine-tuned biomedical visual language model to the automatic classification and recognition task of biomedical images. After processing the input biomedical images, it outputs category labels or identification information.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image classification program based on a fine-tuned biomedical visual language model stored in the memory and executable on the processor. When the image classification program based on the fine-tuned biomedical visual language model is executed by the processor, it implements the steps of the image classification method based on the fine-tuned biomedical visual language model as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image classification program based on a fine-tuned biomedical visual language model, and when the image classification program based on the fine-tuned biomedical visual language model is executed by a processor, it implements the steps of the image classification method based on the fine-tuned biomedical visual language model as described above.

[0017] In this invention, an adaptive prototype visual locator extracts an adaptive visual prototype from the multi-scale representation of the input image, and uses the adaptive visual prototype to correct text prompts to obtain target text prompts. A co-causal decoupling module decouples the target text prompts into causal and non-causal prompts, and controls the learning of the biomedical visual language model through an overall loss function, suppressing pseudo-related information associated with the training category to obtain a fine-tuned biomedical visual language model. This fine-tuned biomedical visual language model is then applied to the automatic classification and recognition task of biomedical images, processing the input biomedical images and outputting category labels or identification information. This invention provides a framework for generating image-conditional dynamic prompts, enabling prompts to adaptively adjust based on the fine-grained features of the input image. It utilizes multi-scale adaptive visual prototypes for prompt generation, extracting different levels of visual information from the shallow, middle, and deep layers of the image, preserving multi-scale semantics from coarse-grained anatomical structures to fine-grained pathological textures, and avoiding the loss of detail caused by global averaging. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the image classification method based on a fine-tuned biomedical visual language model of the present invention; Figure 2 This is a schematic diagram of the composition of the biomedical visual language model in a preferred embodiment of the image classification method based on the fine-tuned biomedical visual language model of the present invention. Figure 3 This is a schematic diagram of the front-door adjustment strategy adopted by the collaborative causal decoupling module in a preferred embodiment of the image classification method based on the fine-tuned biomedical visual language model of the present invention. Figure 4 This is a structural diagram of a preferred embodiment of the image classification system based on a fine-tuned biomedical visual language model of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] Pre-trained visual language models have become a crucial foundational model in the field of computer vision. The CLIP model, pre-trained on approximately 400 million image-text pairs through contrastive learning, demonstrated powerful zero-shot generalization capabilities. Inspired by this success, researchers have developed several dedicated visual language models for the biomedical field, including PMC-CLIP, PubMedCLIP, and BiomedCLIP. BiomedCLIP, employing BERT as the text encoder and VisionTransformer as the visual encoder, was pre-trained on the PMC-15M dataset (containing 15 million biomedical image-text pairs) and achieved significant performance improvements across various biomedical visual language tasks. Despite the strong zero-shot capabilities of these pre-trained models, few-shot adaptation for specific downstream tasks remains necessary, as task-specific fine-tuning can further enhance the model's performance in the target domain.

[0021] Recent research has explored parameter-efficient strategies to enable visual language models to adapt to downstream tasks with limited labeled data. These methods mainly fall into two categories: The first is adapter methods. These methods train only a small number of parameters in the adapter while keeping the pre-trained backbone network frozen, thus achieving effective task adaptation while reducing computational overhead. Representative works include methods such as MMA. The second category is prompt-based methods. These methods, while keeping the pre-trained backbone network frozen, introduce learnable prompts to provide the model with task-related contextual information. Representative works include methods such as CoOp. CoOp replaces static, hand-crafted prompt templates with learnable continuous context vectors, allowing the prompts to be optimized for specific datasets. CoCoOp further builds upon CoOp by generating image-conditional dynamic prompts, introducing a lightweight meta-network to predict prompt offsets related to the input image. In addition, biomedical-specific methods such as BiomedCoOp have been proposed, integrating domain prior knowledge to improve model performance on medical tasks.

[0022] Causal inference theory has been widely applied in the field of representation learning in recent years. Existing research mainly explores two directions: counterfactual inference and causal intervention. In causal intervention methods, ICL-MSR uses backdoor adjustment to treat observable semantic attributes as proxy variables for confounding factors; CDC uses frontdoor adjustment to suppress spurious correlations. These methods provide new theoretical perspectives for solving the model generalization problem.

[0023] The closest prior art to this invention is the CoCoOp method. The core idea of ​​this method is to generate dynamic, image-conditional cues, rather than learning static cues fixed for each category. CoCoOp introduces a lightweight meta-network that takes global image features as input and outputs a cue offset, allowing the learned cue to be adjusted according to the input image. This design enables the model to generate more suitable cues based on image content when faced with unseen categories. However, CoCoOp has the following limitations: First, the method uses global image features for cue generation, which are obtained through average pooling, easily leading to the dilution of local detection details (such as small lung nodules); second, the global image representation is easily dominated by artifacts, watermarks, or background interference in the image; finally, the method fails to explicitly model and separate causal features from spurious features, limiting its performance in generalization tasks from base classes to new classes.

[0024] Analysis of existing technologies reveals the following main drawbacks: (1) Generalization limitations of static cues: Cue learning methods, such as CoOp, learn a fixed text cue for each category, which is aligned with the common visual patterns of the training images. Such static cues are difficult to match with the visual features of unseen categories, resulting in poor model performance in distinguishing different new categories and limited generalization ability.

[0025] (2) Loss of detail in global features: Although dynamic cueing methods such as CoCoOp can adjust the cue content according to the input image, they rely on global image features for cue generation. Global features are obtained by average pooling of all image patches, a process that blurs local detection features. In medical imaging, fine-grained pathological features (such as small lesion areas and subtle texture changes) are crucial for accurate detection, and excessive averaging of global features leads to the loss of this key information.

[0026] (3) Generalization degradation due to spurious correlations: Existing cue learning methods are susceptible to the effects of shortcut learning. The learned cue may be associated with non-detectable visual features in the image, such as device-specific markings, imaging artifacts, or background interference. These features are discriminative in the base class of the training dataset, but are absent or behave differently in the new class. In generalization scenarios from the base class to the new class, the model is prone to learning features that are only discriminative in the base class but cannot be transferred to the new class, thus causing confusion between causal features and spurious correlations.

[0027] The objectives of this invention are as follows: First, to design a framework for generating image-conditional dynamic cues, enabling cues to adaptively adjust based on the fine-grained features of the input image, rather than relying on fixed static cue templates; Second, to propose a method for cue generation using multi-scale adaptive visual prototypes, extracting visual information from different levels of the image (shallow, mid-, and deep), preserving multi-scale semantics from coarse-grained anatomical structures to fine-grained pathological textures, and avoiding detail loss due to global averaging; Third, to introduce a front-door adjustment mechanism from causal inference theory, explicitly decoupling the cue representation into causal and non-causal features, and through collaborative learning objectives such as classification loss, non-causal entropy maximization loss, and orthogonal constraints, enabling the model to learn transferable detection features while suppressing spurious correlations related to the training category, thus solving the generalization degradation problem caused by confusion between causal and spurious features.

[0028] The preferred embodiment of the image classification method based on a fine-tuned biomedical visual language model of the present invention includes an adaptive prototype visual locator and a co-causal decoupling module; as shown in the following example. Figure 1 and Figure 2 As shown, the image classification method based on the fine-tuned biomedical visual language model includes the following steps: Step S10: The adaptive prototype visual locator extracts an adaptive visual prototype from the multi-scale representation of the input image, and uses the adaptive visual prototype to correct the text prompt to obtain the target text prompt.

[0029] This invention employs a pre-trained biomedical visual language model, BiomedCLIP (Causal Conditional Prompt Learning), as the backbone model. This model includes an image encoder and a text encoder. For C categories, each category has a corresponding category name text. In the few-shot adaptation setting, only K labeled samples are provided for each category (K is typically 1, 2, 4, 8, or 16). During training, only the learnable parameters related to the prompts are updated, keeping the pre-trained encoder frozen. In the base-to-new-class generalization setting, only labeled samples of the base class are used during training, and the classification performance of both the base class and the new class needs to be evaluated during testing.

[0030] Specifically, the core idea of ​​the Visual Grounder with Adaptive Prototype (VGAP) is to extract adaptive visual prototypes from the multi-scale representation of the input image. These prototypes are representative visual features obtained by weighted aggregation of multiple patch features of the input image, calculated by a lightweight learnable module based on their contribution weights. The adaptability is reflected in the end-to-end optimization of its parameters through backpropagation, which can adaptively change with the content of the input image and the downstream task objective. These prototypes are then used to correct text prompts. Specifically, VGAP extracts patch tokens (image patch feature labels or image patch feature vectors: feature representations obtained by mapping each small region after dividing the input image into multiple small regions by the visual encoder) in the shallow, middle, and deep layers of the image encoder. A lightweight network aggregates a large number of patch tokens into a small number of adaptive visual prototypes. These adaptive visual prototypes correct the features of the text prompts through a cross-attention mechanism, enabling the text prompts to align with the multi-scale semantic information in medical images, ranging from coarse-grained anatomical structures to fine-grained pathological textures.

[0031] The VGAP module is designed to address the limitations of static cues in generalization and the loss of details in global features. This module generates image conditional cues through multi-scale adaptive visual prototypes (image conditional cues are cues that are dynamically generated or adjusted based on the visual features of the current input image. Unlike pre-fixed static cues, they can change with the content of the input image to better align the cues with the semantic features of the current image).

[0032] Adaptive Prototype Generation: Traditional clustering-based prototyping methods generate prototypes solely based on visual similarity, independent of downstream tasks. The adaptive prototype proposed in this invention is generated by a lightweight, learnable module and optimized through backpropagation. This end-to-end optimization approach enables the prototype to adapt to the classification target, generating task-relevant representations rather than relying solely on geometric similarity. The specific implementation is as follows: Image After being input into the image encoder, in the image encoder's... The set of image block feature vectors output by the block is , Indicates the number of image patches. Indicates the first Feature vectors of image patches.

[0033] The Adaptive Prototype Visual Positioner (VGAP) uses a contribution matrix , For the scope, Aggregate into An adaptive visual prototype ( much smaller ): ; in, Represents the first feature projection matrix. , This represents the second characteristic projection matrix. , It is the dimension of the image patch feature vector. It is the dimension of the intermediate feature vector. Representation layer normalization, This represents the activation function. This represents the probability distribution function.

[0034] Contribution Matrix Each element in Quantitative First The feature vector of the i-th image patch is paired with the i-th image patch feature vector. The contribution level of each adaptive visual prototype, the set of adaptive visual prototypes The calculation method is as follows: ; in, This indicates transpose.

[0035] Cross-attention fusion: the first After inputting the text prompts corresponding to the category labels into the text encoder, the text encoder will then... Category labeling features of layer output As a query feature vector, the adaptive visual prototype set As key and value feature vectors, visual information related to category label features is obtained from the adaptive visual prototype through a cross-attention mechanism. First, to ensure that attention computation is performed in the same feature dimension space, a query projection matrix is ​​used. Key projection matrix Value projection matrix This maps the query feature vector, key feature vector, and value feature vector to the same dimensional space. The resulting features are the query-mapped features. Key mapping features Sum value mapping features : ; in, , and Both represent the learnable projection matrix in cross-attention.

[0036] Then, query mapping features based on the same feature dimensions. Key mapping features Sum value mapping features The attention mechanism is used to compute the visual feature representations related to the category label features in the adaptive visual prototype. : ; in, It is the dimension size of the query mapping features.

[0037] Visual features are represented using a momentum update mechanism. Integrating into category labeling features : ; in, It is a category label feature that incorporates visual information. It is a hyperparameter that controls the degree of fusion of visual information.

[0038] Multi-scale fusion strategy: VGAP is applied at three scales (shallow, intermediate, and deep) of the image encoder, specifically at layers 3, 7, and 11. This design ensures that the final text representation can robustly locate multi-scale detection details.

[0039] Step S20: The co-causal decoupling module decouples the target text prompt into causal prompts and non-causal prompts, and controls the learning of the biomedical visual language model through the overall loss function to suppress pseudo-related information associated with the training category, thereby obtaining a fine-tuned biomedical visual language model.

[0040] Specifically, the Synergistic Causal Disentanglement (SCD) module decouples the target text prompts into two branches: causal prompts and non-causal prompts, corresponding to transferable detection features and training category-specific pseudo-correlation features, respectively. Three collaborative learning objectives work together: cross-entropy loss drives causal prompts to learn discriminative detection information; non-causal entropy maximization loss forces non-causal prompts to produce a uniform probability distribution, reducing their predictive power; and orthogonality constraint loss ensures that causal and non-causal prompts are orthogonal to each other. Regarding the overall learning objective, this invention also introduces semantic anchor regularization loss, which uses the features of hand-crafted category prompts as semantic anchors to stabilize the training process. The final overall loss function is a weighted sum of the loss terms.

[0041] The design goal of the SCD module is to solve the spurious correlation problem in cue learning, enabling the model to learn transferable causal detection features.

[0042] like Figure 3As shown, this invention constructs a structural causal model to formally analyze the generalization process from a base class to a new class. This model includes the following variables: input image. Learning text prompts Image-text alignment semantics Confounding factors and prediction labels .in, This indicates the semantic information shared between the image and text prompts; This represents non-diagnostic visual features present in the base class, such as device-specific markings or imaging artifacts. Confounding factors. As a common cause, it simultaneously affects The formation (path) ) and prediction labels (path This opened a backdoor path. This introduces spurious correlations and impairs the generalization of new classes. Due to confounding factors... Since it cannot be directly observed, backdoor adjustments cannot be directly applied. This invention employs a frontdoor adjustment strategy, introducing an intermediate variable—image-text alignment causal semantics. . This indicates the association between textual prompts and underlying diagnostic visual features that are transferable between the base class and the new class, such as basic lesion morphology, tissue texture, and abnormal density regions.

[0043] Decoupling Causal and Non-Causal Hints: The target text hints are decoupled into causal and non-causal hints. For the first... The categories and causal clues are: Non-causal indication ,in, The first in the causal prompt Each word element, Indicates the first Category words corresponding to each category This represents an empty string or a placeholder. The first in the non-causal prompt Each word element.

[0044] Collaborative Learning Objectives: SCD achieves causal decoupling through three collaborative objectives. These include calculating the causal image-text matching score matrix between the causal text prompt features output by the text encoder and the image features output by the image encoder. Cross-entropy loss Detection information used to drive causal branch learning discriminative ability: ; ; in, Representing image features, Text features representing the causal text prompts output by the text encoder. Indicates transpose. Indicates temperature parameter, Indicates the number of input images. This represents the actual category label.

[0045] Calculate the noncausal image-text matching score matrix between the noncausal text cue features output by the text encoder and the image features output by the image encoder. Non-causal entropy maximization loss Used to control non-causal branches to produce a uniform probability distribution, making them non-discriminatory: ; ; in, Text features representing non-causal text prompts output by the text encoder. Denotes KL divergence, Indicates a discrete uniform distribution. Indicates the number of categories. Indicates the first The text features of the first category of non-causal text prompts are similar to those of the second category. The inner product between the image features of the images. Indicates the first The probability of each category, Minimize loss This is equivalent to maximizing the prediction entropy, making the predictions of non-causal branches uncertain.

[0046] Orthogonal constraint loss Used to ensure that the representation spaces of causal and non-causal prompts are separated from each other: ; in, This indicates retrieving the elements from the main diagonal of the matrix.

[0047] Semantic anchor regularization: This invention also introduces semantic anchor regularization loss. Semantic anchor regularization loss Use manual tips (For example The characteristics of ) are used as anchor points: ; in, This indicates that the trace of a matrix is ​​calculated, which is equivalent to calculating the sum of the elements on the main diagonal. Text features representing the hand-crafted prompts output by the text encoder.

[0048] The final overall loss function is the weighted sum of the individual loss terms. for: ; in, and It is a balancing hyperparameter; The overall loss function is used to control the learning of the biomedical visual language model. During the inference phase, the output of the causal branch is used for prediction, the results of the non-causal branch are discarded, and pseudo-related information associated with the training category is suppressed to obtain the fine-tuned biomedical visual language model.

[0049] Step S30: Apply the fine-tuned biomedical visual language model to the automatic classification and recognition task of biomedical images. After processing the input biomedical images, output category labels or identification information.

[0050] Specifically, after obtaining the fine-tuned biomedical visual language model, it can be applied to the automatic classification and recognition task of biomedical images. For this task, the biomedical image is input into the fine-tuned model, which then processes the image to obtain category labels or identification information. Furthermore, the applications of the fine-tuned biomedical visual language model also include: pre-annotation of medical image data, automatic archiving, sample sorting, data management, and retrieval assistance.

[0051] This invention employs a multi-scale adaptive visual prototype for cue generation, which can preserve multi-scale information from coarse-grained anatomical structures to fine-grained pathological textures. This invention explicitly decouples cuees into causal and non-causal components through the SCD module, allowing the model to focus on transferable detection features. This invention does not require the injection of external domain knowledge and can learn task-related representations through pure end-to-end training.

[0052] The feasibility of this invention has been fully verified through experiments. This invention was systematically evaluated on a benchmark comprising 11 biomedical datasets covering 9 imaging modalities, such as computed tomography (CTKidney), dermoscopy (DermaMNIST), endoscopy (Kvasir), fundus photography (RETINA), histopathology (LC25000, CHMNIST), magnetic resonance imaging (BTMRI), optical coherence tomography (OCTMNIST), ultrasound (BUSI), and X-ray (COVID-QU-Ex, KneeXray).

[0053] This invention evaluates two core tasks: few-shot classification, assessing the model's adaptability with limited labeled samples; and base-to-new-class generalization, evaluating the model's knowledge transfer ability from training classes to unseen classes. In the few-shot classification task, training configurations of 1, 2, 4, 8, and 16 samples per class were set. Results show that this invention consistently outperforms existing methods across all sample settings. In the base-to-new-class generalization task, the model is trained on the base class and then evaluated on both the base and new classes. The Harmonic Mean (HM) metric is used to measure cross-class balance performance. This invention achieves an HM of 79.98%, a 6.45 percentage point improvement over the best baseline method, MMA.

[0054] This invention employs multi-scale feature extraction at layers 3, 7, and 11 of the image encoder. Alternative solutions could explore feature extraction at other layer combinations, such as layers 4, 8, and 12. The default prototype number K is set to 14 (the square root of 196 patches), but alternative solutions can adjust the prototype number based on task complexity. This invention uses orthogonality to separate the learning spaces of causal and non-causal cues; alternative solutions could utilize other more effective separation methods.

[0055] Furthermore, such as Figure 4 As shown, based on the above-mentioned image classification method based on a fine-tuned biomedical visual language model, this invention also provides an image classification system based on a fine-tuned biomedical visual language model, wherein the image classification system based on the fine-tuned biomedical visual language model includes: The text prompt correction module 51 is used to extract an adaptive visual prototype from the multi-scale representation of the input image through an adaptive prototype visual locator, and use the adaptive visual prototype to correct the text prompt to obtain the target text prompt. The visual language model fine-tuning module 52 is used to decouple the target text prompt into causal prompts and non-causal prompts through the co-causal decoupling module, control the learning of the biomedical visual language model through the overall loss function, suppress pseudo-related information related to the training category, and obtain the fine-tuned biomedical visual language model. The biomedical image classification module 53 is used to apply the fine-tuned biomedical visual language model to the automatic classification and recognition task of biomedical images. After processing the input biomedical images, it outputs category labels or identification information.

[0056] Furthermore, such as Figure 5 As shown, based on the above-mentioned image classification method and system based on the fine-tuned biomedical visual language model, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0057] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores an image classification program 40 based on a fine-tuned biomedical visual language model, which can be executed by the processor 10 to implement the image classification method based on the fine-tuned biomedical visual language model in this application.

[0058] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the image classification method based on the fine-tuned biomedical visual language model.

[0059] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminal's processor 10, memory 20, and display 30 communicate with each other via a system bus.

[0060] In one embodiment, when the processor 10 executes the image classification program 40 based on the fine-tuned biomedical visual language model in the memory 20, it implements the steps of the image classification method based on the fine-tuned biomedical visual language model as described above.

[0061] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image classification program based on a fine-tuned biomedical visual language model, and the image classification program based on the fine-tuned biomedical visual language model, when executed by a processor, implements the steps of the image classification method based on the fine-tuned biomedical visual language model as described above.

[0062] In summary, this invention provides an image classification method, system, terminal, and computer-readable storage medium based on a fine-tuned biomedical visual language model. The method includes: an adaptive prototype visual locator extracting an adaptive visual prototype from the multi-scale representation of the input image; using the adaptive visual prototype to correct text prompts to obtain target text prompts; a co-causal decoupling module decoupling the target text prompts into causal and non-causal prompts; controlling the learning of the biomedical visual language model through an overall loss function to suppress pseudo-related information associated with the training category, resulting in a fine-tuned biomedical visual language model; and applying the fine-tuned biomedical visual language model to an automatic classification and recognition task of biomedical images, processing the input biomedical images, and outputting category labels or identification information. This invention can generate a framework for conditional dynamic image prompts, enabling prompts to adaptively adjust according to the fine-grained features of the input image. It utilizes multi-scale adaptive visual prototypes for prompt generation, extracting different levels of visual information from the shallow, middle, and deep layers of the image, preserving multi-scale semantics from coarse-grained anatomical structures to fine-grained pathological textures, and avoiding the loss of detail caused by global averaging.

[0063] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0064] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0065] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An image classification method based on fine-tuned biomedical visual language model, characterized in that, The biomedical visual language model includes an adaptive prototype visual locator and a collaborative causal decoupling module. The image classification method based on the fine-tuned biomedical visual language model includes: The adaptive prototype visual locator extracts an adaptive visual prototype from the multi-scale representation of the input image, and uses the adaptive visual prototype to correct the text prompt to obtain the target text prompt. The collaborative causal decoupling module decouples the target text prompt into causal and non-causal prompts, controls the learning of the biomedical visual language model through the overall loss function, suppresses pseudo-related information associated with the training category, and obtains a fine-tuned biomedical visual language model. The finely tuned biomedical visual language model is applied to the automatic classification and recognition task of biomedical images. After processing the input biomedical images, it outputs category labels or identification information.

2. The method of claim 1, wherein the fine-tuned biomedical visual language model is a CLIP model. The biomedical visual language model includes an image encoder and a text encoder.

3. The method of claim 2, wherein the fine-tuned biomedical visual language model is a CLIP model. The adaptive prototype visual locator extracts an adaptive visual prototype from the multi-scale representation of the input image, and uses the adaptive visual prototype to correct the text prompt to obtain the target text prompt, specifically including: Image After being input into the image encoder, in the image encoder's... The set of image block feature vectors output by the block is , Indicates the number of image patches. Indicates the first Feature vectors of image patches; The adaptive prototype visual locator uses a contribution matrix , For the scope, Aggregate into An adaptive visual prototype: ； in, Denotes the first feature projection matrix. , This represents the second characteristic projection matrix. , It is the dimension of the image patch feature vector. It is the dimension of the intermediate feature vector. Representation layer normalization, This represents the activation function. Represents the probability distribution function; Contribution Matrix Each element in Quantitative First The feature vector of the i-th image patch is paired with the i-th image patch feature vector. The contribution level of each adaptive visual prototype, the set of adaptive visual prototypes The calculation method is as follows: ； in, Indicates transpose; The first After inputting the text prompts corresponding to the category labels into the text encoder, the text encoder will then... Category labeling features of layer output As a query feature vector, the adaptive visual prototype set As key and value feature vectors, visual information related to category label features is obtained from the adaptive visual prototype through a cross-attention mechanism, using the query projection matrix. Key projection matrix Value projection matrix This maps the query feature vector, key feature vector, and value feature vector to the same dimensional space. The resulting features are the query-mapped features. Key mapping features Sum value mapping features : ； in, , and Both represent the learnable projection matrix in cross-attention; Query mapping features based on the same feature dimension Key mapping features Sum value mapping features The attention mechanism is used to compute the visual feature representations related to the category label features in the adaptive visual prototype. : ； in, It is the dimension size of the query mapping features; Visual features are represented using a momentum update mechanism. Integrating into category labeling features : ； in, It is a category label feature that incorporates visual information. It is a hyperparameter that controls the degree of visual information fusion.

4. The image classification method based on a fine-tuned biomedical visual language model according to claim 2 or 3, characterized in that, The adaptive prototype visual locator extracts image patch feature vectors in the shallow, middle and deep layers of the image encoder.

5. The image classification method based on a fine-tuned biomedical visual language model according to claim 1, characterized in that, The overall loss function includes: cross-entropy loss, non-causal entropy maximization loss, orthogonal constraint loss, and semantic anchor regularization loss.

6. The image classification method based on a fine-tuned biomedical visual language model according to claim 5, characterized in that, The co-causal decoupling module decouples the target text prompt into causal and non-causal prompts. It controls the learning of the biomedical visual language model through an overall loss function, suppressing pseudo-related information associated with the training category to obtain a fine-tuned biomedical visual language model, specifically including: The target text prompt is decoupled into causal prompts and non-causal prompts, for the first... The categories and causal clues are: Non-causal indication ,in, The first in the causal prompt Each word element, Indicates the first Category words corresponding to each category This represents an empty string or a placeholder. The first in the non-causal prompt Each word element; Calculate the causal image-text matching score matrix between the causal text cue features output by the text encoder and the image features output by the image encoder. Cross-entropy loss Detection information used to drive causal branch learning discriminative ability: ； ； in, Representing image features, Text features representing the causal text prompts output by the text encoder. Indicates transpose. Indicates temperature parameter, Indicates the number of input images. Indicates the true category label; Calculate the noncausal image-text matching score matrix between the noncausal text cue features output by the text encoder and the image features output by the image encoder. Non-causal entropy maximization loss Used to control non-causal branches to produce a uniform probability distribution: ； ； in, Text features representing non-causal text prompts output by the text encoder. Denotes KL divergence, Indicates a discrete uniform distribution. Indicates the number of categories. Indicates the first The text features of the first category of non-causal text prompts are similar to those of the second category. The inner product between the image features of the images. Indicates the first The probability of each category ; Orthogonal constraint loss This is used to ensure that the representation spaces of causal and non-causal prompts are separated from each other: ； in, This indicates retrieving the elements from the main diagonal of the matrix; Semantic anchor regularization loss Use manual tips Features used as anchor points: ； in, This represents the computation of the trace of a matrix. Text features representing the hand-crafted prompts output by the text encoder; The final overall loss function is the weighted sum of the individual loss terms. for: ； in, and It is a balancing hyperparameter; The overall loss function is used to control the learning of the biomedical visual language model. During the inference phase, the output of the causal branch is used for prediction, the results of the non-causal branch are discarded, and pseudo-related information associated with the training category is suppressed to obtain the fine-tuned biomedical visual language model.

7. The image classification method based on a fine-tuned biomedical visual language model according to claim 1, characterized in that, The applications of the fine-tuned biomedical visual language model also include: pre-annotation of medical image data, automatic archiving, sample sorting, data management, and retrieval assistance.

8. An image classification system based on a fine-tuned biomedical visual language model, characterized in that, The image classification system based on the fine-tuned biomedical visual language model includes: The text prompt correction module is used to extract an adaptive visual prototype from the multi-scale representation of the input image through an adaptive prototype visual locator, and use the adaptive visual prototype to correct the text prompt to obtain the target text prompt. The visual language model fine-tuning module is used to decouple the target text prompt into causal and non-causal prompts through the co-causal decoupling module, control the learning of the biomedical visual language model through the overall loss function, suppress pseudo-related information associated with the training category, and obtain the fine-tuned biomedical visual language model. The biomedical image classification module is used to apply the fine-tuned biomedical visual language model to the automatic classification and recognition task of biomedical images. After processing the input biomedical images, it outputs category labels or identification information.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and an image classification program based on a fine-tuned biomedical visual language model stored in the memory and executable on the processor. When the image classification program based on the fine-tuned biomedical visual language model is executed by the processor, it implements the steps of the image classification method based on the fine-tuned biomedical visual language model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image classification program based on a fine-tuned biomedical visual language model, which, when executed by a processor, implements the steps of the image classification method based on a fine-tuned biomedical visual language model as described in any one of claims 1-7.

Images