A long-tail medical image classification method based on lesion-aware double calibration network

Abstract

The application discloses a long-tail medical image classification method based on a lesion perception double-calibration network, which comprises the following steps: step 1: a lesion perception feature extraction is used to construct a lesion perception feature modeling mechanism, and a model is guided to pay attention to a local area in a feature extraction process; step 2: a semantic double-calibration and classification modeling is constructed on the basis of a lesion perception visual feature; and step 3: a cross-domain adaptation and joint optimization is introduced in a model training process, a parameter-efficient cross-domain adaptation strategy is introduced, and a visual feature extraction and a text semantic encoding process are jointly optimized. Through the combination of the lesion perception feature modeling and the semantic double-calibration, local discriminative information is enhanced at a visual level, and class confusion is relieved at a semantic level, so that the overall performance of head, middle and tail classes is improved, and an efficient and reliable technical scheme is provided for intelligent identification of medical long-tail distribution data.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and long-tail recognition, and relates to a long-tail medical image classification method based on a lesion perception dual calibration network, which can be used for medical image recognition. Background Technology

[0002] Medical image classification is a key technology in intelligent assisted diagnostic systems, widely used in clinical scenarios such as skin lesion screening and fundus disease diagnosis. However, due to factors such as high data acquisition and annotation costs and significant differences in disease incidence rates, medical image data generally exhibits a long-tail distribution, meaning that there are sufficient samples of a few common diseases, while samples of a large number of clinically critical but low-incidence disease categories are severely insufficient. Under these conditions, existing models are prone to being dominated by head categories during training, with the decision boundary shifting towards head categories. This leads to a significant decrease in the recognition ability of middle and tail categories, thereby increasing the risk of misdiagnosis and missed diagnosis of rare and high-risk diseases, and limiting the reliability of the models in clinical applications.

[0003] For the long-tail image classification problem in medicine, existing methods mainly focus on two aspects: visual feature representation enhancement and data distribution adjustment. These include lesion feature modeling based on convolutional networks and attention mechanisms, as well as strategies such as resampling, reweighting, contrastive learning, and generative data augmentation. Although these methods alleviate the class imbalance problem to some extent, they still rely on information from a single visual modality. When samples in the tail class are extremely scarce, the model struggles to learn sufficiently discriminative feature representations, resulting in limited performance improvements.

[0004] In recent years, vision-language pre-trained models have demonstrated strong generalization capabilities in zero-shot and few-shot classification tasks by jointly modeling image and text semantics, providing a new approach for medical long-tail recognition. However, these models are typically pre-trained on natural images and general text data, and their image encoders tend to focus on global semantic modeling, making it difficult to effectively characterize local lesion features with diagnostic value in medical images. Simultaneously, their text semantic representations often rely on simple category labels, which can easily lead to inter-class confusion when categories are similar and extremely imbalanced. Furthermore, existing cross-domain adaptation methods often focus on single-path image-text alignment, lacking collaborative modeling of lesion perception features and multi-level semantic information, making it difficult to significantly improve the recognition capabilities of middle and tail categories while maintaining the performance of head categories.

[0005] To address the aforementioned issues, this invention proposes a lesion-aware dual-calibrated network for long-tailed medical image classification (LADC-Net). This method is based on a vision-language pre-trained model. By introducing a lesion-aware feature modeling mechanism, it guides the model to focus on diagnostically significant local lesion regions in medical images. Combined with a multi-path semantic calibration strategy, it achieves effective alignment between visual features and disease semantics, thereby mitigating inter-class confusion caused by class imbalance and improving the model's ability to identify middle and tail categories under long-tailed distribution conditions. Summary of the Invention

[0006] This invention proposes a long-tail medical image classification method based on a lesion-sensing dual-calibration network. This method, based on a vision-language pre-trained model, effectively improves the medical image classification model's ability to recognize different categories, especially the middle and tail categories, under long-tail distribution conditions by introducing a lesion-sensing feature modeling mechanism and a dual-path semantic calibration strategy.

[0007] LADC-Net enhances the model's ability to focus on key lesion regions in medical images through implicit localization, overcoming the problem of traditional vision-language models over-relying on global semantic features and neglecting local diagnostic information. Simultaneously, by constructing a semantic dual calibration mechanism, it introduces complementary disease semantic representations to calibrate classification results, thereby mitigating inter-class confusion caused by the combined effects of class similarity and sample imbalance. This method significantly improves the classification performance of middle and tail classes while maintaining stable performance for head classes, achieving overall performance balance in long-tail medical scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] Step 1: Lesion perception feature extraction. Considering the localized and irregular nature of lesion areas in medical images, a lesion perception feature modeling mechanism is constructed to guide the model to focus on local areas with diagnostic value during feature extraction. The saliency information of lesions is fused with global visual features to enhance the discriminative representation ability of medical images and avoid information loss caused by relying solely on global semantic features.

[0010] Step 2: Semantic Dual Calibration and Classification Modeling. Based on the visual features perceived by lesions, a semantic dual calibration strategy is constructed. On the one hand, the textual semantic information corresponding to the disease category is used to perform cross-modal alignment with medical image features to obtain classification results based on textual semantics. On the other hand, learnable category semantic centers with semantic initialization are trained to model the relationship between image features and category centers, forming an auxiliary semantic calibration path. By fusing the above two types of semantic calibration results, the classification output is comprehensively calibrated to reduce inter-class confusion caused by category similarity and sample imbalance.

[0011] Step 3: Cross-domain adaptation and joint optimization. Considering the significant domain differences between medical images and natural images, a parameter-efficient cross-domain adaptation strategy is introduced during model training. The visual feature extraction and text semantic encoding processes are jointly optimized, enabling the model to effectively adapt to medical domain features and semantics without significantly increasing the parameter scale, thereby further improving classification stability and generalization ability under long-tail distribution conditions.

[0012] Traditional medical long-tail image classification methods typically rely on single visual modalities or single-path semantic alignment, which can easily lead to a decline in head category performance while improving tail category performance. This invention combines lesion-perceptual feature modeling with semantic dual calibration to enhance local discriminative information at the visual level and alleviate inter-class confusion at the semantic level, achieving an overall improvement in the performance of head, middle, and tail categories. This method is characterized by its general structure, strong adaptability, and good scalability, and can be widely applied to various medical image long-tail classification scenarios, providing an efficient and reliable technical solution for the intelligent identification of medical long-tail distributed data. Attached Figure Description

[0013] Figure 1 A framework diagram for long-tailed medical image classification based on a lesion-sensing dual calibration network;

[0014] Figure 2 Visualization of the category decision space after introducing different components into LADC-Net;

[0015] Figure 3 Attention heatmaps of the visual encoder before and after the introduction of the Lesion Perception Feature Fusion (LAFF) module were compared. Detailed Implementation

[0016] Network framework introduction: feature extraction for lesion perception, dual-path score fusion, fine-tuning of disease perception in the visual-language model, and loss function.

[0017] Feature extraction network for lesion perception: LADC-Net model, such as Figure 1As shown, it mainly consists of a CLIP encoder and the modules proposed in this paper: a lesion-aware feature fusion (LAFF) module and a dual-path score fusion (DPLF) module. Furthermore, we enhance the CLIP text encoder through text fine-tuning (P-Tuning) and adjust the CLIP image encoder using an adapter to better capture features from the medical field.

[0018] Vision-language models like CLIP have demonstrated impressive capabilities in aligning the visual and textual semantics of natural images, leading to their widespread adoption. However, medical images exhibit significantly different features from natural images, focusing more on lesion regions. Therefore, the image encoder in CLIP often fails to adequately address lesion regions crucial for diagnostic tasks. To address this limitation, we propose a lesion-aware medical image feature extraction method that adaptively captures spatial heatmaps of lesion regions in images. This method leverages cross-attention to couple these spatial features with the semantic features of the CLIP image encoder, achieving reliable implicit lesion localization and feature enhancement. Specifically, the method comprises several steps: using Conv4Net... Feature extraction is performed; through The system generates patch-level lesion perception heatmaps (HMs); converts these lesion perception heatmaps into attention weights through block weighted integration (PWI); and global information perception (GIA) combines lesion heatmap information with PatchToken representations through a self-attention mechanism, and further fuses them with classification tokens (CLS tokens) to enhance the global perception capability of CLS tokens within the visual encoder.

[0019] We define the long-tail medical image dataset as ,in It is the first One input image, This is its corresponding category label. Here and Let represent the total number of categories and the total number of samples, respectively. The dataset exhibits a significant class imbalance, where... Indicates the first The number of samples per class. Dataset processing relies on the CLIP text encoder. and CLIP image encoder First, the input image is processed by Conv4Net (a four-layer convolutional neural network) to extract low-level visual features through convolution operations. .

[0020] (1)

[0021] Then, image features It is input into HM to generate a heatmap, which is then compressed to a size of The single-channel representation is denoted as This resolution is consistent with the number of image patches in the CLIP image encoder (ViT-B / 16).

[0022] (2)

[0023] Subsequently, After PWI processing, the heatmap is injected into the visual tokens (PatchTokens) of the visual Transformer. In ), specifically, the original The attention heatmap was flattened into 196 spatial weights, normalized, and further converted to learnable coefficients. Scaling is used to emphasize lesion-related areas within the patch. The enhanced patch representation... The calculation is as follows:

[0024] (3)

[0025] This process not only supports subsequent patch-by-patch enhancement operations but also implicitly encodes lesion localization information, laying the foundation for further analysis. Finally, in the GIA step, feature-weighted pooling is applied to... Region heatmaps normalized with Softmax Fusion to generate region-aware image features Meanwhile, by embedding the position of the CLIP image encoder... Heatmap with Softmax normalization Combined, location encoding weighted pooling is performed to generate location features that assign higher weights to the lesion region. These location features were subsequently incorporated. The final image representation is obtained. This encodes implicit lesion location information. The detailed process definition is as follows:

[0026] (4)

[0027] in , express The number of spatial locations within. Simultaneously, the query vector. With keys derived from CLIP image Patch Tokens ( ) and value ( Together, the matrices are transformed through learnable linear transformations. The representation is projected onto a unified low-dimensional feature space. The resulting representation is then processed through cross-attention to generate a global context representation. :

[0028] (5)

[0029] Subsequently, pass Upsampling, and using learnable weighting coefficients Represented by CLS Token Fusion enhances the global feature representation while assigning greater importance to lesion-related regions. The fusion formula is as follows:

[0030] (6)

[0031] Finally, the merged Patch Tokens and CLS Tokens are concatenated and input into the CLIP image encoder. To extract classification features .

[0032] Dual-Path Score Fusion: Medical images often exhibit severe inter-class imbalance and complex intra-class variability. Specifically, common diseases are associated with a large number of samples, while rare diseases are associated with very few. This leads to embedding vectors from the majority class dominating the feature space, while representations of the minority class are often not robust enough. Therefore, when calculating classification scores based on cosine similarity, the model tends to favor the majority class prediction. To address this issue, we propose a Dual-Path Score Fusion (DPLF) strategy that combines learnable class centers initialized with textual semantics and dynamically gated weighting. This method aims to mitigate inter-class confusion in imbalanced data distributions, particularly for the minority class, through joint loss optimization. In addition to image-label pairs, we define a set of textual descriptions. Each of them Representation and Category Relevant natural language descriptions. For each training sample The associated category description is denoted as Each text description By CLIP text encoder Processing to extract its high-level semantic features, denoted as The complete set of category-level semantic features is collected into a matrix:

[0033] (7)

[0034] This matrix is ​​used to initialize a learnable class center matrix. , of which OK Corresponding to category Learning center This is the output dimension of the encoder. Next, given the image encoder via CLIP... from Extracted image features We calculate classification scores from two complementary branches, each capturing different semantic cues. The first branch is based on... With the center of learnable categories Cosine similarity calculation between categories The score is defined as:

[0035] (8)

[0036] In parallel, the second branch is computed Text features The similarity between them, of which Indicate category Learnable cues enhance text descriptions, as mentioned earlier, by applying trainable text prefixes:

[0037] (9)

[0038] To dynamically balance the contributions of the two branches, we introduce a learnable gating parameter. .category The fusion score is calculated as follows:

[0039] (10)

[0040] Disease Perception Fine-Tuning for Visual-Language Models: To effectively adapt pre-trained visual-language models (such as CLIP) to the medical imaging domain, we propose a disease perception fine-tuning strategy to jointly enhance text and image encoders in a lightweight and domain-sensitive manner. Specifically, we introduce a category-aware cue module (called P-Tuning) for the text encoder, which learns a set of trainable prefix embeddings to enrich the text encoder's understanding of medical-specific terms in a zero-shot setting. In parallel, we integrate a domain adaptation module (called Adapter) for the image encoder, embedding it into each Transformer layer to achieve targeted domain adaptation with minimal parameter overhead. This bimodal adaptation equips the model with fine-grained cross-modal alignment for lesion-rich content, significantly improving its performance on long-tailed medical categories.

[0041] To enhance the model's understanding of medical terminology, we introduce a category-aware cue module. This module learns a set of... 1 context token, and append them to each category name. Previously, the goal was to build medical perception prompts:

[0042] (11)

[0043] Each of them ( ) is a trainable vector with the same dimensions as the word embeddings used by CLIP (i.e., 512). The number of context tokens It is a hyperparameter that follows the CoOp setting. The sentence with enhanced prompts. It was then input into the frozen text encoder. In order to obtain category-specific text features with learnable context: .

[0044] The domain adaptation module, a standard Transformer encoder block, consists of two main sub-layers: a multi-head self-attention (MHSA) layer and a feedforward network (FFN) layer. The self-attention formula is as follows:

[0045] (12)

[0046] in Indicates the first The output tokens of the layer MHSA. These tokens are then passed through an FFN—comprising a LayerNorm followed by an MLP—to produce the final output of the block:

[0047] (13)

[0048] in Indicates the first The output of each encoder block. To achieve more efficient fine-tuning in medical imaging, this study employs the Adapter method, integrating the Adapter module into the FFN via residual connections. Given input features The Adapter module generates adaptive features through its transformation branch. The formula is as follows:

[0049] (14)

[0050] The resulting adaptive representation Subsequently, residual connections and processed features were used. The fusion produces the final output:

[0051] (15)

[0052] scalar The contribution of the adaptive features is controlled.

[0053] Loss Function: To effectively learn image feature representations and dual-path fractional probability predictions for long-tailed distributions in medical imaging, we employ... The loss function serves as a constraint. This loss function is specifically designed to address long-tail learning scenarios, and its formula is as follows:

[0054] (16)

[0055] in

[0056] Experimental Section

[0057] Experimental datasets: This invention was evaluated on two publicly available medical long-tail image datasets, ISIC2018 and APTOS2019.

[0058] The ISIC 2018 dataset contains 10,015 images of skin lesions, covering seven common skin disease categories: melanoma (MEL), melanocytic nevus (NV), basal cell carcinoma (BCC), actinic keratosis (AKIEC), benign keratosis (BKL), dermatofibroma (DF), and vascular lesions (VASC). The imbalance ratio of this dataset is ρ=58. The APTOS 2019 dataset contains 3,662 high-resolution color fundus images, classified into five severity pathological grades: no DR, mild DR, moderate DR, severe DR, and proliferative DR. The imbalance ratio of this dataset is ρ=9. The data imbalance ratio is... Defined as (The ratio of the number of samples in the category with the most samples to the number of samples in the category with the fewest samples) This value reflects the degree of imbalance in the data.

[0059] Evaluation indicators:

[0060] We use Top-1 accuracy and F1 score as evaluation metrics.

[0061] Experimental setup: This invention is built on the PyTorch framework and trained on a single RTX 4090 GPU with a batch size of 128. During data preprocessing, all samples were scaled from 256×256 and then cropped to 224×224 pixels. The entire process uses... The loss function was constrained, and the dataset was uniformly trained for 60 epochs on the ISIC2018 and APTOS2019 datasets. The SGD optimizer was chosen, with an initial learning rate set to [value missing]. Furthermore, a cosine learning rate scheduling strategy is used to optimize the model's convergence performance.

[0062] Comparative Experiments: This invention was compared and evaluated against several baseline methods and state-of-the-art long-tail recognition methods on the ISIC2018 and APTOS2019 datasets. Table 1 shows the quantitative results on the ISIC2018 and APTOS2019 datasets, achieving the best results in both.

[0063] Table 1: Top-1 accuracy (%) and F1 score (%) of different methods on the ISIC2018 and APTOS2018 datasets.

[0064]

[0065] Ablation experiments: This invention conducted a series of ablation studies on the ISIC2018 dataset to verify the function of each module. Table 2 shows the component ablation results of LADC-Net on the ISIC2018 dataset; the category decision space visualization after adding each component is shown below. Figure 2 As shown; attention heatmaps before and after adding the LAFF module, for example. Figure 3 As shown.

[0066] Table 2: Ablation studies of different components on the ISIC2018 dataset.

[0067]

Claims

1. A long-tail medical image classification method based on lesion-aware double calibration network, characterized by: Includes the following steps, Step 1: Extraction of lesion perception features. Construct a lesion perception feature modeling mechanism to guide the model to focus on local areas with diagnostic value during feature extraction, and fuse lesion saliency information with global visual features to enhance the discriminative representation ability of medical images. Step 2: Semantic Dual Calibration and Classification Modeling. Based on the visual features of lesion perception, a semantic dual calibration strategy is constructed. On the one hand, the textual semantic information corresponding to the disease category is used to perform cross-modal alignment with medical image features to obtain classification results based on textual semantics. On the other hand, learnable category semantic centers with semantic initialization are trained to model the relationship between image features and category centers, forming an auxiliary semantic calibration path. By fusing the above two types of semantic calibration results, the classification output is comprehensively calibrated to reduce inter-class confusion caused by category similarity and sample imbalance. Step 3: Cross-domain adaptation and joint optimization. Considering the domain differences between medical images and natural images, a parameter-efficient cross-domain adaptation strategy is introduced during model training to jointly optimize the visual feature extraction and text semantic encoding processes.

2. The long-tail medical image classification method based on a lesion-sensing dual-calibration network according to claim 1, characterized in that: The network framework for implementing the above method includes: a feature extraction network for lesion perception, dual-path score fusion, fine-tuning of disease perception in the visual-language model, and a loss function.

3. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 1, characterized in that: The feature extraction network for lesion perception is the LADC-Net model, which consists of a CLIP encoder and the modules proposed in this paper: a lesion perception feature fusion LAFF module and a dual-path score fusion DPLF module; the text encoder of CLIP is enhanced by text fine-tuning P-Tuning, and the image encoder of CLIP is adjusted using an adapter.

4. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 2, characterized in that: Couple these spatial features with the semantic features of the CLIP image encoder using cross-attention, which involves several steps: using Conv4Net Feature extraction is performed; through Generate patch-level lesion-sensing heatmaps (HMs); convert these lesion-sensing heatmaps into attention weights through block-weighted ensemble (PWI). In addition, Global Information Awareness (GIA) combines lesion heatmap information with PatchToken representation through a self-attention mechanism and merges it with classification token CLSToken, thereby enhancing the global awareness capability of CLSToken within the visual encoder.

5. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 1, characterized in that: Define the long-tail medical image dataset as ,in It is the first One input image, It is its corresponding category label; here and Let represent the total number of categories and the total number of samples, respectively. The dataset exhibits a significant class imbalance, where... Indicates the first The number of samples per class; the processing of the dataset depends on the CLIP text encoder. and CLIP image encoder First, the input image is processed by Conv4Net, which extracts low-level visual features through convolution operations. ; (1)； Then, image features are input into the HM to generate a heat map, which is compressed into a single channel representation of size denoted as ; (2)； Subsequently, After PWI processing, the heatmap is injected into the visual tokens (PatchTokens) of the visual Transformer. (in) the original The attention heatmap was flattened into 196 spatial weights, normalized, and calculated using learnable coefficients. Scaling, thereby emphasizing the lesion-related area within the patch; the enhanced patch representation The calculation is as follows: (3)； Finally, in the GIA step, feature-weighted pooling is applied to... Region heatmaps normalized with Softmax Fusion to generate region-aware image features Simultaneously, by embedding the position of the CLIP image encoder... Heatmap with Softmax normalization Combined, location encoding weighted pooling is performed to generate location features that assign higher weights to the lesion region. ; These location features are then incorporated to obtain a final image representation in which implicit lesion location information is encoded; the detailed procedure is defined as follows: (4)； in , express Number of spatial locations in the query vector; With keys derived from CLIP image PatchTokens Sum Together, matrices are transformed through learnable linear transformations. Projected onto a unified low-dimensional feature space; By cross-attention processing, a global context representation is generated : (5)； Subsequently, pass Upsampling, and using learnable weighting coefficients With CLSToken Fusion enhances the global feature representation while assigning greater importance to lesion-related regions; the fusion formula is as follows: (6)； Finally, the merged PatchTokens and CLSTokens are concatenated and input into the CLIP image encoder. To extract classification features .

6. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 2, characterized in that: In dual-path score fusion, when calculating classification scores based on cosine similarity, the model tends to favor the prediction of the majority class. A dual-path score fusion strategy, DPLF, is proposed, combining learnable class centers initialized with textual semantics and dynamic gating weighting. In addition to image-label pairs, a set of textual descriptions is defined. Each of them Representation and Category Relevant natural language descriptions; for each training sample The associated category description is denoted as Each text description By CLIP text encoder Processing to extract its high-level semantic features, denoted as The category-level semantic feature set is collected into a matrix: (7)； This matrix is ​​used to initialize a learnable class center matrix. , of which OK Corresponding to category Learning center It is the output dimension of the encoder; given by the CLIP image encoder from Extracted image features The classification score is calculated from two complementary branches, each capturing different semantic cues; the first branch is based on... With the center of learnable categories Cosine similarity calculation between categories The score is defined as: (8)； In parallel, a second branch computes the similarity between the text features where the learned cues for the class enhance the textual description, trainable text prefixes: (9)； A learnable gating parameter is introduced to dynamically balance the contributions of the two branches ; class The fusion score is computed as follows: (10)。 7. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 2, characterized in that: In the disease perception fine-tuning of the visual-language model, a category-aware cue module P-Tuning was introduced for the text encoder; and a domain adaptation module Adapter was integrated for the image encoder and embedded into each Transformer layer to achieve targeted domain adaptation with minimal parameter overhead.

8. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 2, characterized in that: To enhance the model's understanding of medical terminology, a category-aware prompting module is introduced. This module learns a set of... 1 context token, and append them to each category name. Previously, the goal was to build medical perception prompts: (11)； Each of them ( ) is a trainable vector with the same dimension as the word embeddings used in CLIP, i.e., 512; the number of context tokens It is a hyperparameter that follows the CoOp setting. ;Suggest the enhanced sentence It was then input into the frozen text encoder. In order to obtain category-specific text features with learnable context: .

9. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 2, characterized in that: The domain adaptation module, a standard Transformer encoder block, consists of two main sub-layers: a multi-head self-attention (MHSA) layer and a feedforward network (FFN) layer; the self-attention formula is as follows: (12)； wherein denotes the the output Tokens of the layer MHSA; these Tokens are then passed through a FFN comprising a LayerNorm followed by an MLP to produce the final output of the block: (13)； wherein represents the output of the encoder block. The Adapter method is used, integrating the Adapter module into the FFN through residual connections; given input features The Adapter module generates adaptive features through its transformation branch. The formula is as follows: (14)； The resulting adaptive representation Subsequently, residual connections and processed features were used. The fusion produces the final output: (15)； where the scalar Controlling the contribution of adaptive features.

10. The long-tail medical image classification method based on lesion-aware dual calibration network according to claim 2, characterized in that: To effectively learn image feature representation and two-path fractional probability prediction for medical imaging long-tailed distribution, we adopt loss function as constraints.

Images