Medical image analysis method and system based on visual language model

Abstract

The invention discloses a medical image analysis method and system based on a visual language model, and belongs to the technical field of medical image intelligent diagnosis, and the system comprises an image preprocessing unit which carries out the down-sampling of an original retina OCT image to 256 * 256 and carries out the normalization of the original retina OCT image; the feature encoding unit comprises an image encoder based on RET Found in combination with LoRA optimization and a text encoder based on BioClinicalBERT; the class balance comparison learning unit is used for adjusting loss through class balance coefficients so as to relieve the class imbalance problem; the uncertainty estimation unit is used for calculating confidence quality and uncertainty scores based on Dirichlet distribution, and determining a threshold value in combination with an improved Youden index; and the model training unit adopts a total loss function of class balance loss and uncertainty loss, outputs a diagnosis result and an uncertainty score through transfer learning, and further comprises an image input module, a result display module and a data storage module. Rare disease classification performance and reliability are improved, training efficiency is improved through LoRA optimization, and an accurate and reliable scheme is provided for detection of the rare retina diseases.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical imaging diagnosis technology, specifically relating to a medical image analysis method and system based on a visual language model. Background Technology

[0002] Vitreoretinal lymphoma (VRL) is a rare non-Hodgkin's lymphoma that typically occurs outside the lymph nodes. It is characterized by high invasiveness and a poor prognosis, with a 5-year survival rate of approximately 61%. Over 95% of VRL cases are diagnosed as diffuse large B-cell lymphoma. Notably, 56%–96% of VRL patients eventually develop central nervous system (CNS) involvement, requiring more precise and personalized treatment and management. Therefore, early diagnosis and timely intervention are crucial for preventing or delaying CNS involvement, protecting vision, and improving survival rates in VRL patients. However, due to the insidious nature of VRL and the diversity of its clinical presentations, early diagnosis is challenging. It is frequently misdiagnosed as other ophthalmic diseases, such as uveitis or other inflammatory conditions, leading to delays in optimal treatment. Currently, vitreous cytology and pathology are the gold standard for diagnosing VRL. However, its diagnostic accuracy is limited by specimen quality, cell integrity, sectioning techniques, and the pathologist's experience. Furthermore, it is an invasive procedure, time-consuming, and suffers from poor patient compliance, resulting in a low positive rate. In recent years, Optical Coherence Tomography (OCT), due to its high resolution and non-invasive nature, has become an important tool for assisting in the diagnosis and management of VRL, reducing reliance on vitreous biopsy. It can clearly display detailed signs of lymphoma infiltration, such as nodular hyperreflective lesions within or beneath the retinal pigment epithelium. Therefore, OCT imaging features have significant diagnostic value in VRL diagnosis.

[0003] Given the rarity of VRL in clinical practice (incidence less than 0.5%), many ophthalmologists lack sufficient experience in identifying VRL-related features on OCT images, and some have never even encountered a VRL case in their careers, further increasing the risk of misdiagnosis or delayed diagnosis. Therefore, developing an automated VRL detection model based on OCT images can provide clues for invasive biopsies in suspected cases and reduce diagnostic delays in clinical practice. Deep learning (DL) has shown great promise in OCT image-based retinal disease detection. However, existing artificial intelligence (AI) models mainly target common retinal diseases, and their generalization ability is limited due to class imbalance during training caused by the scarcity of rare diseases such as VRL. In recent years, Vision-Language Models (VLMs) have enhanced the interpretability and robustness of medical image analysis by integrating visual and textual information. This method enhances feature learning by utilizing large-scale visual-language pre-trained knowledge and improves performance on classification tasks through contrastive learning. However, traditional VLMs rely on global feature alignment, ignoring subtle pathological changes, and therefore often struggle to identify rare diseases. Furthermore, their lack of an inherent uncertainty quantification mechanism makes them less reliable in high-risk medical scenarios.

[0004] To overcome these limitations, it is necessary to propose a medical image analysis method and system based on a visual language model to improve the accuracy and reliability of rare disease (especially VRL) detection. Summary of the Invention

[0005] The purpose of this invention is to provide a medical image analysis method and system based on a visual language model, which solves the problems of data class imbalance, modal information fragmentation, and insufficient diagnostic reliability in existing VRL-assisted diagnostic technologies.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution: A medical image analysis method and system based on a visual language model, comprising: The image preprocessing unit is used to downsample the original retinal OCT image to 256×256 pixels through bilinear interpolation and perform normalization processing to map the pixel values ​​to the range of [0,1]. The feature encoding unit includes an image encoder and a text encoder. The image encoder is based on the pre-trained RETFound model, introduces the LoRA optimization strategy, freezes the query and value projection layer weight matrices of the multi-head attention module in RETFound, and adjusts the intermediate layer weights through low-rank matrix factorization. The text encoder uses the BioClinicalBERT model pre-trained on the MIMIC III medical text dataset to extract the feature vector of the input prompt text. The class-balanced contrastive learning unit adjusts the contrastive loss function based on the class balance coefficient, which is calculated based on the total amount of data in the training set and the amount of data in the corresponding class, in order to alleviate the problem of data class imbalance. The uncertainty estimation unit is based on the Dirichlet distribution. It obtains the evidence features of the image-text comparison direction through the Softplus activation function, parameterizes the evidence features into Dirichlet distribution parameters to calculate the confidence quality and uncertainty score, and determines the optimal decision threshold through an improved Youden exponent variant. The model training unit uses a total loss function that includes class balance loss and uncertainty loss to train the model through transfer learning. After training, it outputs diagnostic results and corresponding uncertainty scores.

[0007] Furthermore, in the LoRA optimization strategy of the image encoder, the rank r of the low-rank matrix factorization is 8, and the weights are updated in the following way: Construct a linear mapping layer and ,in ; Through formula Update the weight matrix, where These are the pre-trained weights of RETFound; Through formula Calculate the output features, where This is the input feature vector.

[0008] Furthermore, the LoRA optimization strategy in the multi-head attention is as follows: Where T represents the matrix transpose operation. , and For the frozen projection layer in RETFound, , , and These are trainable parameters.

[0009] Furthermore, the feature extraction process of the text encoder includes: The input prompt text features are segmented, truncated to 512 tokens, and then padded. Extract the hidden states of the first, second, and last layers of the BioClinicalBERT model and calculate the average value as the text feature; The text features are mapped to a 256-dimensional space through a projection layer and then L2 normalized to obtain the text embedding vector.

[0010] Furthermore, the class balance coefficient of the class-balanced contrastive learning unit Calculated using the following formula: Where total represents the total amount of data in the training set. Let be the amount of data in the i-th category.

[0011] Furthermore, in the uncertainty estimation unit: Evidence feature E is obtained through the following formula:

[0012] Where I2T is the image-to-text comparison direction, T2I is the text-to-image comparison direction, I is the image embedding vector, T is the text embedding vector, and Tr is the transpose operation; The Dirichlet distribution parameter α is calculated using the Dirichlet distribution: Where k is the category index. and Let represent the evidence for image-text similarity of the k-th class of samples and the Dirichlet distribution parameters for text-image similarity of the k-th class, respectively. and Let represent the evidence of text-image similarity for the k-th class of samples and the Dirichlet distribution parameters of the text-image similarity for the k-th class, respectively. The probability density of the Dirichlet distribution is defined as follows: in, It is a dimensionless distribution parameter. Represents K-dimensional multivariate beta parameters. Let K be the unit simplex, defined as: The confidence quality b and the uncertainty score u are calculated using the following formulas: in, Representing the Dirichlet intensity of image-to-text and text-to-image respectively, where N is the total number of categories; the optimal threshold θ Determined by maximizing the improved Youden exponent variant: Among them, TPR is the true positive rate and FPR is the false positive rate.

[0013] Furthermore, the total loss function By class balance loss and uncertainty loss Composition, defined as follows: The class-balance loss is calculated using the following formula: Uncertainty loss include:

[0014] in, It is an improved cross-entropy loss. For KL divergence loss, It is a balance factor and satisfies and Each sample that is used to ensure the highest similarity between the image and the text generates more evidence when it is a correct prediction, and generates less evidence when it is an incorrect prediction. Its initial value is 0 and it gradually increases to 1 as the number of training rounds increases. It is the digamma function. It is the adjustment parameter of the Dirichlet distribution. It is the gamma function.

[0015] Furthermore, it also includes: Image input module for receiving raw retinal OCT images; The image preprocessing unit performs standardization processing on the input image; Feature encoding unit, extracts image features and text features; The system consists of a balanced contrastive learning unit and an uncertainty estimation unit, which output diagnostic results and uncertainty scores. The results display module triggers a secondary diagnostic prompt when the uncertainty score is higher than the threshold θ; when the uncertainty score is lower than or equal to the threshold θ, the diagnostic result is output directly.

[0016] Furthermore, it also includes a data storage module for storing input images, processed feature data, diagnostic results, and uncertainty scores, in DICOM or JSON format. Beneficial effects

[0017] 1. This invention proposes for the first time a visual language foundation model for uncertainty perception, UA-VLFM, which enables automatic classification of VRL and five other retinal diseases based on retinal OCT images. This provides an accurate and reliable method for the automatic detection of rare retinal diseases based on OCT images and contributes to the clinical decision-making process in the real world.

[0018] 2. A contrastive learning method based on class balance is proposed for the first time, which improves the performance of visual language basic models in rare disease classification; 3. For the first time, an uncertainty estimation strategy is introduced into the training of a visual language model to improve the model's classification ability of OCT retinal images, and a threshold strategy is combined to achieve reliability assessment of the predicted output.

[0019] 4. Integrate the LoRA optimization strategy into the CLIP framework to improve training speed and save memory consumption. Attached Figure Description

[0020] Figure 1 A schematic diagram of two-dimensional OCT images used for retinal disease classification according to the present invention, wherein: Figure 1 (a) Schematic diagram of proliferative diabetic retinopathy (PDR); Figure 1 (b) Schematic diagram of retinitis pigmentosa (RP); Figure 1 (c) Schematic diagram of acute Vogt-Koyanagi-Harada syndrome (acute VKH); Figure 1 (d) Schematic diagram of age-related macular degeneration (AMD); Figure 1 (e) Schematic diagram of myopic choroidal neovascularization (mCNV); Figure 1 (f) Schematic diagram of vitreoretinal lymphoma (VRL); Figure 2 This is a flowchart illustrating the application of the UA-VLFM of the present invention in clinical practice; Figure 3 This is a schematic diagram of the RETFound pre-trained model framework of the present invention, wherein: Figure 3 (a) A schematic diagram of constructing RETFound through self-supervised learning; Figure 3 (b) A schematic diagram of how RETFound adapts to downstream tasks through supervised learning with internal and external evaluations; Figure 4 This is a schematic diagram outlining the large visual language model based on class balance and uncertainty estimation for retinal disease classification according to the present invention, wherein: Figure 4 (a) A schematic diagram of model training; Figure 4 (b) Schematic diagram of inference in clinical medicine; Figure 5 This is a schematic diagram of the low-rank adaptive learning (LoRA) strategy of the present invention; Figure 6 The diagram shows four OCT image samples detected using the standard AI model (RETFound) and the UA-VLFM of this invention, wherein: Figure 6 (a) and (b) show two sample illustrations where both RETFound and UA-VLFM correctly predicted the results; Figure 6 (c) and (d) are schematic diagrams of two samples where RETFound and UA-VLFM made incorrect predictions. Detailed Implementation

[0021] like Figure 1-6 As shown, this invention combines the Contrastive Language-Image Pre-training (CLIP) model and uncertainty estimation theory to propose an Uncertainty-aware Vision-Language Foundation Model (UA-VLFM) based on OCT images for classification. This method mainly includes three steps: image preprocessing, network structure construction, and model training and testing.

[0022] 1) Image preprocessing: In order to prevent GPU memory overflow, the original 1024×1177 two-dimensional retinal OCT image was downsampled to 256×256 using bilinear interpolation and then normalized to normalize the pixel values ​​to [0,1].

[0023] 2) Network structure construction: Considering that clinical data includes two major categories of image data and text data, this invention improves upon the CLIP framework to enhance the model's understanding of image-text correlation and improve the model's feature representation capabilities; First, in order to improve training speed and save memory consumption, the image encoder introduces the LoRA optimization strategy on the basis of RETFound; Secondly, in order to alleviate the class imbalance problem, this invention designs a loss function based on class-balanced sampling on the basis of the traditional contrastive loss function, which significantly improves the model's ability to identify small samples. In addition, to reduce the risk of misdiagnosis and increase the level of trust that medical practitioners have in the model, an uncertainty estimation strategy is introduced during the network optimization process.

[0024] 3) Model Training and Testing: Transfer learning is used to employ RETFound, pre-trained on large-scale OCT images, as the backbone network of the image encoder to extract image features. BioClinicalBERT, pre-trained on the MIMIC III medical text dataset, is used as the text encoder to extract text features, thereby improving model classification accuracy and accelerating network convergence. The designed and constructed visual-language foundational model UA-VLFM is trained using data from the training set. After training, the model's performance is tested using a test set to evaluate the invention's effectiveness.

[0025] Figure 1 These are retinal OCT images used for classification and identification in this invention, including VRL and five types of retinal diseases, as shown below. Figure 1 As shown in (a) to (f). To reduce the computational cost of the learning task, all 2D OCT images were downsampled to 256×256 using bilinear interpolation and then normalized to improve discriminability.

[0026] Design and construction of a visual language foundational model for uncertainty perception: 1) Overall structure of the model: Figure 2 This is a flowchart illustrating the clinical application of the Uncertainty Perception Model for Retinal Disease Classification (UA-VLFM), designed and constructed in this invention. Figure 2 As shown, the UA-VLFM of this invention can not only obtain diagnostic results for retinal diseases, but also calculate the corresponding uncertainty score to evaluate the reliability of the prediction results. Cases with high uncertainty can be referred to human image interpreters, achieving effective interaction between doctors and AI. This interaction can provide guidance to doctors, helping them decide whether referral or other examinations are necessary, thereby effectively avoiding misdiagnosis and missed diagnosis.

[0027] The Visual Language Foundation Model (UA-VLFM) for uncertainty perception in retinal disease classification designed and constructed in this invention is an improvement based on the RETFound visual encoder and CLIP model. The original RETFound network (see...) Figure 3 () is a basic model for retinal images, the core of which is a mask autoencoder with a specific configuration. The training objective is to reconstruct the original retinal image from a highly masked image. Specifically: The encoder first divides the image data into 16×16 pixel image patches, and transforms them into 1024-dimensional embedding vectors through linear projection; then, it extracts high-level semantic features through 24 Transformer blocks (including multi-head self-attention (MHSA) and multilayer perceptron (MLP)). To reduce computational costs, the model employs a targeted masking strategy: masking 75% of the CFP image and 85% of the OCT image, inputting only the unmasked regions into the encoder. The decoder concatenates the encoder output with dummy tokens of the masked regions, reconstructing the original image patches through 8 layers of Transformer blocks (a lightweight design focused on detail restoration). The mean squared error (MSE) between the output and the original image is used as the loss function. Downstream classification tasks require not only adding a classifier after the pre-trained encoder but also fine-tuning the model using data from relevant tasks to classify retinal diseases.

[0028] The framework of this invention is as follows Figure 4 As shown, the invention utilizes an image encoder and a text encoder to extract image and text data features respectively. Contrastive learning is used to calculate the distance between positive and negative sample pairs, bringing paired image-text pairs closer together and widening the spatial distance between unpaired image-text pairs. The aim is to unify the representation spaces of visual and linguistic modalities through contrastive learning, enabling the model to perform joint reasoning across image and linguistic domains. Furthermore, an uncertainty estimation strategy is introduced during the training phase to enhance the model's ability to learn complex pathological features in OCT images. In addition, considering the problem of data imbalance, a class-balanced contrastive learning approach is proposed. During the training phase, the model is optimized using both a class-balanced contrastive learning loss function and an uncertainty-aware loss function, improving diagnostic accuracy. During the inference phase, an uncertainty estimation classifier outputs the model's prediction results and corresponding uncertainty scores. Results exceeding a threshold are referred to doctors for secondary diagnosis.

[0029] Text Encoder Considering the highly specialized and complex nature of ophthalmic OCT image descriptions, this invention employs a BioClinicalBERT model pre-trained on the MIMIC III medical text dataset as the model's text encoder to extract text features, ensuring accurate understanding of ophthalmic-specific terminology and semantic representations. Specifically: for the input prompt text "An OCT image of {CLS}" ({CLS} represents the specific disease name), the BioClinicalBERT tokenizer first performs word segmentation, truncation (maximum length 77), and padding to generate an input ID and attention mask. This is then input into the pre-trained BioClinicalBERT model to obtain the hidden states of each layer. The hidden states of layers 1, 2, and 3 are stacked, and after dimensionality transformation, the average along the sequence length dimension and layer number dimension is calculated to obtain 768-dimensional text features. Finally, the features are mapped to a 256-dimensional space through a projection layer and L2 normalized to obtain the final text embedding vector, which is used for subsequent vision-language contrastive learning.

[0030] Image encoder In this invention, we use a pre-trained RETFound encoder as an image encoder to extract high-level features from OCT images. First, the OCT image is preprocessed by resizing it to 256×256 pixels and normalizing it using ImageNet standard mean and variance. Then, the processed OCT image is input into the RETFound-based encoder, and a LoRA optimization strategy (e.g., ...) is introduced into the model optimization. Figure 5 As shown, based on the frozen RETFound pre-trained weights, the model is efficiently fine-tuned through low-rank matrix factorization (r=8) to capture structural features and pathological information in OCT images, outputting a 512-dimensional original feature vector. Subsequently, this feature is mapped to a 256-dimensional unified embedding space through a projection layer and L2 normalized to distribute the features on a unit hypersphere, ultimately generating a visual embedding vector suitable for vision-language contrastive learning. Specifically, the LoRA optimization strategy freezes the projection layer weight matrices of the query (Q) and value (V) in the multi-head attention module of the original Transformer layer (e.g., ...). Figure 5 (As shown), and a bypass is added to complete the low-rank approximation. The bypass consists of two linear mapping layers. and ,in Assume the input token sequence, output token sequence, and pre-trained weights corresponding to the multi-head attention module are as follows: , and The LoRA layer update steps are as follows: (1); (2); Based on the above analysis, the LoRA optimization strategy in multi-head attention can be described as follows: (3); (4); (5); (6); Where T represents the matrix transpose operation. , and For the frozen projection layer in RETFound, , , and These are trainable parameters.

[0031] Compared to fully fine-tuned training strategies, the LoRA optimization strategy can keep pre-trained weights frozen and optimize by automatically adjusting the weights of intermediate layers of the model. This reduces memory consumption and shortens training time without sacrificing model accuracy, making it easier for the model to be deployed clinically.

[0032] 4) Contrastive learning based on class balance To align image and text representations in a shared semantic space, we use linear projection heads from image and text encoders to map image and text feature vectors from independent modal representations into a unified embedding space. This invention employs cosine similarity to measure the similarity between image and text embeddings, denoted as ITrT, where Tr is the transpose operation. The training objective is to maximize the similarity between paired images and text while minimizing the similarity between unpaired samples. Furthermore, considering the data class imbalance problem caused by the scarcity of rare fundus diseases, this invention proposes an improved contrastive loss based on class balance to guide model optimization, building upon traditional contrastive loss. The class balance coefficient is calculated based on the total amount of training data and the corresponding class data; detailed calculation is shown in formula (20).

[0033] 5) Uncertainty estimation optimization strategy For images with blurred or complex pathological features, models are prone to outputting unreliable results. Existing visual language models typically only output a single classification prediction in downstream classification tasks, failing to assess the reliability of the model's output. This invention introduces an uncertainty-aware loss function based on the Dirichlet distribution during the training phase, ensuring that more evidence is generated for correct predictions of each sample with the highest similarity between the image and text, thus enhancing the learning ability for complex pathological features in OCT images. During the inference phase, the model can output the prediction result and the corresponding uncertainty score.

[0034] Specifically, the following steps are included: a) Using the Softplus activation function to obtain evidence features for different categories of retinal diseases (7); (8); Where I2T represents the image-to-text contrast direction, and T2I represents the text-to-image contrast direction.

[0035] b) Parameterize the evidence feature E as a Dirichlet distribution: ; ; in, and Let represent the evidence for the image-text similarity of the k-th class of samples and the Dirichlet distribution parameters of the image-text similarity of the k-th class, respectively. and Let represent the evidence for the text-image similarity of the k-th class of samples and the Dirichlet distribution parameters for the text-image similarity of the k-th class, respectively. The Dirichlet distribution is an exponential family distribution, and its probability density is defined as follows: (11); among which, It is a dimensionless distribution parameter. Represents K-dimensional multivariate beta parameters. Let K be the unit simplex, defined as: (12); c) The confidence quality and the corresponding uncertainty score are calculated as follows: (13); (14); (15); in, , These represent the Dirichlet intensity for image-to-text and text-to-image, respectively.

[0036] d) Threshold calculation The True Positive Rate (TPR) and False Positive Rate (FPR) were calculated based on the validation set using an improved variant of the Youden index, ℓ(θ). The optimal threshold θ is determined by maximizing this function. Once the threshold θ is calculated, during the inference phase, results with uncertain scores greater than the threshold will be referred to a specialized ophthalmologist for secondary diagnosis.

[0037] ℓ(θ) = 2×TPR(θ)−FPR(θ) (16); (17); 6) Loss Function In this invention, to align image and text representations in a shared semantic space, model optimization is performed by extracting the similarity of image and text features using an image encoder and a text encoder. Assume... Represents an image encoder and its corresponding projection head , , To represent a text encoder and its corresponding projection head For the input OCT image High-level image feature representations are obtained using an image encoder: , It is the dimension of the image feature space. For input text prompts... Text features are extracted using a text encoder and are represented as follows: ,in, This is the dimension of the text feature space. Then, using an image projection head and a text projection head, the independent modal representations are mapped to a unified embedding space, resulting in... and Cosine similarity is used to measure the alignment between the image and the text embedding, denoted as... Where Tr is the transpose operation. Our goal is to maximize the similarity between paired images and text while minimizing the similarity between unpaired samples. To address the class imbalance problem, a class-balanced loss is designed in addition to the traditional contrastive loss to improve the model's ability to identify rare disease classes (such as VRL). Furthermore, to improve the reliability of VRL diagnosis, an uncertainty estimation optimization strategy based on Dirichlet distribution is integrated into the pre-trained CLIP architecture to obtain prediction results with corresponding uncertainty scores. Therefore, the total loss function consists of the class-balanced loss... and uncertainty loss It consists of two parts, defined as follows: (18); The loss based on class equilibrium is defined as follows: The balance coefficient for class i is calculated using the following formula: (20); This invention correlates the Dirichlet distribution with the feature similarity between images and text descriptions, thereby obtaining the confidence quality and corresponding overall uncertainty score of the image-text description similarity for each sample based on evidence collected from the feature similarity matrix. Therefore, we can calculate the minimum values ​​for image-to-text and text-to-image similarities. Minimum Dirichlet distribution parameters (multinomial opinion) ),in and Let represent the probability of the i-th class. Based on the above analysis, this invention improves the traditional CrossEntropy (CE) loss function to obtain a loss function for uncertainty estimation. As shown below: (twenty one); (twenty two); (twenty three); ; (25); in, and Correct predictions for each sample, used to ensure the highest similarity between images and text, generate more evidence, while incorrect predictions generate less evidence. It is a balancing factor, with an initial value of 0 that gradually increases to 1 as the number of training rounds increases. It is the digamma function. It is the adjustment parameter of the Dirichlet distribution. It is the gamma function.

[0038] Experiments were conducted on the contents shown in this invention: 1) Database This study used Triton DRI OCT (Topcon, Tokyo, Japan), 3D-OCT-2000 (Topcon, Tokyo, Japan), and Spectralis (Heidelber Engineering, Heidelberg, Germany) to acquire OCT images, including six diseases: proliferative diabetic retinopathy (PDR), retinitis pigmentosa (RP), acute Vogt-Koyanagi-Harada syndrome (acute VKH), age-related macular degeneration (AMD), myopic choroidal neovascularization (mCNV), and vitreoretinal lymphoma (VRL). The first five relatively common diseases were from JSIEC, while the OCT images for VRL came from the Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong (JSIEC), Zhongshan Ophthalmic Center of Sun Yat-sen University in Guangzhou, and Xiangya No. 2 Hospital of Central South University. The OCT images were categorized according to the corresponding classification criteria. Two senior ophthalmologists independently labeled the images. If there was disagreement, a final confirmation was made by an experienced ophthalmologist. Finally, this invention incorporated 35,029 OCT images with consistent labels from the three ophthalmologists and divided the dataset into training, validation, and test sets in an 8:1:1 ratio for training the model and evaluating the performance of this invention. To reduce the computational cost of the learning task, all two-dimensional OCT images were downsampled to 256×256 using bilinear interpolation and then normalized to improve discriminancy.

[0039] 2) Data augmentation To prevent overfitting and improve the model's generalization ability, online data augmentation methods were employed, including random cropping, random horizontal flipping, and random cropping.

[0040] 3) Experimental platform and parameters All experiments were conducted on the PyTorch platform using an NVIDIA A800-SXM 4GPU with 80GB of memory. The model was trained by minimizing the loss function using the backpropagation algorithm (see Equation (16)), with a batch size of 128, 100 training epochs, and the Adam optimizer used to minimize the loss function. The learning rate was set to 5e-3.

[0041] 4) Evaluation Indicators To quantitatively evaluate the performance of this invention, four commonly used classification metrics are used: accuracy (Accuracy, Acc), sensitivity (Sen), precision (Precision, Pre), and F1 score (F1), which are defined as follows: (26); (27); (28); (29); TP, FP, TN, and FN represent true positive, false positive, true negative, and false negative, respectively.

[0042] Verification and Results: This invention evaluates and compares Swin-Transform (Swin-Trans), RETFound, convolutional neural network models with uncertain policies, and base models (UIOS and FMUE) with the UA-VLFM proposed in this invention on a test set. This invention also removes samples with high uncertainty scores from the UA-VLFM inference to calculate the final performance metrics. Experimental results are shown in Tables 1 to 4. In Tables 1 to 4, "RETFound" represents the method published on September 13, 2023, in the journal "Nature" (Digital Object Identifier, DOI 10.1038 / s41586-023-06555-x); "UIOS" represents the method published on October 24, 2023, in the journal "Nature Communications" (Digital Object Identifier, DOI 10.1038 / s41467-023-42444-7); "UIOS_T" represents the performance after filtering out samples with high uncertainty scores based on UIOS; and "FMUE" represents the method published on October 18, 2025, in the journal "Cell Reports Medicine" (Digital Object Identifier, DOI 10.1038 / s41586-023-06555-x). The method (DOI) is 10.1016 / j.xcrm.2024.101876. "FMUE_T" represents the performance after filtering out samples with high uncertainty scores based on FMUE, "UA-VLFM" represents the method of the present invention, and "UA-VLFM_T" represents the performance of the method of the present invention after filtering out samples with high uncertainty scores.

[0043] Table 1 shows that the classification accuracy of the Swin-Trans model is 0.8750, that of RETFound is 0.8637, that of UIOS is 0.9655, and that of FMUE is 0.9671. The accuracy of this invention is 0.9751, representing an overall improvement of 11.44%, 12.90%, 0.99%, and 0.83%, respectively, with p-values ​​all less than 0.05, indicating statistical significance. Compared to the suboptimal method (UIOS), the sensitivity, precision, and F1 score of the proposed UA-VLFM increased from 0.8532, 0.7848, and 0.8176 to 0.9174, 0.9259, and 0.9217, respectively. To improve the reliability of the model, this invention obtains the uncertainty score for each OCT image and uses a threshold strategy to remove samples with high uncertainty, allowing clinicians to perform secondary review. In UA-VLFM, 3.46% of the samples had uncertainty scores higher than the threshold, lower than FMUE (5.13%, Table 1) and UIOS (5.91%, Table 1). Furthermore, after filtering out these samples with high uncertainty scores, the performance of UA-VLFM (Tables 1 to 4) was further improved. For example, the overall accuracy, sensitivity, precision, and F1 score increased from 0.9751, 0.9671, 0.9680, and 0.9684 to 0.9890, 0.9814, 0.9859, and 0.9839, respectively, outperforming the thresholded FMUE and UIOS. Four samples of OCT images detected using the standard AI model (RETFound) and the UA-VLFM of this invention are shown below. Figure 6 As shown. From Figure 6 It can be seen that RETFound only generates the final result with the highest probability, without providing any information to illustrate the reliability of the prediction. In contrast, UA-VLFM provides the final prediction result and the corresponding uncertainty score, reflecting the reliability of the prediction. For images with typical and distinguishable features ( Figure 6 In (a) and (b)), both RETFound and UA-VLFM can produce correct predictions, and this invention can assign an uncertainty score below a threshold to indicate the reliability of the prediction results. For challenging and complex samples ( Figure 6 (c) and (d) Although both RETFound and UA-VLFM made incorrect predictions, the present invention can provide a high uncertainty score, effectively marking these results as potentially unreliable and recommending a second examination to an ophthalmologist.

[0044] In summary, the uncertainty-perceived visual language foundation model UA-VLFM proposed in this invention, applicable to the classification of retinal diseases, ensures the accuracy and reliability of the classification and identification of rare retinal diseases.

[0045] Thus, the uncertainty-perceived visual language model (UA-VLFM) for retinal disease classification based on OCT images, as described in this invention, has been implemented and validated. Its performance in experiments outperforms the Swin-Trans, RETFound, UIOS, and FMUE methods. This invention can make better judgments on OCT images. Furthermore, the class-balanced contrastive learning optimization strategy designed in this invention can be extended to any VLM, enabling the model to better identify imbalanced data. In addition, the uncertainty estimation method used in this invention can be applied to any other VLM classification network, improving the reliability of the model's prediction results and making it more consistent with actual clinical applications. This invention combines image preprocessing, deep neural network model construction and training, and testing, providing a reliable method for automatic detection of retinal diseases based on OCT images and contributing to real-world clinical decision-making processes.

[0046]

[0047]

[0048]

[0049]

[0050] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A method and system for medical image analysis based on visual language model, characterized in that, include: The image preprocessing unit is used to downsample the original retinal OCT image to 256×256 pixels through bilinear interpolation and perform normalization processing to map the pixel values ​​to the range of [0,1]. The feature encoding unit includes an image encoder and a text encoder. The image encoder is based on the pre-trained RETFound model, introduces the LoRA optimization strategy, freezes the query and value projection layer weight matrices of the multi-head attention module in RETFound, and adjusts the intermediate layer weights through low-rank matrix factorization. The text encoder uses the BioClinicalBERT model pre-trained on the MIMIC III medical text dataset to extract the feature vector of the input prompt text. The class-balanced contrastive learning unit adjusts the contrastive loss function based on the class balance coefficient, which is calculated based on the total amount of data in the training set and the amount of data in the corresponding class, in order to alleviate the problem of data class imbalance. The uncertainty estimation unit is based on the Dirichlet distribution. It obtains the evidence features of the image-text and text-image contrast directions through the Softplus activation function, parameterizes the evidence features into Dirichlet distribution parameters to calculate the confidence quality and uncertainty score, and determines the optimal decision threshold through an improved Youden exponent variant. The model training unit uses a total loss function that includes class balance loss and uncertainty loss to train the model through transfer learning. After training, it outputs diagnostic results and corresponding uncertainty scores.

2. The method and system of claim 1, wherein, In the LoRA optimization strategy of the image encoder, the rank r of the low-rank matrix factorization is 8, and the weights are updated in the following way: constructing a linear mapping layer and wherein ; By the formula updating the weight matrix, wherein are the pre-trained weights for RETFound; The output feature is calculated by the formula where is the input feature vector.

3. The method and system of claim 2, wherein, The LoRA optimization strategy in multi-head attention is as follows: where T denotes the matrix transpose operation, , and are the frozen projection layers in RETFound, , , and are trainable parameters.

4. The method and system of claim 3, wherein, The feature extraction process of the text encoder includes: segmenting the input prompt text features into words, truncating them to 512 tokens and padding them; extracting the hidden states of the first, second and last layers of the BioClinicalBERT model and calculating the average value as the text features; The text features are mapped to a 256-dimensional space through a projection layer and then L2 normalized to obtain the text embedding vector.

5. The method and system according to claim 4, characterized in that, The category balance coefficient of the class balance contrast learning unit The calculation is performed by the following formula: ; wherein total is the total data amount of the training set, is the data amount of the i-th category.

6. The method and system of claim 5, wherein, In the uncertainty estimation unit: The evidence feature E is obtained by the following formula: wherein, I2T is the contrast direction from image to text, T2I is the contrast direction from text to image, I is the image embedding vector, T is the text embedding vector, and Tr is the transpose operation; the Dirichlet distribution parameter a is calculated by Dirichlet distribution: where k is the class index, and and and and 7. The method and system according to claim 6, characterized in that, The Dirichlet distribution has a probability density defined as follows: where, is a dimensionless distribution parameter, denotes a K-dimensional beta parameter, is a K-dimensional unit simplex, defined as: The confidence mass b and the uncertainty fraction u are computed by the following equations: where, , denote the Dirichlet intensities for image-to-text and text-to-image, respectively, N is the total number of classes; the optimal threshold is determined by maximizing a modified Youden index variant: where TPR is the true positive rate and FPR is the false positive rate.

8. The method and system according to claim 7, characterized in that, The total loss function consists of a class-balancing loss and an uncertainty loss defined as follows: ; where the class-balanced loss is calculated by the following equation: the uncertainty loss includes: where, is the improved cross-entropy loss, is the KL divergence loss, is the balance factor and satisfies , and are respectively used to ensure that the correct prediction of each sample with the highest similarity between the image and the text produces more evidence, while the incorrect prediction produces less evidence, and their initial values are 0 and gradually increase to 1 with the increase of the training round number; is the digamma function, is the adjustment parameter of the Dirichlet distribution, is the gamma function.

9. The method and system of claim 8, wherein, Also includes: Image input module, used to receive raw retinal OCT images; The image preprocessing unit performs standardization processing on the input image; Feature encoding unit, extracts image features and text features; The system consists of a balanced contrastive learning unit and an uncertainty estimation unit, which output diagnostic results and uncertainty scores. The result display module triggers a secondary diagnosis prompt when the uncertainty score is above a threshold value and outputs the diagnosis result directly when the uncertainty score is below or equal to the threshold value .

10. The method and system of claim 9, wherein, It also includes a data storage module for storing input images, processed feature data, diagnostic results, and uncertainty scores, in DICOM or JSON format.

Images