Uncertainty-aware multimodal large language model ophthalmic reliable consultation method and system

Abstract

The application discloses a multimodal large language model eye care method and system with uncertainty perception. Firstly, a visual encoder with uncertainty perception is constructed. A balanced evidence deep learning and prototype-based contrast learning method is used to realize confidence calibration estimation and enhance the robust representation of minority classes. Then, the uncertainty quantized by the model is discretized and injected into the system prompt word to guide the model to generate more conservative responses. By combining the large language model with the domain ophthalmology knowledge base, a retrieval-enhanced multi-turn dialogue corpus is constructed to support the generated content with authoritative medical evidence, thereby reducing the factitious delusion. Finally, the visual features and language domains are aligned, and the parameters of the backbone network are efficiently fine-tuned. The application can enable the model to generate ophthalmic clinical inquiry replies with uncertainty perception and safety guidance, and has great potential as a reliable diagnostic assistant in actual ophthalmic applications.

Description

Technical Field

[0001] This invention relates to the field of clinical medicine, specifically to a reliable ophthalmic consultation method and system based on a multimodal large language model for uncertainty perception. Background Technology

[0002] Multimodal language models have shown great potential in clinical medicine, serving as tools for initial patient screening, clinical interpretation, and interactive consultations, with particularly broad application prospects in specialized settings such as ophthalmology. However, most existing medical multimodal language models rely on domain training and visual semantic alignment using fundus images and text pairs. While possessing some cross-modal understanding capabilities, they generally suffer from model illusions and output errors, severely limiting their clinical reliability. Simultaneously, high-quality case samples are often scarce in ophthalmology, requiring models to undergo domain fine-tuning on small-scale or class-imbalanced datasets, further exacerbating model learning instability. However, existing research and inventions have not yet effectively addressed these challenges. For example, patent CN120297420A proposes a method for constructing and training multi-task models based on fundus images. By fusing multi-layered visual features, the enhanced visual representation is directly mapped to a large language model for medical reasoning and text generation. However, this method assumes that visual features possess consistent and reliable semantic representation capabilities, ignoring the predictive uncertainty caused by factors such as blurred lesion boundaries, degraded image quality, similar disease manifestations, or insufficient training samples. Furthermore, it fails to consider the inherent risks of the model learning disease boundaries under small sample conditions. When visual evidence itself is ambiguous or unclear, the model still treats it as deterministic semantic input to the large language model, causing the loss of uncertain information during cross-modal interaction. This leads to safety issues such as model hallucination, overconfidence, and misinterpretation, significantly increasing the risk of clinical auxiliary diagnosis. Summary of the Invention

[0003] To address the problems existing in the prior art, the present invention aims to provide a credible ophthalmic consultation method and system based on a multimodal large language model guided by evidence theory and characterized by uncertainty perception. This method constructs a visual encoder that perceives uncertainty to model the confidence level of fundus images and introduces quantified uncertainty into the consultation prompt generation process. Simultaneously, it combines a domain knowledge base retrieval enhancement mechanism to impose factual constraints on multi-turn dialogues, thereby effectively suppressing model illusion and overconfidence issues, improving the accuracy, reliability, and security of fundus disease consultation results, and promoting the credible application of multimodal consultation models in clinical scenarios.

[0004] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: A multimodal large language model-based method for reliable ophthalmological consultation based on uncertainty perception includes the following steps: Step 1: Preprocess the fundus images containing fundus disease information to make them suitable for the size designed for the input model; Step 2: Input the preprocessed fundus image into the pre-trained visual encoder to extract visual features containing pathological information and diagnostic results. At the same time, model the uncertainty of the diagnostic results to obtain quantified uncertainty information. Step 3: The obtained visual features are further input into the cross-modal alignment module to obtain aligned visual embeddings that are easy for the language model to understand; Step 4: Based on the user's input query related to fundus diseases as the consultation text, perform data retrieval on the consultation text, that is, retrieve similar typical cases or pathological knowledge segments from the constructed vector database to obtain the retrieval text, and then convert the consultation text and the retrieval text into latent space vector embeddings together; the language model generates a reliable response based on the vector embeddings of the consultation text and the retrieval text, the alignment of visual embeddings, and the quantification of uncertainty information.

[0005] The fundus image is preprocessed by converting the original fundus image into the RGB standard color space, then cropping it from the center, and uniformly scaling it to a tensor format of appropriate size.

[0006] The visual encoder is a feature extraction network built on a high-performance convolutional structure. This network can represent lesions and structures at different scales in fundus images and has more stable feature generalization ability compared to general visual models in small-sample training scenarios. The pre-trained visual encoder is jointly trained using fundus images with category labels based on evidence modeling and prototype contrastive learning. A training method combining balanced evidence learning and prototype contrastive learning is adopted, wherein the balanced evidence learning loss function is:

[0007] in, The coefficient of the training penalty term guided by misclassification; The Kullback-Leibler divergence term, representing the posterior Dirichlet distribution and the uniform prior, is used to suppress the accumulation of false evidence. The class balancing weights based on the number of valid samples are defined as follows:

[0008] in, As the attenuation factor, For the first The number of training samples for the class; Furthermore, a prototype contrastive learning loss is introduced to enhance the expressive power of minority class features. Let the sample features be represented as... The category prototype is For the two enhancement methods of the original input image and Then the prototype contrastive learning loss is defined as:

[0009] Finally, the joint training loss function used for the pre-trained visual encoder is:

[0010] in, and This is the loss term balancing coefficient, used to coordinate the weight allocation between uncertainty modeling and minority class discrimination ability. This refers to the temperature parameter.

[0011] The uncertainty modeling is based on Dempster-Shafer theory and evidence theory. It constructs Dirichlet distribution parameters to characterize the predicted class probabilities and uncertainty. The quantified uncertainty information is obtained from the statistical distribution of the uncertainty represented by the Dirichlet distribution parameters on a certain internal / external validation set. Based on the quantile threshold of the distribution, it is mapped to the corresponding semantic label set, and the corresponding uncertainty semantics are injected into the system prompts in text form as quantified uncertainty information. The uncertainty modeling process includes: the network outputs a non-negative evidence vector. Construct the Dirichlet distribution parameters:

[0012] Among them, the predicted class probability With forecast uncertainty They are respectively:

[0013] in, This represents the total number of categories of fundus diseases.

[0014] The cross-modal alignment module is a pre-trained Q-Former network based on the Transformer architecture. This network introduces a multi-head self-attention mechanism and a cross-modal cross-attention mechanism to model and reweight the high-dimensional visual features output by the visual encoder, and maps them to generate visual word embeddings with the same dimension as the word vector space of the language model. This achieves accurate alignment of visual semantics to the language semantic space and provides a unified representation basis for subsequent multimodal reasoning and diagnostic text generation.

[0015] The language model is a language model fine-tuned using the LoRA method. The steps for fine-tuning the language model include: firstly, constructing a multi-turn dialogue dataset, and then using the LoRA method to fine-tune the language model. The construction of the multi-turn dialogue dataset refers to constructing a multi-turn dialogue dataset containing high-performance large-scale ophthalmology multi-turn consultation dialogue data and real ophthalmology outpatient dialogue text. The high-performance large-scale ophthalmology multi-turn consultation dialogue data is multi-turn dialogue data obtained based on the dialogue between two independent high-performance models. Its construction method is as follows: given a fundus image, model A simulates a patient asking a relevant question, and then another high-performance multimodal large-scale model B makes an appropriate and credible response based on the provided fundus image and the consultation text of model A. Then, through multiple iterations, a multi-turn dialogue sample is generated. The above process is repeated to finally obtain the high-performance large-scale ophthalmology multi-turn consultation dialogue dataset, which can be used for fine-tuning training of the language model. The LoRA method used to fine-tune the language model involves multi-turn dialogue learning based on the constructed dataset. The loss function for this process is:

[0016] In the formula, Indicates the first In turn-based dialogue, the model outputs a prediction of the first turn. Cross-entropy loss of each word in the answer text; Indicates the first i The total length of the response text in the round-robin consultation dialogue; Indicates the first Before answering the text in a turn-based dialogue Each word element; This indicates its predicted probability; This represents the semantic embedding of corresponding fundus image visual features; Indicates from the Text embedding of historical dialogues.

[0017] The constructed vector database is based on an open-source ophthalmic medical knowledge text library and typical cases of the gold standard for consultation. The pre-trained medical text embedding model BLIP is used to vectorize the text, and the constructed locally usable vector database can perform data retrieval functions, that is, to perform similarity retrieval based on the consultation text provided by the patient, in order to match a number of ophthalmic medical knowledge items and typical cases of the gold standard for consultation, thereby obtaining retrieval text containing medical evidence support.

[0018] The present invention provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned multimodal consultation method for fundus diseases based on uncertainty perception and retrieval enhancement.

[0019] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned multimodal consultation method for fundus diseases based on uncertainty perception and retrieval enhancement.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects: (1) By combining the training strategy of balanced evidence learning and prototype contrastive learning, the feature degradation problem caused by class imbalance and sample scarcity is effectively alleviated, so that the model can still maintain stable disease recognition ability with a small amount of labeled data.

[0021] (2) An enhanced retrieval mechanism is introduced to integrate external knowledge bases and typical cases into the consultation context in real time, which greatly reduces the risk of factual deviation or logical contradiction in the language model during continuous dialogue.

[0022] (3) The quantitative uncertainty output by the visual encoder is explicitly injected into the system prompt words in the form of semantic tags, so that the language model can perceive and respond to the degree of uncertainty of visual evidence, thereby generating a more cautious response that is more in line with clinical safety requirements.

[0023] In summary, this invention significantly improves the safety and credibility of the model by introducing evidence theory to guide uncertainty perception modeling and mechanisms within the ophthalmological multimodal consultation framework, combined with retrieval-enhanced generation technology. This is achieved while maintaining diagnostic performance. On one hand, the visual encoder based on evidence-based deep learning can explicitly quantify the uncertainty of prediction results, enabling the model to have "self-awareness" when facing out-of-distribution samples or ambiguous lesions, thus suppressing overconfidence and erroneous assertions at their source. On the other hand, quantified uncertainty guides the language model to proactively express diagnostic reservations and suggest further examinations or referrals in low-confidence scenarios, reducing potential clinical risks at the mechanism level. Simultaneously, by constructing a retrieval-enhanced multi-round consultation corpus that integrates an authoritative ophthalmological knowledge base, the generated responses are always supported by verifiable medical evidence, effectively alleviating the common problem of factual illusion in multimodal language models in professional medical scenarios. Combined with cross-modal alignment and efficient parameter fine-tuning strategies, a deep fusion of visual information, textual semantics, and medical knowledge is achieved with only a small number of training parameters, resulting in a highly reliable, low-illusion, and clinically practical intelligent ophthalmological disease consultation system with limited computing power. Attached Figure Description

[0024] Figure 1 The implementation steps of this invention are described below.

[0025] Figure 2 This is a schematic diagram of the overall network structure of the multimodal diagnostic model constructed in this embodiment of the invention, including a visual encoder, a Q-Former cross-modal mapping module, and a multimodal language model.

[0026] Figure 3 This is a schematic diagram of the partial structure of the visual encoder and language model in an embodiment of the present invention, including a convolutional neural network structure based on Xception and a language model structure based on Transformer.

[0027] Figure 4 This is a graph showing the change in diagnostic accuracy as a function of referral rate in an embodiment of the present invention.

[0028] Figure 5 This is a performance comparison chart between the model of this invention embodiment and other general multimodal models.

[0029] Figure 6 This is an example of a model simulating a doctor's participation in multiple rounds of consultation with a patient, as described in an embodiment of the present invention. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0031] like Figure 1 , Figure 2 As shown, this invention provides a reliable diagnostic method for ophthalmic diseases based on multimodal language models and evidence theory to guide uncertainty perception. The overall process includes the following steps: S1. Preprocess the color fundus diagnostic images containing fundus disease information to make them suitable for the size designed for the input model.

[0032] In this step, the fundus image can be acquired by an ophthalmic imaging acquisition device such as a fundus camera or a non-mydriatic fundus camera. It should be understood that the image can be a fundus image acquired in real time, or it can be retrieved from a local or cloud database that stores historical fundus examination data.

[0033] In one possible implementation, the fundus image preprocessing method includes converting the original fundus color image into the RGB standard color space, performing center cropping, and uniformly scaling it to a tensor form of size 3×224×224 to meet the input requirements of the subsequent visual encoder.

[0034] S2. Input the above-mentioned color fundus diagnostic images into the pre-trained visual encoder to extract visual features containing pathological information, and at the same time, perform quantitative modeling of diagnostic uncertainty.

[0035] The visual encoder is a feature extraction network built on a high-performance convolutional structure. This network can represent lesions and structures at different scales in fundus images and has more stable feature generalization ability than general visual models in small-sample training scenarios. The pre-trained visual encoder is jointly trained using fundus images with class labels based on evidence modeling and prototype contrastive learning. To balance the discriminative performance of minority class samples under severe class imbalance conditions with the reliability of model prediction confidence, a training method combining balanced evidence learning and prototype contrastive learning is adopted. In one possible implementation, the visual encoder is based on the Xception architecture, with the specific structure as follows: Figure 3 As shown, it consists of three main modules: the inlet flow module, the intermediate flow module, and the outlet flow module. Each module uses depthwise separable convolution as the core operation unit. By separating spatial convolution and channel convolution, it achieves efficient feature extraction. While significantly reducing parameter scale and computational complexity, it maintains a strong ability to represent the texture and local structure of complex fundus lesions. Thus, it has the ability to robustly model fine-grained pathological regions and effectively capture lesion features at different scales.

[0036] In this step, the uncertainty quantification modeling is based on Dempster-Shafer theory and evidence theory. Specifically, the uncertainty modeling process begins with the network outputting a non-negative evidence vector. Construct the Dirichlet distribution parameters:

[0037] Then, predict the category probability. With forecast uncertainty They are represented as follows:

[0038] in, This represents the total number of categories of fundus diseases.

[0039] In one possible implementation, the quantified uncertainty information is primarily determined by the continuous numerical uncertainty index used in uncertainty modeling and its statistical distribution on the training dataset. First, statistical analysis is performed on the uncertainty values ​​of all samples. Based on the quantile thresholds of their empirical distribution curves, the uncertainty is divided into three intervals: high uncertainty, medium uncertainty, and low uncertainty, and each interval is mapped to a corresponding set of semantic labels. Then, the uncertainty semantics corresponding to different intervals are injected into the system prompts in text form, thereby generating three differentiated diagnostic guidance texts for high, medium, and low uncertainty. These texts guide the language model to adopt different expression strategies in its responses, such as cautious diagnosis, suggestion for supplementary examinations, and explicit treatment recommendations.

[0040] In one possible implementation, the multimodal language model is a ChatGLM language model based on the Transformer architecture, which uses the LoRA method to perform parameter-efficient model fine-tuning by introducing a low-rank adaptation matrix in the Transformer layer.

[0041] In one possible implementation, the multimodal model of fundus diseases based on uncertainty perception and retrieval enhancement is obtained by first-stage full training of an Xception-based visual encoder and second-stage fine-tuning of a language module.

[0042] Specifically, the full training process for the Xception-based visual encoder employs a training method combining balanced evidence deep learning and prototype-guided contrastive learning. The balanced evidence learning loss function used is:

[0043] in, The coefficient of the training penalty term guided by misclassification; The Kullback-Leibler divergence term, representing the posterior Dirichlet distribution and the uniform prior, is used to suppress the accumulation of false evidence. The class balancing weights based on the number of valid samples are defined as follows:

[0044] in, As the attenuation factor, For the first The number of training samples for each class. Furthermore, a prototype contrastive learning loss is introduced to enhance the expressive power of minority class features, assuming the sample features are represented as... The category prototype is For the two enhancement methods of the original input image and Then the prototype contrastive learning loss is defined as:

[0045] Finally, the joint training loss function for the visual encoder is:

[0046] in, and This is the loss term balancing coefficient, used to coordinate the weight allocation between uncertainty modeling and minority class discrimination ability. This refers to the temperature parameter.

[0047] S3. The extracted visual features are further input into the cross-modal alignment module to obtain an aligned visual embedding with the same text embedding dimension as the language model. In one possible implementation, the cross-modal alignment module is a Q-Former-based query Transformer structure, the structure of which is as follows: Figure 3 As shown, this network introduces a multi-head self-attention mechanism and a cross-modal cross-attention mechanism to model and reweight the high-dimensional visual features output by the visual encoder, mapping them to generate visual lexical embeddings with the same dimension as the word vector space of the language model. This achieves accurate alignment of visual semantics to the language semantic space without introducing large-scale parameter updates, and effectively preserves key discriminative information of fundus lesion regions, providing a unified embedding representation basis for subsequent multimodal language models to understand and reason about visual content.

[0048] The language model is a language model fine-tuned using the LoRA method. The steps for fine-tuning the language model include: firstly, constructing a multi-turn dialogue dataset, and then using the LoRA method to fine-tune the language model. The construction of the multi-turn dialogue dataset refers to constructing a multi-turn dialogue dataset containing high-performance large-scale ophthalmology multi-turn consultation dialogue data and real ophthalmology outpatient dialogue text. The high-performance large-scale ophthalmology multi-turn consultation dialogue data is multi-turn dialogue data obtained based on the dialogue between two independent high-performance models. Its construction method is as follows: given a fundus image, model A simulates a patient asking a relevant question, and then another high-performance multimodal large-scale model B makes an appropriate and credible response based on the provided fundus image and the consultation text of model A. Then, through multiple iterations, a multi-turn dialogue sample is generated. The above process is repeated to finally obtain the high-performance large-scale ophthalmology multi-turn consultation dialogue dataset, which can be used for fine-tuning training of the language model. The LoRA method used to fine-tune the language model involves multi-turn dialogue learning based on the constructed dataset. The loss function for this process is:

[0049] In the formula, Indicates the first In turn-based dialogue, the model outputs a prediction of the first turn. Cross-entropy loss of each word in the answer text; Indicates the first i The total length of the response text in the round-robin consultation dialogue; Indicates the first Before answering the text in a turn-based dialogue Each word element; This indicates its predicted probability; This represents the semantic embedding of corresponding fundus image visual features; Indicates from the Text embedding of historical dialogues. In this process, the model can automatically learn how to infer the semantic expression of the current condition, diagnostic intent, and logical structure of the answer from the visual features of fundus images and the context of historical multi-turn dialogues containing patient inquiries and system responses, thereby generating consultation response text that is highly relevant to the patient's questions and conforms to medical norms.

[0050] S4. Based on the user's input query related to fundus diseases, the consultation text is transformed into a vector embedding in the latent space. Finally, the model integrates the consultation text vector embedding, visual embedding, and quantified uncertainty information to generate a reliable response.

[0051] Specifically, users input their query questions via natural language, such as: "What type of disease is shown in this fundus image?", "What are the possible causes of this disease?", or "How should this disease be treated?". After receiving the text, the system converts the abstract text into a continuous vector in the latent space through a pre-trained lexical embedding layer. This vector is then combined with visual semantic features aligned using a Q-Former-based query Transformer structure and fed into the language model.

[0052] Regarding the reply format, for example, such as Figure 2 As shown, when a user asks the question "What does the fumndus image suggest about my ocular health?", the pre-trained ChatGLM language model outputs the response "The fumndus image suggests that there may be .."

[0053] Specifically, the process of fine-tuning the training language module includes: constructing a vector database based on an ophthalmology medical knowledge base; performing similarity retrieval based on patient queries to obtain relevant medical knowledge; injecting the retrieval results into system prompts to generate a multi-round consultation dialogue dataset supported by medical evidence; and fine-tuning the language model based on the obtained dataset.

[0054] Experimental results verification In the following specific implementation case, publicly available fundus image datasets, an ophthalmology knowledge base based on Wikipedia, and real clinical desensitized samples are used to construct the data required for training the model. The fundus images are sourced from multiple mainstream datasets, covering various common fundus diseases such as normal fundus, diabetic retinopathy, glaucoma, macular degeneration, and pathological myopia. Based on the retrieval-enhanced multi-turn dialogue construction method described in this invention, multi-turn consultation text data supported by medical evidence is automatically generated, thus forming a multimodal training dataset that simultaneously includes fundus images, standard diagnostic labels, and multi-turn consultation text.

[0055] Experimental results show that the uncertainty-aware visual encoder based on Xception proposed in this invention achieves the best comprehensive diagnostic performance among different mainstream skeleton structures. It significantly outperforms the comparative models such as ResNet-50, DenseNet-121, Inception V3 and ViT in terms of accuracy, precision, recall, F1 score and AUROC score. At the same time, it achieves higher diagnostic accuracy while keeping the number of trainable parameters moderate. The specific results are shown in Table 1.

[0056] Table 1. Diagnostic performance index evaluation of different visual encoder skeletons ResNet-50 0.7126 0.3233 0.3373 0.3151 0.7862 27.12 M DenseNet-121 0.5423 0.3607 0.3377 0.2280 0.7698 9.13 M Inception V3 0.7505 0.4506 0.4529 0.4424 0.8147 25.40 M ViT 0.6196 0.2481 0.2701 0.2194 0.7511 87.61 M ViT (LORA Fine-tuning) 0.3769 0.0538 0.1429 0.0782 0.3791 2.37 M Xception 0.7968 0.6905 0.6020 0.6243 0.9536 24.42 M Furthermore, the experiments demonstrate that evidence-based uncertainty modeling methods can effectively improve the mismatch between the predicted confidence distribution and the true accuracy in traditional cross-entropy training models. For example... Figure 4As shown, after introducing uncertainty modeling, the model's prediction confidence and actual diagnostic accuracy exhibit a more consistent trend: when the prediction uncertainty is high, the corresponding diagnostic accuracy decreases synchronously. Although the model's simple classification accuracy is slightly lower than the model without uncertainty modeling, the overall diagnostic performance is significantly improved after referring approximately 10% of high-uncertainty samples, outperforming traditional methods. Given that high-uncertainty samples are more prone to misclassification, while low-uncertainty samples generally have higher reliability, this invention proposes a strategy of pre-screening samples based on prediction uncertainty and introducing manual intervention when necessary. Specifically, uncertainty values ​​are divided into three categories—low uncertainty, medium uncertainty, and high uncertainty—based on empirical distribution. High-uncertainty samples, accounting for approximately 10%, can be prioritized for referral in clinical practice for further examination or manual review, thereby improving the overall reliability of the diagnostic system and reducing the workload of clinicians.

[0057] This invention also evaluates the constructed multimodal medical consultation model from the perspective of the quality of the generated consultation text, to verify its ability to understand text queries and the reliability of diagnostic responses. Good text understanding, instruction compliance, and high-quality generation capabilities not only ensure semantic correctness and logical consistency in single-modal text scenarios, but also reduce the accumulation of errors caused by semantic drift or misunderstanding during multi-round interactions, laying the foundation for subsequent multimodal and cross-modal tasks. Regarding evaluation metrics, in addition to using commonly used automated evaluation metrics such as BLEU, METEOR, ROUGE-L, and BERT F1 scores to measure the fluency and semantic consistency of responses, a no-reference artificial intelligence evaluation method based on a high-capacity, large-language model is further introduced to comprehensively evaluate the semantic correctness and reasonableness of the generated content under given instructions. Specifically, four dimensions—Correlation, Fluency, Credibility, and Obedience—are selected as comprehensive evaluation criteria to reflect the model's performance in instruction understanding, content organization, and factual consistency, respectively. Using GPT-5 as a neutral evaluation model, the comparative methods were randomly scored multiple times on a five-point Likert scale, and the average scores are listed in Tables 2 and 3.

[0058] Table 2. Response Generation Quality Assessment Based on Quantitative Indicators

[0059] Table 3. Quality Assessment of Response Generation Based on High-Performance Large Language Model

[0060] Table 1 presents the comparison results of different models on objective metrics such as BLEU-1, BLEU-4, METEOR, ROUGE-L, and BERT F1 scores. It can be seen that the method of this invention has significant advantages in metrics such as BLEU-1 and BERT F1 scores, indicating that it maintains higher consistency with the reference answer at both the lexical and semantic levels, and can more effectively capture contextual semantic information and generate responses closer to the target answer in single-modal text scenarios. Table 2 further analyzes the performance of each model in terms of instruction compliance ability from the perspective of large-scale model evaluation. The method of this invention achieves the highest or tied highest scores in dimensions such as relevance, credibility, and compliance, with an average score of 3.7, significantly better than other compared models, proving that it has higher stability and reliability in understanding instruction semantics, maintaining content relevance, and complying with user constraints. Although it is slightly lower than some general large-scale models in fluency, the difference is small and has a limited impact on overall performance.

[0061] In multimodal input scenarios, this invention further introduces two core dimensions—coverage and faithfulness—to evaluate the quality of diagnostic responses. Coverage measures the completeness of the response's coverage of key information, while faithfulness measures the factual accuracy and medical consistency of the generated content. This multidimensional evaluation system replaces traditional string matching methods at the semantic and knowledge levels, systematically evaluating the model in terms of content quality, reliability, and security. The comparison results are as follows: Figure 5 As shown, the method of this invention significantly outperforms general multimodal models such as DeepSeek-VL-2, Qwen2.5-VL-7B, and GLM-4.5V in key indicators such as relevance, authenticity, and credibility, fully demonstrating its ability to generate clinically relevant and factually reliable diagnostic responses in fundus disease scenarios. Although there are still some gaps in indicators such as fluency and compliance, this phenomenon mainly stems from the model's tendency to conservatively express diagnostic conclusions in uncertain sample scenarios, thereby avoiding overconfident assertions. This conservative behavior has positive significance in medical dialogue scenarios, effectively reducing the risk of misleading or unsafe outputs, and verifying the safety and practical value of the method of this invention in clinical applications.

[0062] Finally, the multi-round consultation capability of the invention model under different ophthalmic disease categories is demonstrated qualitatively and intuitively. For example... Figure 6 As shown, two representative cases were randomly selected (cataract on the left and pathological myopia on the right) to simulate the consultation process between the patient and the intelligent ophthalmology diagnostic assistant.

[0063] As can be seen, the model of this invention can accurately understand the questions raised by patients and generate fluent, coherent, and clinically accurate professional responses. Furthermore, by comparing the generated responses with the constructed domain vector knowledge base, it was found that the model can retrieve highly similar factual statements from external medical knowledge bases and incorporate them into the final response; relevant matching segments are marked with purple underlines in the figure. These results demonstrate that the retrieval-enhanced generation mechanism plays a positive role in the model training and inference process, enabling the model to provide verifiable answers based on authoritative medical literature, rather than relying solely on parameter memorization. This effectively reduces the risk of factual illusions and significantly improves the authenticity and reliability of multimodal diagnostic consultation responses.

[0064] In addition, the present invention also includes: A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a reliable diagnostic method for ophthalmic diseases based on multimodal language models and evidence theory to guide uncertainty perception, as described in claim 1.

[0065] A computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the computer program implements, as described in claim , a reliable diagnostic method for ophthalmic diseases based on multimodal language models and evidence theory to guide uncertainty perception.

[0066] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reliable ophthalmological consultation method based on a multimodal large language model with uncertainty perception, characterized in that: Includes the following steps: Step 1: Preprocess the fundus images containing fundus disease information to make them suitable for the size designed for the input model; Step 2: Input the preprocessed fundus image into the pre-trained visual encoder to extract visual features containing pathological information and diagnostic results. Model the uncertainty of the diagnostic results to obtain quantified uncertainty information. Step 3: Input the obtained visual features into the cross-modal alignment module to obtain aligned visual embeddings that are easy for the language model to understand; Step 4: The user's query related to fundus diseases is used as the consultation text. The data retrieval function is performed on the consultation text, that is, similar typical cases are retrieved from the constructed vector database or pathological knowledge segments are extracted to obtain the retrieval text. Then, the consultation text and the retrieval text are converted into latent space vector embeddings. The language model generates a reliable response based on the vector embeddings of the consultation text and the retrieval text, the alignment of visual embeddings, and the quantification of uncertainty information.

2. The multimodal large language model ophthalmology reliable consultation method with uncertainty perception according to claim 1, characterized in that, The fundus image is preprocessed by converting the original fundus image into the RGB standard color space, then cropping it from the center, and uniformly scaling it to a tensor format of appropriate size.

3. The multimodal large language model ophthalmology reliable consultation method with uncertainty perception according to claim 1, characterized in that, The visual encoder is a feature extraction network built on a convolutional structure, which can represent lesions and structures at different scales in fundus images. The pre-trained visual encoder is jointly trained using fundus images with category labels, based on evidence modeling and prototype contrastive learning. A training method combining balanced evidence learning and prototype contrastive learning is employed, wherein the balanced evidence learning loss function is: in, The coefficient of the training penalty term guided by misclassification; The Kullback-Leibler divergence term, representing the posterior Dirichlet distribution and the uniform prior, is used to suppress the accumulation of false evidence. The class balancing weights based on the number of valid samples are defined as follows: in, As the attenuation factor, For the first The number of training samples for the class; Prototype contrastive learning loss is used to enhance the representation of minority class features. Let the sample features be represented as... The category prototype is For the two enhancement methods of the original input image and Then the prototype contrastive learning loss is defined as: Finally, the joint training loss function used for the pre-trained visual encoder is: in, and This is the loss term balancing coefficient, used to coordinate the weight allocation between uncertainty modeling and minority class discrimination ability. This refers to the temperature parameter.

4. The multimodal large language model ophthalmology reliable consultation method with uncertainty perception according to claim 1, characterized in that, The modeling of the uncertainty of the diagnostic results is based on Dempster-Shafer theory and evidence theory. Dirichlet distribution parameters are constructed to characterize the probability of the predicted category and the uncertainty. The quantified uncertainty information is obtained by the statistical distribution law of the uncertainty characterized by the Dirichlet distribution parameters on a certain internal / external validation set. According to its distribution quantile threshold, it is mapped to the corresponding semantic label set, and the corresponding uncertainty semantics are injected into the system prompt words in text form as quantified uncertainty information. Its uncertainty modeling process includes: the network outputs a non-negative evidence vector. Construct the Dirichlet distribution parameters: Among them, the predicted class probability With forecast uncertainty They are respectively: in, This represents the total number of categories of fundus diseases.

5. The multimodal large language model ophthalmology reliable consultation method with uncertainty perception according to claim 1, characterized in that, The cross-modal alignment module is a pre-trained Q-Former network based on the Transformer architecture. This network introduces a multi-head self-attention mechanism and a cross-modal cross-attention mechanism to model and reweight the high-dimensional visual features output by the visual encoder, and maps them to generate visual word embeddings with the same dimension as the word vector space of the language model. This achieves accurate alignment of visual semantics to the language semantic space and provides a unified representation basis for subsequent multimodal reasoning and diagnostic text generation.

6. The multimodal large language model ophthalmic reliable consultation method with uncertainty perception according to claim 1, characterized in that, The language model is a language model fine-tuned using the LoRA method. The steps for fine-tuning the language model include: firstly, constructing a multi-turn dialogue dataset, and then using the LoRA method to fine-tune the language model. The construction of the multi-turn dialogue dataset refers to constructing a multi-turn dialogue dataset containing high-performance large-scale ophthalmology multi-turn consultation dialogue data and real ophthalmology outpatient dialogue text. The high-performance large-scale ophthalmology multi-turn consultation dialogue data is multi-turn dialogue data obtained based on the dialogue between two independent high-performance models. The construction method is as follows: given a fundus image, model A simulates a patient asking a relevant question. Then, another high-performance multimodal large-scale model B makes an appropriate and credible response based on the provided fundus image and the consultation text of model A. Then, through multiple iterations, a multi-turn dialogue sample is generated. The above process is repeated to finally obtain the high-performance large-scale ophthalmology multi-turn consultation dialogue dataset, which is used for fine-tuning training of the language model. The LoRA method used to fine-tune the language model involves multi-turn dialogue learning based on the constructed dataset. The loss function for this process is: In the formula, Indicates the first In turn-based dialogue, the model outputs a prediction of the first turn. Cross-entropy loss of each word in the answer text; Indicates the first i The total length of the response text in the round-robin consultation dialogue; Indicates the first Before answering the text in a turn-based dialogue Each word element; This indicates its predicted probability; This represents the semantic embedding of corresponding fundus image visual features; Indicates from the Text embedding of historical dialogues.

7. The multimodal large language model ophthalmology reliable consultation method with uncertainty perception according to claim 1, characterized in that, The constructed vector database is based on an open-source ophthalmic medical knowledge text library and typical cases of the gold standard for consultation. The pre-trained medical text embedding model BLIP is used to vectorize the text, and the constructed locally usable vector database can perform data retrieval functions, that is, to perform similarity retrieval based on the consultation text provided by the patient, in order to match a number of ophthalmic medical knowledge items and typical cases of the gold standard for consultation, thereby obtaining retrieval text containing medical evidence support.

8. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a reliable diagnostic method for ophthalmic diseases based on multimodal language models and evidence theory to guide uncertainty perception, as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a reliable diagnostic method for ophthalmic diseases based on multimodal language models and evidence theory to guide uncertainty perception, as described in any one of claims 1 to 7.

Images