A method and system for intelligent diagnosis and treatment management of ophthalmic diseases based on large models

Abstract

The present invention relates to the technical field of ophthalmic disease diagnosis, and discloses an intelligent diagnosis and treatment management method and system for ophthalmic diseases based on a large model, the method comprising: obtaining a fundus image of a patient to be diagnosed; inputting the fundus image of the patient to be diagnosed into a fundus disease classification and segmentation model to obtain the ophthalmic disease category and disease marker segmentation result corresponding to the fundus image to be diagnosed; inputting the ophthalmic disease category, disease marker segmentation result and clinical records of the patient to be diagnosed into a large language model to obtain a fundus disease treatment and management plan for the patient to be diagnosed; the large language model is based on the LLaMA model, and a LoRA layer is added to each multi-head attention layer of the LLaMA model; an Adapter layer is added between each feedforward layer and the normalization layer of the LLaMA model. The present invention can realize the rapid diagnosis and treatment of ophthalmic diseases, improve the diagnostic efficiency, and reduce the medical burden of patients.

Description

Technical Field

[0001] The present invention relates to the technical field of ophthalmic disease diagnosis, and in particular to an intelligent diagnosis and treatment management method and system for ophthalmic diseases based on a large model. Background Art

[0002] Currently, more than 2.2 billion people worldwide are visually impaired or blind, and the rising prevalence poses a major challenge to public health, especially in these resource-poor settings. There are many types of eye diseases with similar symptoms, and even experienced professionals are prone to misdiagnosis and missed diagnosis. With the continuous development of modern medicine, the diagnosis and treatment of eye diseases are increasingly dependent on high-tech means.

[0003] As a revolutionary technology in the field of artificial intelligence, deep learning has demonstrated amazing capabilities and potential. Its power lies in its ability to automatically learn and extract complex features and patterns from data without manual feature engineering. This has enabled deep learning models to achieve excellent performance in the screening and diagnosis of a variety of eye diseases. However, current deep learning-based eye disease screening models can only provide disease diagnosis results, and cannot further provide professional treatment recommendations based on the diagnosis results. Personalized treatment management recommendations require fully trained professional physicians to provide. In low- and middle-income and resource-poor areas, there are few trained professional physicians.

[0004] In recent years, large language models (LLMs) have developed rapidly. In computer vision, natural language processing and other fields, the wide applicability and advantages of large models have been demonstrated through many application cases. At the same time, these large models also show stronger generalization capabilities, can process data and fields that have never been seen, and provide a tailored experience for personalized artificial intelligence applications. In the medical field, LLMs also show good prospects in enhancing medical services, such as LLaMA, etc. LLMs may provide patients with personalized diagnosis and treatment recommendations and plans, which may improve the treatment outcomes of patients, but the current LLMs can only answer general medical questions and are not enough to provide professional and detailed disease management and treatment recommendations for a variety of specific ophthalmic diseases. Combining large language models (LLMs) and deep learning models for ophthalmic disease diagnosis and management can greatly improve diagnostic efficiency, provide patients with personalized treatment management recommendations, and reduce the burden of patients' visits. Therefore, the present invention proposes to combine large language model (LLM) technology with deep learning models and apply them to the field of ophthalmic disease diagnosis to achieve rapid diagnosis and treatment of diseases.

[0005] The existing methods and systems for diagnosing ophthalmic diseases have the following problems:

[0006] (1) Traditional ophthalmic disease diagnosis requires patients to go to the hospital for diagnosis, which is a cumbersome process and a poor patient experience. In addition, the diagnosis relies on professional doctors, which is inefficient and the doctor's diagnosis results may be subjective.

[0007] (2) The existing deep learning-based ophthalmic disease diagnosis system can only provide fundus disease screening results, and professional doctors are still required to give further treatment recommendations based on the patient's condition;

[0008] (3) Existing large language models (LLMs) used in the medical field can only answer general medical questions and cannot provide professional and personalized treatment management recommendations based on the specific conditions of ophthalmic patients.

[0009] Therefore, there is an urgent need for an intelligent diagnosis and treatment management method and system for ophthalmic diseases based on a large model, which can realize the rapid diagnosis and treatment of ophthalmic diseases, improve diagnostic efficiency, and reduce the burden of medical treatment for patients. Summary of the invention

[0010] In order to solve the above technical problems, the present invention provides an intelligent diagnosis and treatment management method and system for ophthalmic diseases based on a large model, which can realize the rapid diagnosis and treatment of ophthalmic diseases, improve diagnostic efficiency, and reduce the patient's medical burden.

[0011] The present invention provides an intelligent diagnosis and treatment management method for ophthalmic diseases based on a large model, comprising the following steps:

[0012] S1. Acquire fundus images of patients to be diagnosed;

[0013] S2, inputting the fundus image of the patient to be diagnosed into the fundus disease classification and segmentation model to obtain the ophthalmic disease category and disease marker segmentation results corresponding to the fundus image of the patient to be diagnosed;

[0014] S3, inputting the ophthalmic disease category, disease marker segmentation results and clinical records of the patient to be diagnosed into the large language model to obtain the fundus disease treatment and management plan for the patient to be diagnosed;

[0015] The large language model is based on the LLaMA model, and a LoRA layer is added to each multi-head attention layer of the LLaMA model; the LoRA layer includes a dimensionality reduction layer and a dimensionality expansion layer;

[0016] An Adapter layer is added between each feed-forward layer and the normalization layer of the LLaMA model; the Adapter layer includes two layers of multilayer perceptrons and an activation layer.

[0017] In S2, the fundus image of the patient to be diagnosed is input into the fundus disease classification and segmentation model to obtain the ophthalmic disease category and disease marker segmentation result corresponding to the fundus image of the patient to be diagnosed, wherein the training method of the fundus disease classification and segmentation model includes:

[0018] Acquire original fundus images;

[0019] Performing enhancement processing on the original fundus image;

[0020] Label the enhanced original fundus images to construct an original fundus image dataset; wherein the labels include disease classification and disease markers;

[0021] The deep learning model is trained using the original fundus image dataset to obtain a fundus disease classification and segmentation model.

[0022] Furthermore, the deep learning model uses the Transformer module as the backbone network;

[0023] The deep learning model consists of a classification branch and a segmentation branch;

[0024] The classification branch is used to output ophthalmic disease categories based on the original fundus image dataset;

[0025] The segmentation branch is used to output disease marker segmentation results based on the original fundus image dataset.

[0026] Furthermore, the dimension reduction matrix of the dimension reduction layer is initialized using a random Gaussian distribution;

[0027] The expansion matrix of the expansion layer is initialized to a zero matrix.

[0028] Further, in S3, the ophthalmic disease category, the disease marker segmentation result, and the clinical record of the patient to be diagnosed are input into the large language model to obtain the fundus disease treatment and management plan of the patient to be diagnosed, wherein the training method of the large language model includes:

[0029] Obtain the patient's clinical records corresponding to the original fundus images;

[0030] Construct an ophthalmic disease question-answering dataset based on patient clinical records, ophthalmic disease categories, disease marker segmentation results, and fundus disease treatment and management plans;

[0031] The ophthalmic disease question-and-answer dataset is used to train the large language model to obtain a trained large language model.

[0032] Furthermore, when training the large language model, the parameters of the LLaMA model remain unchanged, and only the parameters of the LoRA layer are trained.

[0033] Furthermore, in S2, the fundus image of the patient to be diagnosed is input into the fundus disease classification and segmentation model, including:

[0034] Performing image segmentation on the fundus image of the patient to be diagnosed to obtain a number of image blocks;

[0035] Several image patches are input into the fundus disease classification and segmentation model.

[0036] The present invention also provides an intelligent diagnosis and treatment management system for ophthalmic diseases based on a large model, which is used to execute any of the above-mentioned intelligent diagnosis and treatment management methods for ophthalmic diseases based on a large model. The system includes the following modules:

[0037] An image acquisition module, used for acquiring fundus images of patients to be diagnosed;

[0038] The classification and segmentation module is connected to the image acquisition module and is used to input the fundus image of the patient to be diagnosed into the fundus disease classification and segmentation model to obtain the ophthalmic disease category and disease marker segmentation results corresponding to the fundus image of the patient to be diagnosed;

[0039] A result output module, connected to the classification and segmentation module, is used to input the ophthalmic disease category, the disease marker segmentation results and the clinical records of the patient to be diagnosed into the large language model to obtain the fundus disease treatment and management plan for the patient to be diagnosed;

[0040] The large language model is based on the LLaMA model, and a LoRA layer is added to each multi-head attention layer of the LLaMA model; the LoRA layer includes a dimensionality reduction layer and a dimensionality expansion layer;

[0041] An Adapter layer is added between each feed-forward layer and the normalization layer of the LLaMA model; the Adapter layer includes two layers of multilayer perceptrons and an activation layer.

[0042] The embodiments of the present invention have the following technical effects:

[0043] The present invention fine-tunes the large language model by adding the LoRA layer and the Adapter layer to the large language model. During training, only the parameters of the LoRA layer and the Adapter layer need to be updated, while the parameters of the original large language model remain unchanged, so that the parameters of the large language model can be fine-tuned and trained efficiently, thereby improving the training efficiency of the model; the automatic diagnosis and management of ophthalmic diseases are realized through the large language model and the deep learning model, which greatly improves the diagnosis efficiency, reduces the medical burden of patients, improves the efficiency and accuracy of ophthalmic medical diagnosis, and provides patients with better medical services;

[0044] The present invention can diagnose ophthalmic diseases based only on the patient's fundus photographs and clinical data without any other additional intervention, and can simultaneously achieve fully automated rapid disease screening diagnosis and disease management. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a flow chart of a method for intelligent diagnosis and treatment management of ophthalmic diseases based on a large model provided by an embodiment of the present invention;

[0047] Figure 2 It is a structural schematic diagram of a large model used in an intelligent diagnosis and treatment management method for ophthalmic diseases based on a large model provided in an embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of the structure of a large language model of a fine-tuning part provided by an embodiment of the present invention;

[0049] Figure 4 It is a structural diagram of an intelligent diagnosis and treatment management system for ophthalmic diseases based on a large model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0051] The present invention proposes an intelligent diagnosis and treatment management method for ophthalmic diseases based on a large model. Figure 1 is a flow chart of a large model-based intelligent diagnosis and treatment management method for ophthalmic diseases provided by an embodiment of the present invention. Figure 2 is a schematic diagram of the structure of a large model used in a large model-based intelligent diagnosis and treatment management method for ophthalmic diseases provided in an embodiment of the present invention, see Figure 1 and Figure 2 , including:

[0052] S1. Acquire a fundus image of a patient to be diagnosed.

[0053] S2. Input the fundus image of the patient to be diagnosed into the fundus disease classification and segmentation model to obtain the ophthalmic disease category and disease marker segmentation results corresponding to the fundus image of the patient to be diagnosed.

[0054] Specifically, the fundus image of the patient to be diagnosed is segmented to obtain a number of image blocks;

[0055] Several image patches are input into the fundus disease classification and segmentation model.

[0056] Specifically, the training method of the fundus disease classification and segmentation model includes:

[0057] Acquire original fundus images;

[0058] Performing enhancement processing on the original fundus image;

[0059] Among them, the enhancement processing may include random rotation, random horizontal flipping, random vertical flipping, etc.

[0060] Label the enhanced original fundus images to construct an original fundus image dataset; wherein the labels include disease classification and disease markers;

[0061] Exemplarily, the disease classification may include: glaucoma, macular degeneration, central or branch retinal artery occlusion, central or branch retinal vein occlusion, hypertensive retinopathy, diabetic retinopathy, age-related macular degeneration, leopard-like lesions, high myopia retinopathy and other ophthalmic diseases. Disease markers may include: arteriovenous vessels, optic cup and disc, leopard pattern, arcuate spots, hemorrhage, exudation, microaneurysm, edema, drusen and the like.

[0062] The deep learning model is trained using the original fundus image dataset to obtain a fundus disease classification and segmentation model.

[0063] Among them, the deep learning model uses the Transformer module as the backbone network;

[0064] The deep learning model consists of a classification branch and a segmentation branch;

[0065] The classification branch is used to output ophthalmic disease categories based on the original fundus image dataset;

[0066] The segmentation branch is used to output disease marker segmentation results based on the original fundus image dataset.

[0067] Furthermore, the Transformer module consists of a series of Transformer layers. After the last Transformer layer, a global average pooling operation is used to obtain the feature representation of the original fundus image. After the global average pooling operation, a classification branch and a segmentation branch are set. The classification branch consists of a fully connected layer and an output layer. The number of output units is the number of ophthalmic disease categories. For the classification branch, the cross entropy loss is used as the objective function to optimize the model parameters:

[0068] ;

[0069] in, Represents the cross entropy loss function, which is used to predict the category probability of the classification branch and true disease labels of fundus image samples Calculate the loss value, L cls Represents the classification loss, N represents the number of samples in the training set, and i represents the i-th sample.

[0070] The segmentation branch consists of a convolutional layer, an upsampling layer, and an output layer. For the segmentation branch, the cross entropy loss is used as the objective function to optimize the model parameters:

[0071] ;

[0072] in, Represents the cross entropy loss function, which is used to predict the category probability of the segmentation branch And the disease markers of fundus image samples are truly labeled m i Calculate the loss value, represents the dice loss function, which is used to alleviate the imbalance phenomenon and improve the model's recognition performance for foreground objects. seg In this embodiment, the classification loss and the segmentation loss are combined to optimize the overall model parameters.

[0073] S3. Input the ophthalmic disease category, disease marker segmentation results, and clinical records of the patient to be diagnosed into the large language model to obtain the fundus disease treatment and management plan for the patient to be diagnosed.

[0074] Among them, clinical records include age, gender, medical history, etc.; the results of ophthalmic disease Q&A include the causes of fundus diseases, risk groups, clinical manifestations, imaging manifestations, diagnosis and treatment, and health advice, etc.

[0075] Specifically, Figure 3 is a schematic diagram of the structure of a large language model of a fine-tuning part provided by an embodiment of the present invention, see Figure 3The large language model is based on the LLaMA (Large Language Model Meta AI) model, and adds a LoRA layer to each multi-head attention layer of the LLaMA model; the LoRA layer includes a dimensionality reduction layer and a dimensionality expansion layer;

[0076] An Adapter layer is added between each feedforward layer and the normalization layer of the LLaMA model; the Adapter layer includes two layers of multilayer perceptrons and an activation layer.

[0077] The dimension reduction matrix of the dimension reduction layer is initialized with a random Gaussian distribution; the dimension expansion matrix of the dimension expansion layer is initialized to a zero matrix.

[0078] Specifically, the original LLaMA model cannot provide professional treatment and management recommendations for ophthalmic diseases. Therefore, this embodiment collects paired clinical data and actual fundus disease treatment and management plans from patients in multiple regions, de-identifies the data and inputs it into the LLaMA model, and fine-tunes the LLaMA model; this method can enhance the model's ability to generate fundus disease treatment and management plans, can efficiently and accurately analyze patients' fundus images, and can identify lesions in fundus photographs, abnormalities in the eyeball structure, and potential risk factors for ophthalmic diseases, can achieve rapid detection and diagnosis of early-stage diseases, and can also quickly generate professional treatment and management recommendations based on the condition of ophthalmic diseases. Among them, fundus disease treatment and management plans may include: treatment recommendations, dietary recommendations, medication recommendations, and lifestyle recommendations, etc.

[0079] Specifically, the training method of the large language model includes:

[0080] Obtain the patient's clinical records corresponding to the original fundus images;

[0081] Construct an ophthalmic disease question-answering dataset based on patient clinical records, ophthalmic disease categories, disease marker segmentation results, and fundus disease treatment and management plans;

[0082] The ophthalmic disease question-and-answer dataset is used to train the large language model to obtain a trained large language model.

[0083] Furthermore, when training the large language model, the parameters of the LLaMA model remain unchanged, and only the parameters of the LoRA layer are trained.

[0084] Furthermore, for the output generated by the fundus disease classification and segmentation model, this embodiment designs standardized language templates, such as "Disease category: age-related macular degeneration; Lesion: drusen". These language templates can be combined with other clinical metadata as input to the LLaMA model to generate personalized management recommendations for patients with ophthalmic diseases.

[0085] The present invention fine-tunes the large language model by adding the LoRA layer and the Adapter layer to the large language model. During training, only the parameters of the LoRA layer and the Adapter layer need to be updated, while the parameters of the original large language model remain unchanged, so that the parameters of the large language model can be fine-tuned and trained efficiently, thereby improving the training efficiency of the model; the automatic diagnosis and management of ophthalmic diseases are realized through the large language model and the deep learning model, which greatly improves the diagnosis efficiency, reduces the medical burden of patients, improves the efficiency and accuracy of ophthalmic medical diagnosis, and provides patients with better medical services;

[0086] The present invention can diagnose ophthalmic diseases based only on the patient's fundus photographs and clinical data without any other additional intervention, and can simultaneously achieve fully automated rapid disease screening diagnosis and disease management.

[0087] Figure 4 is a schematic diagram of the structure of an intelligent diagnosis and treatment management system for ophthalmic diseases based on a large model provided by an embodiment of the present invention. The system is used to execute an intelligent diagnosis and treatment management method for ophthalmic diseases based on a large model described in the above embodiment. Figure 4 As shown, the system includes the following modules:

[0088] An image acquisition module, used for acquiring fundus images of patients to be diagnosed;

[0089] The classification and segmentation module is connected to the image acquisition module and is used to input the fundus image of the patient to be diagnosed into the fundus disease classification and segmentation model to obtain the ophthalmic disease category and disease marker segmentation results corresponding to the fundus image of the patient to be diagnosed;

[0090] A result output module, connected to the classification and segmentation module, is used to input the ophthalmic disease category, the disease marker segmentation results and the clinical records of the patient to be diagnosed into the large language model to obtain the fundus disease treatment and management plan for the patient to be diagnosed;

[0091] The large language model is based on the LLaMA model, and a LoRA layer is added to each multi-head attention layer of the LLaMA model; the LoRA layer includes a dimensionality reduction layer and a dimensionality expansion layer;

[0092] An Adapter layer is added between each feed-forward layer and the normalization layer of the LLaMA model; the Adapter layer includes two layers of multilayer perceptrons and an activation layer.

[0093] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0094] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent diagnosis and treatment management of ophthalmic diseases based on a large model, characterized in that: The steps include: S1. Acquire fundus images of patients to be diagnosed; S2, inputting the fundus image of the patient to be diagnosed into a fundus disease classification and segmentation model to obtain an ophthalmic disease category and disease marker segmentation result corresponding to the fundus image of the patient to be diagnosed; The training method of the fundus disease classification and segmentation model includes: Acquire original fundus images; Performing enhancement processing on the original fundus image; Labeling the enhanced original fundus image to construct an original fundus image dataset; wherein the labels include disease classification and disease markers; Using the original fundus image dataset to train a deep learning model to obtain the fundus disease classification and segmentation model; The deep learning model uses the Transformer module as the backbone network; The deep learning model includes a classification branch and a segmentation branch; The classification branch is used to output an ophthalmic disease category according to the original fundus image dataset; The segmentation branch is used to output a disease marker segmentation result according to the original fundus image data set; The Transformer module is composed of a series of Transformer layers. After the last Transformer layer, a global average pooling operation is used to obtain the feature representation of the original fundus image. After the global average pooling operation, a classification branch and a segmentation branch are set. The classification branch is composed of a fully connected layer and an output layer. The number of output units is the number of ophthalmic disease categories. The classification loss of the classification branch is as follows: ； in, represents the cross entropy loss function, represents the category probability predicted by the classification branch, represents the true disease label of the fundus image sample, L cls Represents classification loss, N represents the number of samples in the training set, and i represents the i-th sample; The segmentation branch consists of a convolutional layer, an upsampling layer and an output layer; The segmentation loss of the segmentation branch is as follows: ； in, represents the cross entropy loss function, represents the category probability predicted by the segmentation branch, m i The real annotations of disease markers representing fundus image samples, represents the dice loss function, L seg represents the segmentation loss; Optimizing overall model parameters based on the classification loss and the segmentation loss; S3, inputting the ophthalmic disease category, the disease marker segmentation result and the clinical record of the patient to be diagnosed into a large language model to obtain a fundus disease treatment and management plan for the patient to be diagnosed; The large language model is based on the LLaMA model, and a LoRA layer is added to each multi-head attention layer of the LLaMA model; the LoRA layer includes a dimensionality reduction layer and a dimensionality expansion layer; An Adapter layer is added between each feedforward layer and the normalization layer of the LLaMA model; the Adapter layer includes two layers of multilayer perceptrons and an activation layer; When the large language model is trained, the parameters of the LLaMA model remain unchanged, and only the parameters of the LoRA layer are trained.

2. The method for intelligent diagnosis and treatment of ophthalmic diseases based on a large model according to claim 1, characterized in that: The dimension reduction matrix of the dimension reduction layer is initialized using a random Gaussian distribution; The expanded matrix of the expanded layer is initialized to a zero matrix.

3. The method for intelligent diagnosis and treatment of ophthalmic diseases based on a large model according to claim 2, characterized in that: In S3, the ophthalmic disease category, the disease marker segmentation result, and the clinical record of the patient to be diagnosed are input into a large language model to obtain a fundus disease treatment and management plan for the patient to be diagnosed, wherein the training method of the large language model includes: Obtain the patient's clinical records corresponding to the original fundus images; Constructing an ophthalmic disease question-and-answer dataset based on the patient's clinical records, the ophthalmic disease categories, the disease marker segmentation results, and the fundus disease treatment and management plan; The ophthalmic disease question-and-answer dataset is used to train a large language model to obtain a trained large language model.

4. The method for intelligent diagnosis and treatment of ophthalmic diseases based on a large model according to claim 1, characterized in that: In S2, the fundus image of the patient to be diagnosed is input into a fundus disease classification and segmentation model, including: Performing image segmentation on the fundus image of the patient to be diagnosed to obtain a plurality of image blocks; Several image patches are input into the fundus disease classification and segmentation model.

5. An intelligent diagnosis and treatment management system for ophthalmic diseases based on a large model, used to execute an intelligent diagnosis and treatment management method for ophthalmic diseases based on a large model as described in any one of claims 1 to 4, characterized in that: The system includes the following modules: An image acquisition module, used for acquiring fundus images of patients to be diagnosed; A classification and segmentation module, connected to the image acquisition module, for inputting the fundus image of the patient to be diagnosed into the fundus disease classification and segmentation model to obtain the ophthalmic disease category and disease marker segmentation result corresponding to the fundus image of the patient to be diagnosed; A result output module, connected to the classification and segmentation module, is used to input the ophthalmic disease category, the disease marker segmentation result and the clinical record of the patient to be diagnosed into a large language model to obtain a fundus disease treatment and management plan for the patient to be diagnosed; The large language model is based on the LLaMA model, and a LoRA layer is added to each multi-head attention layer of the LLaMA model; the LoRA layer includes a dimensionality reduction layer and a dimensionality expansion layer; An Adapter layer is added between each feedforward layer and the normalization layer of the LLaMA model; the Adapter layer includes two layers of multilayer perceptrons and an activation layer.

Images