Ophthalmology auxiliary diagnosis report generation method based on multi-mode ophthalmology knowledge graph

Abstract

The invention provides an ophthalmology auxiliary diagnosis report generation method based on a multi-modal ophthalmology knowledge graph. The method comprises the following steps: acquiring the multi-modal ophthalmology knowledge graph and a fundus image of a detection object; generating a corresponding auxiliary diagnosis report according to the eye fundus image by using a trained report generation model, wherein the method comprises the following steps: extracting image features from the eye fundus image of the detected object; generating enhanced visual features after knowledge enhancement; key knowledge features are extracted from key entity nodes recalled from the multi-modal ophthalmologic knowledge graph according to the fundus image of the detection object by using a graph encoder; the big language model is guided by the prompt template to generate an auxiliary diagnosis report according to the image features, the enhanced visual features and the key knowledge features of the eye fundus image of the detected object; the key knowledge features extracted from the key entity nodes recalled from the multi-modal ophthalmic knowledge graph by the fundus image are utilized to help the large language model to generate the auxiliary diagnosis report, and the professional, generalization ability and clinical availability of the model to generate the auxiliary diagnosis report are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology, such as deep learning and large language models, specifically to the field of generative artificial intelligence technology, and more specifically, to a method for generating ophthalmological auxiliary diagnostic reports based on a multimodal ophthalmological knowledge graph. Background Technology

[0002] The World Health Organization’s World Vision Report points out that vision health problems are becoming increasingly prominent globally, with more than 2.2 billion people suffering from vision impairment worldwide, and more than 1 billion people suffering from vision problems due to lack of access to necessary medical interventions.

[0003] The development of digital ophthalmology technology has made remote-assisted medical care possible, especially in areas with scarce medical resources. Through intelligent algorithms and remote image analysis, it can provide patients with timely and efficient ophthalmological services. This not only improves diagnostic and treatment efficiency but also promotes the equitable distribution of medical resources.

[0004] In clinical practice, the writing of ophthalmological diagnostic reports still relies heavily on the professional experience of clinicians, characterized by strong subjectivity and time-consuming processes. To alleviate the burden on doctors and improve diagnostic efficiency, automated report generation technology has gradually gained attention. This technology typically involves joint modeling of image and text modalities. However, most publicly available datasets in the ophthalmology field, such as AIROGS and OIA-DDR, only provide images and corresponding disease category labels, lacking detailed clinical text descriptions. Even the few datasets containing image-text pairs, such as STARE (containing 397 images and brief descriptive text), MESSIDOR (containing 1200 images and 600 reports), and FFA-IR (containing over one million images and only 10790 reports), struggle to support high-quality model training due to limited data size or access constraints. While DeepEyeNet (DEN), currently the largest ophthalmological image-text dataset, covers 15709 image-text pairs, its content heterogeneity is significant, further increasing the complexity of training report generation models.

[0005] Regarding modeling methods, several deep learning-based report generation methods have been proposed, such as keyword-guided models and multi-view visual feature-guided models. These methods are mostly developed for specific tasks and datasets, lacking generality, facing difficulties in model transfer, and exhibiting poor stability in generated results, making them unsuitable for practical clinical applications. In recent years, general-purpose multimodal large language models (such as GPT-4 and LLaVA) have performed well in text-image generation tasks. Their training typically relies on large-scale, high-quality text-image pairs; for example, LLaVA was trained and fine-tuned based on the CC3M-595K dataset and 160,000 visual language instructions. However, these general-purpose models face significant challenges in the medical field.

[0006] On the one hand, medical image and text data are scarce, making it difficult to support pre-training paradigms; on the other hand, medical reports contain a large amount of professional knowledge, and general-purpose large language models are prone to "illusion" phenomena without knowledge injection, affecting their clinical reliability. Recent research shows that introducing medical knowledge can significantly improve the model's performance in ophthalmological assisted diagnostic tasks, providing a new approach to improving the model's interpretability and practical usability.

[0007] However, there is still room for improvement in the professionalism and accuracy of generative models in ophthalmic assisted diagnostic tasks.

[0008] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0009] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method for generating ophthalmological auxiliary diagnostic reports based on a multimodal ophthalmological knowledge graph.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] According to a first aspect of the present invention, a training method for a model for generating ophthalmological auxiliary diagnostic reports is provided, comprising: constructing a report generation model, which includes: a visual encoder, a knowledge enhancement layer, a graph encoder, and a large language model; acquiring a training set, which includes multiple samples, the samples including fundus images and their diagnostic reports; performing multiple iterative training on the report generation model using the training set, each training comprising: extracting image features from the fundus images of the samples using the visual encoder; processing the image features using the knowledge enhancement layer to generate enhanced visual features after knowledge enhancement; extracting report knowledge features from entity nodes contained in the diagnostic reports corresponding to the fundus images using the graph encoder, and extracting key knowledge features from key entity nodes recalled from the multimodal ophthalmological knowledge graph based on the fundus images; generating ophthalmological auxiliary diagnostic reports by the large language model based on the image features, enhanced visual features, and key knowledge features of the fundus images under the guidance of a prompt template; and updating the parameters of the report generation model based on the similarity difference between the ophthalmological auxiliary diagnostic reports and the diagnostic reports, the contrast loss between image characteristics and report knowledge features. This solution can achieve at least the following beneficial technical effects: In addition to image features and enhanced visual features, this solution utilizes key knowledge features containing knowledge graph information extracted from key entity nodes recalled from multimodal ophthalmology knowledge graphs in fundus images to help large language models generate auxiliary diagnostic reports. This helps improve the professionalism, generalization ability and clinical usability of the auxiliary diagnostic reports generated by the model, and reduces the workload of medical staff.

[0012] Optionally, the multimodal ophthalmology knowledge graph includes an image layer, a medical entity layer, intramodal node relationships, cross-modal node relationships, features of image nodes, and semantic features of entity nodes. The recall method for key entity nodes includes: performing a two-way coarse recall, including: recalling multiple image nodes with high similarity based on the similarity between the image features of the fundus image and the features of the image nodes; determining the neighboring entity nodes of multiple image nodes in the graph based on the cross-modal node relationships; recalling multiple entity nodes with high similarity based on the similarity between the image features of the fundus image and the semantic features of the entity nodes; and obtaining a coarse recall set based on multiple entity nodes and neighboring entity nodes; and performing fine ranking, including: reconstructing the relationships of the nodes in the coarse recall set using the intramodal node relationships of the medical entity layer to obtain a reconstructed subgraph, from which key entity nodes are selected. This scheme achieves at least the following beneficial technical effects: It enables coarse recall using only image features from a dual-path (image layer + entity layer), and entity recall based on different semantics, reducing omissions in coarse recall. Furthermore, at the image layer, although starting from image node features, it is guided by cross-modal node relationships to ultimately obtain relevant neighboring entity nodes within the medical entity layer, facilitating knowledge feature extraction based on the graph encoder. Moreover, based on the coarse recall set, it reconstructs relationships using intramodal node relationships within the medical entity layer, obtaining a reconstructed subgraph rich in ophthalmic diagnostic knowledge. From this subgraph, key entity nodes that significantly aid diagnosis are selected, effectively helping the large language model acquire crucial information for generating reports. Therefore, this scheme, combining multimodal ophthalmic knowledge graphs, coarse recall, and fine ranking, can recall key entity nodes highly relevant to the input fundus image and possessing diagnostic decision-making value, providing core reference knowledge for the large language model to generate auxiliary diagnostic reports.

[0013] Optionally, during fine-tuning, a preset number of key entity nodes are selected from the reconstructed subgraph in descending order of node degree. This scheme can achieve at least the following beneficial technical effects: in the reconstructed subgraph, nodes with higher degrees are closely related to other nodes and are often nodes that are more helpful for diagnosis. Therefore, extracting key entity nodes from the reconstructed subgraph in descending order of node degree helps to efficiently provide core reference knowledge for generating auxiliary diagnostic reports for large language models.

[0014] Optionally, the process of determining neighboring entity nodes of multiple image nodes in the atlas based on cross-modal node relationships includes: obtaining neighboring entity nodes directly connected to multiple image nodes based on cross-modal node relationships; and obtaining neighboring entity nodes directly connected to neighboring image nodes of multiple image nodes based on cross-modal node relationships and intra-modal node relationships between image nodes. This scheme can achieve at least the following beneficial technical effects: This scheme not only uses cross-modal node relationships to recall neighboring entity nodes directly associated with image nodes, but also, to avoid situations where a single image node contains few entity nodes, recalls indirectly connected neighboring entity nodes from neighboring image nodes with the same or similar diseases through cross-modal node relationships, further improving the scope of coarse recall, increasing the possibility of recalling neighboring entity nodes with potentially similar relationships, and increasing the accuracy and representativeness of the final obtained key entity nodes.

[0015] Optionally, the multimodal ophthalmology knowledge graph is obtained as follows: Entity nodes contained in a pre-defined medical knowledge base are extracted from diagnostic reports in the ophthalmology diagnostic dataset to obtain a report entity set, wherein the knowledge volume in the ophthalmology diagnostic dataset is smaller than that in the medical knowledge base; connecting edges are added to the entity nodes in the report entity set based on the entity relationships in the pre-defined medical knowledge base to obtain a medical entity layer; fundus images in the ophthalmology diagnostic dataset are treated as image nodes, and intramodal node relationships are added between image nodes with the same pathological features, as well as cross-modal node relationships are added between image nodes and entity nodes with medical associations, to obtain the multimodal ophthalmology knowledge graph. This scheme can achieve at least the following beneficial technical effects: This scheme uses a large-scale medical knowledge base to generate a multimodal ophthalmology knowledge graph, which can improve the knowledge specialization of the multimodal ophthalmology knowledge graph, and at the same time, significantly reduce the construction cost of the multimodal ophthalmology knowledge graph.

[0016] Optionally, the report generation model further includes multiple mapping layers that process image features, enhanced visual features, and key knowledge features based on a fully connected network. The ophthalmology auxiliary diagnosis report is generated based on the image features, enhanced visual features, and key knowledge features projected from the multiple mapping layers. Furthermore, the parameters of the knowledge enhancement layer, graph encoder, multiple mapping layers, and large language model need to be updated with each training iteration. This scheme can achieve at least the following beneficial technical effects: by performing secondary projection processing on various features through multiple mapping layers, it makes them more consistent with the feature input requirements of the large language model and facilitates the large language model's understanding of relevant semantics, thereby improving the accuracy of the report.

[0017] According to a second aspect of the present invention, a method for generating an ophthalmology auxiliary diagnosis report is provided, comprising: acquiring a multimodal ophthalmology knowledge graph and a fundus image of a subject; generating a corresponding ophthalmology auxiliary diagnosis report based on the fundus image (fundus image to be diagnosed) of the subject using a trained report generation model obtained by the method described in the first aspect, comprising: extracting image features from the fundus image of the subject using a visual encoder; processing the image features using a knowledge enhancement layer to generate enhanced visual features after knowledge enhancement; extracting key knowledge features from key entity nodes recalled from the multimodal ophthalmology knowledge graph based on the fundus image of the subject using a graph encoder; and generating the ophthalmology auxiliary diagnosis report based on the image features, enhanced visual features, and key knowledge features of the fundus image of the subject using a large language model under the guidance of a prompt template.

[0018] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the methods described in the first and / or second aspects by executing the executable instructions. Attached Figure Description

[0019] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0020] Figure 1 This is a schematic diagram illustrating the principle of constructing a medical entity layer according to an embodiment of the present invention;

[0021] Figure 2 A schematic diagram illustrating the acquisition of reconstructed subgraphs from a multimodal ophthalmology knowledge graph according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the report generation model according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram illustrating the effect of recalling key entity nodes and generating auxiliary diagnostic reports based on fundus images according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0025] As mentioned in the background section, there is still room for improvement in the professionalism and accuracy of generative models in ophthalmic auxiliary diagnostic tasks. How to integrate professional medical knowledge with the capabilities of large language models to construct a high-quality ophthalmic auxiliary diagnostic report generation system is an important direction for current technological development. In addition to image features and enhanced visual features, the method of this invention utilizes key knowledge features containing ophthalmic knowledge graph information extracted from key entity nodes recalled from a multimodal ophthalmic knowledge graph based on fundus images to assist the large language model in generating auxiliary diagnostic reports. This helps improve the professionalism, generalization ability, and clinical usability of the generated auxiliary diagnostic reports, while reducing the workload of medical personnel.

[0026] The following is an illustrative introduction to the aspects of problem definition, knowledge graph construction, recall method construction, report generation model construction, model training process, and report generation.

[0027] I. Problem Definition

[0028] set up A collection representing fundus images. These represent fundus images 1, 2, ..., n, respectively. This represents a set of medical entities related to fundus images. These represent the entities in the diagnostic reports corresponding to fundus images 1, 2, ..., n (which may contain a single or multiple entities, depending on the number of entities recorded in the report itself). This represents the knowledge graph of the entity layer, where... It is a set of nodes consisting of all medical entities in the training data. It is an edge set representing the relationships between medical entities. These are node features extracted from the text description of medical entities; the visual encoder is... The image encoder is The existing method (EyeGraphGPT) can then be formalized as follows:

[0029]

[0030] ,

[0031]

[0032]

[0033]

[0034] in, Indicates visual characteristics, Representing the features of a knowledge graph, Corresponding to knowledge features related to visual features, This represents the average value of each channel. Represents visual features enhanced by knowledge. This represents a fully connected projection layer that enhances knowledge based on relation-aware contrastive learning. It is a pre-trained large language model. Indicates a prompt word template, It is a diagnostic report generated by the model.

[0035] Although the existing method applies knowledge-enhanced visual features, subsequent studies have shown that while image-based and knowledge-enhanced visual features have improved performance compared to previous techniques, there is still room for improvement in the model's understanding of knowledge.

[0036] II. Knowledge Graph Construction

[0037] Ophthalmology encompasses a broad and complex spectrum of diseases with diverse pathological features and diagnostic criteria. Close pathological correlations exist between different diseases, such as hypertensive retinopathy and diabetic retinopathy, and venous congestive retinopathy and ocular ischemia syndrome. However, high-quality datasets in the field of ophthalmology are currently scarce and often only contain single image modalities, which challenges the generalization ability of deep learning models across different clinical settings. To overcome this challenge, the solution described in this patent uses knowledge graphs as a graph-structured representation of medical information, integrating multimodal data from the ophthalmology field and guiding deep learning models to learn medical knowledge.

[0038] According to one embodiment of the present invention, a multimodal ophthalmology knowledge graph is obtained as follows: Entity nodes contained in a preset medical knowledge base are extracted from diagnostic reports in an ophthalmology diagnostic dataset to obtain a report entity set, wherein the knowledge content in the ophthalmology diagnostic dataset is smaller than that in the medical knowledge base; connecting edges are added to the entity nodes in the report entity set based on the entity relationships in the preset medical knowledge base to obtain a medical entity layer; fundus images in the ophthalmology diagnostic dataset are regarded as image nodes, and intramodal node relationships are added between image nodes with the same pathological features, as well as cross-modal node relationships are added between image nodes and entity nodes with medical associations to obtain the multimodal ophthalmology knowledge graph. The construction of the multimodal ophthalmology knowledge graph consists of two steps: first, a medical entity layer is constructed based on the medical knowledge base, and then an image layer and cross-modal node relationships are constructed.

[0039] 1) Constructing a medical entity layer based on a medical knowledge base

[0040] See Figure 1 The construction process of the medical entity layer includes:

[0041] Step 1: Medical entity recognition and mapping, including recognizing the original entities (highlighted in blue in the figure, such as vitalomacular traction) in each diagnostic report of the ophthalmology diagnostic dataset (such as the DeepEyeNet dataset), and determining the entity nodes in the preset medical knowledge base (such as the Unified Medical Language System UMLS) based on the original entities; this is equivalent to extracting the entity nodes contained in the preset medical knowledge base from each diagnostic report of the ophthalmology diagnostic dataset to obtain the report entity set; Figure 1 The three boxes shown represent partial textual descriptions of diagnostic reports. Since the inventors conducted their tests based on an English dataset, the descriptions are in English. The meanings of the descriptions in the three reports are as follows: First dashed box: OCT examination shows typical vitreomacular traction symptoms, accompanied by a layered macular hole appearance, and contralateral traction visible in the cortical vitreous. Second dashed box: Late fluorescein fundus angiography of the left eye of a 45-year-old male suggests that the most probable diagnosis is candidal endogenous endophthalmitis. Third dashed box: Hollenhorst plaque (also known as yellow cholesterol plaque) is present, causing branch retinal artery occlusion; the patient has a scar from previous panretinal laser photocoagulation.

[0042] Step 2: Query medical expertise, including querying the relationships between entity nodes in the report entity set within the medical knowledge base, to obtain the queried relationships; during this process, ophthalmologists (such as ophthalmologists) can also verify and add or modify some relationships to supplement and correct the queried relationships;

[0043] Step 3: Add edges between medical entity nodes, including adding edges between entity nodes in the report entity set based on the queried relationships, to obtain the medical entity layer (unimodal medical entity knowledge graph).

[0044] After the construction process of the medical entity layer, each fundus image in the current dataset corresponds to a set of medical entities in the report. These entities not only represent the medical pathology information that can be diagnosed from the image, but also cover the relevant symptoms, disease types and their interrelationships.

[0045] For a detailed explanation of the construction process of the medical entity layer, please refer to the inventor's previous proposal: EyeGraphGPT: Knowledge Graph Enhanced Multimodal Large Language Model for Ophthalmic Report Generation, which will not be elaborated here.

[0046] 2) Constructing image layers and cross-modal node relationships

[0047] Further expanding the unimodal medical entity knowledge graph, each fundus image in the ophthalmology diagnostic dataset is abstracted as a node in the knowledge graph, referred to hereafter as an image node. On one hand, based on the entities contained in the diagnostic reports corresponding to the fundus images, the corresponding entity nodes are found in the medical entity layer, and edges are added between them to construct cross-modal node relationships. On the other hand, at the image level, fundus images with similar conditions often exhibit highly similar feature patterns. To capture these similarities, edges are added between fundus image nodes with the same pathological features using disease labeling information, establishing connections and forming intramodal node relationships at the image layer. Finally, a multimodal ophthalmology knowledge graph containing both fundus image nodes and medical entity nodes is constructed. Schematably, this multimodal ophthalmology knowledge graph is represented as follows:

[0048]

[0049]

[0050] in, Represents an image layer. It is a cross-modal edge set (i.e., cross-modal node relationship). This is the set of image nodes corresponding to the fundus image. For the set of entity nodes in the medical entity layer, The disease labels corresponding to the fundus images (such as macular degeneration, uveitis, etc.). This is the set of medical entities corresponding to the fundus image.

[0051] By leveraging a systematic medical knowledge base and employing graph representations of entities and their relationships, medical information is semantically integrated to form a multimodal ophthalmology knowledge graph. This provides the model with rich semantic context and reasoning capabilities. In the process of intelligent assisted diagnosis of eye diseases, the multimodal ophthalmology knowledge graph not only helps the model understand complex medical concepts but also reveals potential relationships between different medical entities, thereby enhancing the professionalism and accuracy of subsequent report generation. Therefore, the ophthalmology knowledge graph can be integrated into a multimodal large language model to enrich training data and guide the model in learning medical knowledge, ultimately improving the quality of report generation and diagnostic capabilities.

[0052] It should be understood that, in some cases, implementers may also choose not to construct the intramodal node relationships of the image layer, thus obtaining another implementation method.

[0053] III. Construction of Recall Methods

[0054] According to one embodiment of the present invention, the recall method for key entity nodes includes: performing a dual-path coarse recall, including: recalling multiple image nodes with high similarity based on the similarity between image features of fundus images and features of image nodes; determining neighboring entity nodes of multiple image nodes in the knowledge graph based on cross-modal node relationships; recalling multiple entity nodes with high similarity based on the similarity between image features of fundus images and semantic features of entity nodes; and obtaining a coarse recall set based on multiple entity nodes and neighboring entity nodes; and performing fine ranking, including: reconstructing the relationships between nodes in the coarse recall set using intra-modal node relationships of the medical entity layer to obtain a reconstructed subgraph; and selecting a preset number of key entity nodes from the reconstructed subgraph in descending order of node degree. A multimodal ophthalmological knowledge graph can provide the model with rich cross-modal retrieval capabilities. Illustratively, for a fundus image to be diagnosed, the feature similarity between its features and those of all image nodes in the knowledge graph is first calculated to identify a set of images with similar visual features. Subsequently, by leveraging the pre-established cross-modal connectivity information in the multimodal knowledge graph, a graph search algorithm is used to recall medical entity nodes related to the image to be diagnosed, thereby providing comprehensive and accurate pathological knowledge references for clinical auxiliary diagnosis.

[0055] According to one embodiment of the present invention, the features of image nodes in a multimodal ophthalmology knowledge graph are pre-extracted using a visual encoder and stored in a database for later retrieval, while the semantic features of entity nodes are pre-extracted using the vectorization module of a large language model and stored in a database for later retrieval. This scheme achieves at least the following beneficial technical effects: it avoids the need for significant computational overhead during training, thereby improving training efficiency.

[0056] According to an optional embodiment of the present invention, the process of determining the neighbor entity nodes of multiple image nodes in the graph based on cross-modal node relationships includes: obtaining the neighbor entity nodes connected to the multiple image nodes only based on cross-modal node relationships.

[0057] According to another optional embodiment of the present invention, the process of determining the neighbor entity nodes of multiple image nodes in the atlas based on cross-modal node relationships includes: obtaining neighbor entity nodes connected to the multiple image nodes based on cross-modal node relationships; and obtaining neighbor entity nodes directly connected to the neighbor image nodes of the multiple image nodes based on cross-modal node relationships and intra-modal node relationships between image nodes.

[0058] According to one embodiment of the present invention, during dual-path coarse recall, the HNSW algorithm is used to recall multiple image nodes and multiple entity nodes most relevant to image features at the image layer and medical entity layer, respectively. This scheme can achieve at least the following beneficial technical effects: the HNSW algorithm can efficiently perform dual-path retrieval, reduce the time overhead of dual-path recall, and improve training efficiency. Illustratively, a graph retrieval enhancement algorithm combining multimodal graph retrieval and graph knowledge injection is proposed to address the challenges of multimodal fusion features being unable to express fine-grained entity information and graph structures, low cross-modal retrieval accuracy in small sample scenarios, and poor interpretability of model output. Specifically, the multimodal ophthalmology knowledge graph includes two layers: a fundus image layer and a medical entity layer, used to represent the visual features of fundus images and the textual features of medical entities, respectively. Through original data annotation and data preprocessing, cross-modal associations between fundus images and medical entities are explicitly established to compensate for the large semantic gap between image features and textual features in small medical sample scenarios, thereby achieving more accurate cross-modal information retrieval.

[0059] The multimodal graph retrieval algorithm first applies hierarchical navigable small-world (HNSW) graph indexing technology to index the feature space of image nodes. The semantic feature space of entity nodes (medical entity text feature space). Building an efficient index structure and To achieve the effect of eigenvectors Efficient nearest neighbor retrieval. During the retrieval phase, a dual-channel parallel retrieval strategy is employed to recall visually similar node sets from the image layer. Recall semantically relevant entity sets from the medical entity layer And through explicit cross-modal association edge sets (cross-modal node relationships). Expand the nodes to form a preliminary recall set (coarse recall set). To enhance the model's perception of the ophthalmology knowledge graph structure, the algorithm further performs subgraph reconstruction operations based on the entity layer knowledge graph. The connection information, for By establishing connections between nodes, a reconstructed subgraph with domain knowledge relationships is generated. And key nodes were selected based on node connectivity. Each medical entity node forms a key entity set, which has diagnostic decision-making value. These are medical entities highly correlated with the fundus image to be diagnosed, providing diagnostic references for the large language model. An illustrative reconstructed sub-image is shown below. Figure 2As shown. Finally, through the graph knowledge injection module (graph encoder and related mapping layer), the retrieval results are integrated into the large language model, further improving the accuracy and interpretability of diagnostic decisions. This algorithm effectively solves the challenges of difficult representation of fine-grained graph information, low cross-modal retrieval accuracy in small sample scenarios, and poor interpretability of model output through multimodal graph retrieval and graph knowledge injection.

[0060] In vector retrieval, index construction is one of the key technologies for improving query efficiency. With the continuous increase in data dimensionality and scale, traditional brute-force search methods can no longer meet the requirements of real-time performance and scalability. By constructing an efficient index structure, the computational complexity of the retrieval algorithm can be significantly reduced, enabling efficient retrieval of large-scale vector data. This example uses the HNSW algorithm for vector indexing and retrieval. This algorithm, by constructing a hierarchical navigation small-world graph structure and combining it with a greedy search strategy, can provide high retrieval accuracy while maintaining low computational complexity, thereby significantly improving vector retrieval efficiency.

[0061] The HNSW algorithm's indexing process employs a multi-layered graph structure to organize high-dimensional data, enabling efficient approximate nearest neighbor search. Each data point to be inserted is randomly assigned a maximum level, and insertion proceeds layer by layer downwards from the top of the graph. In higher levels, the algorithm uses a greedy strategy to find the closest candidate node to the current node, with a relatively small search range to ensure insertion efficiency. Upon entering lower levels, the search range expands, and the selected nearest neighbor nodes are connected to the node to be inserted. Through this hierarchical insertion and connection mechanism, the HNSW algorithm constructs a multi-layered navigation graph structure from sparse high-level to dense low-level. This structure possesses excellent global search capabilities and local connection optimization capabilities, providing a foundation for subsequent fast retrieval.

[0062] During the query phase, the HNSW algorithm starts from a node at the top level and uses a greedy strategy to search downwards layer by layer for the node most similar to the query vector. At each level, the algorithm dynamically maintains the candidate set, the visited set, and the best nearest neighbor set, expanding and updating the current candidate node and its neighborhood to narrow the search range and improve accuracy. After completing each level, the algorithm uses the best node found as the starting point for the next level, and finally selects the nodes closest to the query vector at the bottom level as the final result. By performing fast coarse screening at the top level and fine optimization at the bottom level, the HNSW algorithm significantly improves retrieval efficiency while maintaining accuracy, making it suitable for high-performance approximate retrieval tasks on large-scale datasets.

[0063] For the input image features Let the set of image layer nodes be... The feature (vector) index of the image layer is The set of medical entity nodes is The semantic feature (vector) index of the medical entity layer is This example employs a two-stage recall strategy. First, a coarse ranking is performed based on feature similarity to initially recall medical entity nodes that are relatively relevant to the image to be diagnosed. Then, fine ranking is performed through subgraph reconstruction and graph structure awareness to recall key diagnostic nodes. This is illustrative of a given image feature... Define it with candidate set nodes The feature similarity between them is their cosine similarity:

[0064]

[0065] The first stage of vector retrieval is a dual-path parallel retrieval (coarse recall), using the HNSW algorithm to recall input image features at the image layer and the medical entity layer, respectively. Most relevant Nodes:

[0066]

[0067]

[0068] in, For the recall node set of the image layer, This represents the recall node set for the entity layer. For the image layer's recall node set, the set of associated neighboring entity nodes is determined by querying cross-modal node relationships in the multimodal knowledge graph.

[0069]

[0070] in, This is the first stage of the recall node set, i.e., the coarse recall set.

[0071] To enhance the model's perception of graph structure and simultaneously recall key medical entity nodes, a fine-ranking strategy based on subgraph structure was designed. Utilizing the intra-modal node relationships in the medical entity layer, the coarse recall set is used as input for subgraph reconstruction. Then, a fine-ranking is performed based on the degree of the nodes in the subgraph (higher degree indicates stronger connectivity and higher medical semantic centrality), thus recalling key medical entities. These medical entities are used as the output of the graph retrieval algorithm to obtain key entity nodes:

[0072]

[0073]

[0074]

[0075] in, This represents the initial edge set of the medical entity layer in the knowledge graph, i.e., the intramodal node relationships in the medical entity layer; This represents the final recall result of the graph retrieval algorithm, i.e., the key entity nodes. A schematic diagram of recalling key entity nodes based on fundus images is shown below. Figure 4 As shown, the words highlighted in red in the last column are the key entity nodes for recall.

[0076] Enhancing the quality of retrieved content is crucial for subsequent generation tasks in large language models. High-quality retrieved content provides accurate and rich information support, thereby improving the reliability of the generated results and the rationality of the medical context. Noise or bias in the retrieved content can lead to incorrect model responses or even "hallucinations," impacting the performance of downstream tasks. In the DeepEyeNet dataset, due to the pathological complexity of ophthalmic diseases and the lack of high-quality image-text pairing data, the semantic correlation between visual and text features is weak. This challenge results in low recall rates for single-entity-layer cross-modal retrieval algorithms based on feature similarity in cross-modal retrieval tasks, while also introducing irrelevant or noisy information, ultimately affecting the generation quality. Therefore, the method in this invention leverages multimodal knowledge graphs to fully explore the explicit associations between fundus images and medical entities, improving the accuracy of cross-modal retrieval to some extent. However, this method may still miss some medical entities highly relevant to the image to be diagnosed. The hybrid multimodal graph retrieval algorithm combining entity layer and image layer integrates a single entity layer cross-modal retrieval algorithm with an image layer multimodal graph retrieval algorithm. Furthermore, by leveraging the coarse recall + fine ranking strategy of subgraph reconstruction and graph structure awareness proposed in this invention, the algorithm can effectively recall more key medical entity nodes, reduce interference from irrelevant information, thereby further improving the performance of cross-modal retrieval and ultimately enhancing the professionalism in the task of generating auxiliary diagnostic reports for eye diseases.

[0077] IV. Report Generation Model Construction

[0078] According to one embodiment of the present invention, a report generation model is constructed, see [link to relevant documentation]. Figure 3 It includes: a visual encoder, a knowledge enhancement layer, a graph encoder, multiple mapping layers (first to third mapping layers), and a large language model. Among them:

[0079] A visual encoder is configured to extract image features from an input fundus image. The visual encoder is an image convolutional neural network.

[0080] The knowledge enhancement layer is configured to process image features and generate enhanced visual features after knowledge enhancement. .

[0081] The graph encoder is configured to extract report knowledge features from entity nodes contained in diagnostic reports corresponding to fundus images during training; and to extract key knowledge features from key entity nodes recalled from fundus images based on a multimodal ophthalmology knowledge graph during training and / or during assisted diagnosis. Graph encoders are graph neural networks, such as graph attention neural networks.

[0082] The first mapping layer, a fully connected network, is configured to map the input image features. The image features are obtained by performing a mapping process.

[0083] The second mapping layer, a fully connected network, is configured to enhance visual features. The mapping process is performed to obtain the mapped enhanced visual features.

[0084] The third mapping layer, a fully connected network, is configured to map key knowledge features. A mapping process is performed to obtain the mapped key knowledge features.

[0085] The large language model is configured to generate ophthalmic auxiliary diagnostic reports based on mapped image features, mapped enhanced visual features, and mapped key knowledge features of fundus images, guided by prompt templates.

[0086] It should be understood that alternative implementations can be obtained without setting up separate mapping layers. For example, the functions of each mapping layer can be integrated into the corresponding encoder. For instance, the first mapping layer can be integrated into the visual encoder, the second mapping layer into the knowledge enhancement layer, and the third mapping layer into the graph encoder. Thus, the large language model, guided by prompt templates, generates ophthalmological auxiliary diagnostic reports based on the image features, enhanced visual features, and key knowledge features of the original fundus image.

[0087] V. Model Training Process

[0088] According to one embodiment of the present invention, the report generation model is trained iteratively multiple times using a training set. Each training iteration includes: extracting image features from the fundus images of the samples using a visual encoder; processing the image features using a knowledge enhancement layer to generate enhanced visual features after knowledge enhancement; extracting report knowledge features from entity nodes in the diagnostic report corresponding to the fundus image using a graph encoder, and extracting key knowledge features from key entity nodes recalled from the multimodal ophthalmology knowledge graph based on the fundus image; generating an ophthalmology auxiliary diagnosis report by a large language model based on the image features, enhanced visual features, and key knowledge features of the fundus image under the guidance of a prompt template; and updating the parameters of the report generation model based on the similarity difference between the ophthalmology auxiliary diagnosis report and the diagnostic report, and the contrast loss between image characteristics and report knowledge features. Preferably, the training set can be divided into multiple batches, and the model can be iteratively trained using samples from different batches. During training, the parameters of the large language model (such as the llama-2-7b model or the Qwen model) can be directly updated. However, to reduce the number of parameters that need to be updated during training, a LoRA fine-tuning mechanism can be used to set a LoRA layer, and the LoRA layer can be used for fine-tuning updates instead of updating the original parameters of the large language model. Furthermore, the visual encoder can be updated during training, or a pre-trained visual encoder can be used; however, the visual encoder parameters are not updated during training in this invention. Thus, knowledge augmentation of the large language model is achieved by leveraging recalled key entity nodes. The node features of the key entity nodes are modally injected using a lightweight fully connected projection layer. Simultaneously, task-specific prompt word templates are applied, and the large language model is fine-tuned using the LLaMA-Adapter parameter efficiency fine-tuning technique. Task-specific parameters are injected only into the LoRA layer, thereby achieving model adaptation and optimization for diagnostic language styles and key elements without significantly increasing computational overhead, making the final content more focused on key diagnostic information.

[0089] According to one embodiment of the present invention, the total loss function during training is set as follows:

[0090]

[0091] in, This is the first sub-loss function used to calculate the similarity difference between ophthalmological auxiliary diagnostic reports and diagnostic reports. This is the second sub-loss function used to calculate the contrast loss between image features and reported knowledge features. , This represents the weighting coefficient, which is a hyperparameter that can be set and adjusted according to the implementer's needs or on-site experience.

[0092] VI. Report Generation

[0093] According to one embodiment of the present invention, an ophthalmology auxiliary diagnosis report generation method includes: acquiring a multimodal ophthalmology knowledge graph and a fundus image of a test subject; generating a corresponding ophthalmology auxiliary diagnosis report based on the fundus image of the test subject using a trained report generation model obtained by the training method of the aforementioned embodiment, including: extracting image features from the fundus image of the test subject using a visual encoder; processing the image features using a knowledge enhancement layer to generate enhanced visual features after knowledge enhancement; extracting key knowledge features from key entity nodes recalled from the multimodal ophthalmology knowledge graph based on the fundus image of the test subject using a graph encoder; and generating the ophthalmology auxiliary diagnosis report based on the image features, enhanced visual features, and key knowledge features of the fundus image of the test subject under the guidance of a prompt template. By combining medical expert experience with a structured medical relation database (medical knowledge base), a professional knowledge graph for the ophthalmology field is constructed to characterize the complex relationships between disease entities, lesion manifestations, and diagnostic terms. Through a relation-aware modal alignment mechanism, the alignment of key regions in the image with semantic nodes in the knowledge graph is achieved, and structured knowledge features are dynamically injected into the language generation module to enhance the model's understanding and expression capabilities of medical semantics. Based on a structured ophthalmology knowledge graph, this method integrates professional medical entities such as diseases, symptoms, and signs, along with their semantic relationships, as knowledge intermediaries for model generation. Through a graph retrieval enhancement mechanism, it actively retrieves corresponding multi-granularity medical knowledge fragments based on visual features in fundus images, establishing an explicit mapping between image regions and knowledge graph nodes. This enhances the model's ability to identify key features and invoke knowledge during the diagnostic generation process. Subsequently, leveraging the language organization and expression capabilities of a large language model, it combines image features, retrieved knowledge content, and report structure templates to generate professional diagnostic text that conforms to clinical standards. Simultaneously, a visually guided collaborative fine-tuning strategy is introduced to maintain semantic consistency between the image and text during report generation, improving the professionalism, accuracy, and logical clarity of the generated text. This method effectively improves the model's professionalism, generalization ability, and clinical usability while alleviating dependence on training data, making it suitable for constructing highly reliable ophthalmology auxiliary diagnostic systems.

[0094] According to one embodiment of the present invention, the prompt word template includes a role setting for the large language model, a task description, and reference information for the input (image features, enhanced visual features, and key knowledge features; or mapped image features, mapped enhanced visual features, and mapped key knowledge features). The role setting explicitly informs the large language model of its task role (ophthalmology AI assistant). The task description aims to enable the large language model to generate an ophthalmology auxiliary diagnostic text description (auxiliary diagnostic report) based on relevant input. Additionally, the reference information may also include a set of key entity nodes.

[0095] As an example, during the language generation stage, the following task-specific prompt word templates are used to guide the model in generating structured and professional auxiliary diagnostic reports.

[0096] Human: . You are an AI assistant specialized inophthalmology. Please give a brief diagnosis report for this fundus image. You can refer the following knowledge: . And the possibly related knowledge graph entities are Assistant:”

[0097] The above illustrative prompt template specifies the input image features, enhanced visual features, and knowledge entity information of the relevant key entity nodes recalled. In the prompt, the model receives information including visual features. Enhance visual features Key knowledge features of the mapping The system uses three types of inputs (e.g., average pooling of key entity node features retrieved from the atlas) and activates its medical understanding capabilities through instruction prompts. This strategy significantly improves the focus and professionalism of the report content, making the generated results more in line with the actual needs of clinical auxiliary diagnosis reports. Furthermore, it can guide large language models, requiring them to include symptom analysis, disease associations, and differential diagnoses in the generated auxiliary diagnosis reports.

[0098] It should be noted that the auxiliary diagnostic report generated by the method of this invention is not the final medical diagnosis result. It serves only as reference information for medical personnel and can only be used as the final diagnosis result after modification and / or confirmation by medical personnel. Furthermore, when presenting the auxiliary diagnostic report, the reconstructed subgraph and / or information on key entity nodes can also be shown to medical personnel, enabling them to understand the thought process and key information presented in the report, thus facilitating the identification and correction of obvious errors. This is equivalent to introducing an interpretive path annotation and interactive feedback mechanism, making the generated content traceable and interpretable, thereby enhancing the system's trustworthiness and usability in actual clinical practice.

[0099] See you again Figure 4 It also demonstrates a comparison between manually generated and generated auxiliary diagnostic reports for eye diseases before and after the introduction of a multimodal graph retrieval enhancement mechanism, with successfully matched medical entities marked by underlining. Figure 4In multiple cases, the enhanced model was able to recall key medical entities through the multimodal graph retrieval algorithm, generating more accurate diagnostic descriptions that significantly outperformed the generalized representations produced by the initial model before retrieval enhancement. Furthermore, the recalled entities listed in the table demonstrate that the multimodal graph retrieval mechanism effectively extracts medical knowledge highly relevant to the image to be diagnosed, providing richer knowledge support for the report generation model. This further validates the effectiveness of the multimodal graph retrieval enhancement and graph knowledge injection strategy in the task of generating auxiliary diagnostic reports for eye diseases.

[0100] In summary, the embodiments of the present invention can achieve at least the following technical effects:

[0101] 1. Graph Retrieval Enhancement and Graph-Text Knowledge Alignment Mechanism. This invention proposes a knowledge retrieval enhancement method based on image features. By explicitly aligning image semantics with entities in the knowledge graph, a multi-level mapping relationship between graph and knowledge is constructed. This improves the model's ability to locate key entities and organize diagnostic content in scenarios involving multiple diseases, multiple lesions, or rare diseases, thereby enhancing the model's generalization performance and detail generation capabilities.

[0102] 2. Medical Knowledge Graph Construction and Injection Mechanism. This invention constructs a structured medical knowledge graph for ophthalmic diseases, covering entities such as disease types, lesion manifestations, anatomical locations, and diagnostic terms, as well as their relationships. During the report generation process, the knowledge graph is organically injected into the language modeling path to achieve knowledge constraints and reasoning support for diagnostic semantics, effectively improving the professionalism and accuracy of the generated reports and reducing the generation of "illusionary" content.

[0103] 3. A method for generating auxiliary diagnostic reports driven by a large language model. This invention constructs a specialized text generation module based on a large language model, and combines medical knowledge constraints and retrieval results to automatically generate eye disease diagnostic reports that are structurally complete, logically clear, and semantically accurate.

[0104] 4. To address the problems of insufficient medical knowledge, unclear diagnostic criteria, unstable generated content, and lack of interpretability in existing ophthalmic disease auxiliary diagnosis report generation, a multimodal collaborative generation method integrating medical knowledge graphs, graph retrieval enhancement mechanisms, and large language models is proposed. By constructing a structured medical knowledge graph and guiding the model to acquire pathological entity semantics, accurate retrieval and semantic alignment are achieved by combining image features. Furthermore, a large language model is used as the core to generate professional ophthalmic disease auxiliary diagnosis reports, significantly improving the medical credibility, content accuracy, and clinical applicability of the reports. It possesses good generalization ability and application potential, making it suitable for building a next-generation, highly reliable intelligent ophthalmic auxiliary diagnostic platform.

[0105] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0106] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0107] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0108] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a model that generates ophthalmological auxiliary diagnostic reports, comprising: A report generation model is constructed, which includes: a visual encoder, a knowledge enhancement layer, a graph encoder, and a large language model; Obtain a training set, which includes multiple samples, including fundus images and their diagnostic reports; The report generation model is trained iteratively using the training set, with each training iteration including: Image features were extracted from fundus images of the samples using a visual encoder; The knowledge enhancement layer is used to process image features and generate enhanced visual features after knowledge enhancement. The graph encoder is used to extract report knowledge features from entity nodes in the diagnostic reports corresponding to fundus images, and to extract key knowledge features from key entity nodes recalled from fundus images based on multimodal ophthalmology knowledge graphs. Guided by the prompt template, the large language model generates an ophthalmology auxiliary diagnosis report based on the image features, enhanced visual features, and key knowledge features of the fundus image; The parameters of the report generation model are updated based on the similarity differences between ophthalmic auxiliary diagnostic reports and diagnostic reports, the contrast loss between image characteristics and report knowledge features.

2. The method according to claim 1, characterized in that, The multimodal ophthalmology knowledge graph includes an image layer, a medical entity layer, intramodal node relationships, cross-modal node relationships, features of image nodes, and semantic features of entity nodes. The recall methods for critical entity nodes include: A dual-path coarse recall is performed, including: recalling multiple image nodes with high similarity based on the similarity of image features and image node features in the fundus image; determining neighboring entity nodes of multiple image nodes in the graph based on cross-modal node relationships; recalling multiple entity nodes with high similarity based on the similarity of image features and semantic features of entity nodes in the fundus image; and obtaining a coarse recall set based on multiple entity nodes and neighboring entity nodes; and Fine ranking is performed, including: using the intramodal node relationships of the medical entity layer to reconstruct the relationships of the nodes in the coarse recall set to obtain a reconstructed subgraph, from which key entity nodes are selected.

3. The method according to claim 2, characterized in that, During fine sorting, a preset number of key entity nodes are selected from the reconstructed subgraph in descending order of node degree.

4. The method according to claim 2, characterized in that, The process of determining the neighboring entity nodes of multiple image nodes in the graph based on cross-modal node relationships includes: Based on cross-modal node relationships, obtain the neighboring entity nodes connected to multiple image nodes; and Based on cross-modal node relationships and intramodal node relationships between image nodes, obtain the neighboring entity nodes that are directly connected to the neighboring image nodes of multiple image nodes.

5. The method according to any one of claims 1-4, characterized in that, The multimodal ophthalmology knowledge graph was obtained in the following ways: Extract entity nodes contained in a pre-defined medical knowledge base from the diagnostic reports in the ophthalmology diagnostic dataset to obtain the report entity set. The amount of knowledge in the ophthalmology diagnostic dataset is smaller than that in the medical knowledge base. Based on the relationships between entities in the pre-defined medical knowledge base, connection edges are added to the entity nodes in the report entity set to obtain the medical entity layer; By treating fundus images in the ophthalmology diagnostic dataset as image nodes, and adding intramodal node relationships between image nodes with the same pathological features, as well as cross-modal node relationships between image nodes and entity nodes with medical associations, a multimodal ophthalmology knowledge graph is obtained.

6. The method according to any one of claims 1-4, characterized in that, The report generation model also includes: Multiple mapping layers based on a fully connected network process image features, enhanced visual features, and key knowledge features. The ophthalmology auxiliary diagnosis report is generated based on image features, enhanced visual features, and key knowledge features projected from multiple mapping layers. Furthermore, the parameters of the knowledge enhancement layer, graph encoder, multiple mapping layers, and large language model need to be updated with each training iteration.

7. A method for generating an ophthalmological auxiliary diagnostic report, characterized in that, include: Acquire a multimodal ophthalmology knowledge graph and fundus images of the subjects being tested; The report generation model trained using the method described in any one of claims 1-6 generates a corresponding ophthalmic auxiliary diagnosis report based on the fundus image of the detected object, including: Image features are extracted from fundus images of the detected object using a visual encoder; The knowledge enhancement layer is used to process image features and generate enhanced visual features after knowledge enhancement. A graph encoder is used to extract key knowledge features from key entity nodes recalled from fundus images of the detected object in a multimodal ophthalmology knowledge graph. Guided by the prompt template, the large language model generates an ophthalmological auxiliary diagnosis report based on the image features, enhanced visual features, and key knowledge features of the fundus image of the detected object.

8. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-7 by executing the executable instructions.

Images