Fundus image prediction method, fundus image prediction system, device, and storage medium

Abstract

The application provides an eye fundus image prediction method, an eye fundus image prediction system, a device and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining a target text area image of a target eye fundus detection image to perform text recognition, and obtaining target detection text; inputting the target eye fundus area image and the target detection text into an eye fundus image prediction model; performing entity data recognition on the target detection text to obtain target entity data and target entity relationship data; performing graph construction on the target eye fundus area image, the target entity data and the target entity relationship data to obtain a target to-be-matched subgraph; performing graph matching on the target to-be-matched subgraph according to a pre-constructed eye fundus knowledge graph to determine an eye fundus matched subgraph; and determining a target eye fundus detection type of the target eye fundus detection image according to the eye fundus matched subgraph. The embodiment of the application can improve the prediction efficiency and prediction accuracy of the eye fundus image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an eye fundus image prediction method, an eye fundus image prediction system, a device and a storage medium. BACKGROUND

[0002] At present, in order to screen out abnormal types of eye fundus images, type prediction is performed on the eye fundus images to help the target object to perform medical intervention in time in the early stage of the onset of the eye disease. The eye fundus image prediction method in the related art usually screens a single disease type, and the prediction efficiency is low and the prediction accuracy cannot meet the current demand. Therefore, how to provide an eye fundus image prediction method capable of improving the prediction efficiency and prediction accuracy of the eye fundus image has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide an eye fundus image prediction method, an eye fundus image prediction system, a device and a storage medium, which can effectively improve the prediction efficiency and prediction accuracy of the eye fundus image.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides an eye fundus image prediction method, which comprises:

[0005] obtaining a target eye fundus detection image, the target eye fundus detection image comprising a target eye fundus region image and a target text region image;

[0006] performing text recognition on the target text region image to obtain a target detection text;

[0007] inputting the target eye fundus region image and the target detection text into a pre-trained eye fundus image prediction model, the eye fundus image prediction model comprising an entity relationship sub-model, a graph construction sub-model and an eye fundus type prediction sub-model;

[0008] performing entity data recognition on the target detection text according to the entity relationship sub-model to obtain target entity data and target entity relationship data;

[0009] performing graph construction on the target eye fundus region image, the target entity data and the target entity relationship data according to the graph construction sub-model to obtain a target to-be-matched subgraph;

[0010] performing graph matching on the target to-be-matched subgraph according to the eye fundus type prediction sub-model and a pre-constructed eye fundus knowledge graph to determine an eye fundus matching subgraph;

[0011] determining a target fundus detection type of the target fundus detection image according to the fundus matching subgraph. In some embodiments, before the inputting the target fundus region image and the target detection text into the pre-trained fundus image prediction model, the method further comprises:

[0012] training the fundus image prediction model, specifically comprising:

[0013] obtaining a first fundus region image, a first detection text of the first fundus region image, and a first fundus detection type of the first fundus region image;

[0014] inputting the first fundus region image and the first detection text into a preset initial image prediction model, the initial image prediction model comprising the entity relationship submodel, the graph construction submodel, and the fundus type prediction submodel;

[0015] performing entity data recognition on the first detection text according to the entity relationship submodel to obtain first entity data and first entity relationship data;

[0016] performing graph construction on the first fundus region image, the first entity data, and the first entity relationship data according to the graph construction submodel to obtain an initial to-be-matched subgraph;

[0017] performing graph matching on the initial to-be-matched subgraph according to the fundus type prediction submodel and a pre-constructed fundus knowledge graph to obtain a sample matching subgraph, and determining a predicted fundus detection type according to the sample matching subgraph;

[0018] performing parameter adjustment on the initial image prediction model according to the first fundus detection type and the predicted fundus detection type to obtain the fundus image prediction model.

[0019] In some embodiments, before the performing graph matching on the target to-be-matched subgraph according to the fundus type prediction submodel and the pre-constructed fundus knowledge graph to determine a fundus matching subgraph, the method further comprises:

[0020] constructing the fundus knowledge graph, specifically comprising:

[0021] obtaining a second fundus region image, a second detection text of the second fundus region image, and a second fundus detection type of the second fundus region image;

[0022] inputting the second detection text into a preset entity extraction model to extract second entity data and second entity relationship data;

[0023] data integration is performed according to the second fundus region image, the second entity data and the second entity relationship data, to obtain sample fundus triadic data;

[0024] According to the second fundus detection type and the sample fundus triadic data, a graph is constructed to obtain an initial knowledge graph;

[0025] The initial knowledge graph is input into a preset graph reasoning model for graph completion processing to obtain the fundus knowledge graph.

[0026] In some embodiments, the graph matching of the target to-be-matched subgraph according to the fundus type prediction submodel and the pre-constructed fundus knowledge graph to determine the fundus matching subgraph comprises:

[0027] The entity data of the fundus knowledge graph is extracted to obtain candidate entity data;

[0028] The entity similarity data is calculated by performing entity similarity calculation on the target entity data and the candidate entity data, and the predicted entity data is determined according to the entity similarity data and a preset entity threshold;

[0029] According to the predicted entity data, the fundus knowledge graph is segmented to obtain a candidate subgraph;

[0030] According to the target fundus region image, the candidate subgraph is screened to obtain the fundus matching subgraph.

[0031] In some embodiments, the screening of the candidate subgraph according to the target fundus region image to obtain the fundus matching subgraph comprises:

[0032] The image similarity data is calculated by performing image similarity calculation on the target fundus region image and the candidate fundus image in the candidate subgraph;

[0033] According to the image similarity data, the candidate subgraph is screened to obtain the fundus matching subgraph.

[0034] In some embodiments, the entity data recognition of the target detection text according to the entity relationship submodel comprises:

[0035] The target detection text is text-labeled to obtain fundus detection words, text-labeled types of the fundus detection words and position information of the fundus detection words;

[0036] The fundus detection words are screened according to a preset entity type and the text-labeled types to obtain the target entity data;

[0037] perform entity relation extraction on the target entity data according to the position information, to obtain target entity relation data.

[0038] In some embodiments, the obtaining of the first fundus region image, the first detection text of the first fundus region image, and the first fundus detection type of the first fundus region image comprises:

[0039] obtaining an initial detection image, and performing image contour recognition on the initial detection image to obtain a fundus detection region image;

[0040] performing image blur detection on the fundus detection region image to obtain a blur detection result;

[0041] performing image screening on the initial detection image according to the blur detection result to determine a candidate detection image;

[0042] performing image division on the candidate detection image to obtain the first fundus region image and a sample text region image;

[0043] performing fundus detection type marking on the first fundus region image to obtain the first fundus detection type;

[0044] performing text recognition on the sample text region image to obtain the first detection text.

[0045] To achieve the above-mentioned purpose, a second aspect of the embodiment of the present application proposes an eye fundus image prediction system, the system comprises:

[0046] an image acquisition module, configured to acquire a target eye fundus detection image, the target eye fundus detection image comprising a target eye fundus region image and a target text region image;

[0047] a text recognition module, configured to perform text recognition on the target text region image to obtain target detection text;

[0048] a model input module, configured to input the target eye fundus region image and the target detection text into a pre-trained eye fundus image prediction model, the eye fundus image prediction model comprising an entity relation sub-model, a graph construction sub-model, and an eye fundus type prediction sub-model;

[0049] an entity data recognition module, configured to perform entity data recognition on the target detection text according to the entity relation sub-model to obtain target entity data and target entity relation data;

[0050] a graph construction module, configured to perform graph construction on the target eye fundus region image, the target entity data, and the target entity relation data according to the graph construction sub-model to obtain a target to-be-matched subgraph;

[0051] a graph matching module, configured to perform graph matching on the target subgraph to be matched according to the fundus type prediction submodel and a pre-constructed fundus knowledge graph, and determine a fundus matching subgraph;

[0052] an image prediction module, configured to determine a target fundus detection type of the target fundus detection image according to the fundus matching subgraph.

[0053] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of the first aspect of the embodiments of the present application when executing the computer program.

[0054] To achieve the above object, a fourth aspect of the embodiments of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method according to any one of the first aspect of the embodiments of the present application.

[0055] The fundus image prediction method, fundus image prediction system, device and storage medium provided by the embodiments of the present application first acquire a target fundus detection image, and perform image region recognition on the target fundus detection image to obtain a target fundus region image and a target text region image. Text recognition is performed on the target text region image to obtain a target detection text. Then, the target fundus region image and the target detection text are input into a pre-trained fundus image prediction model, which comprises an entity relationship submodel, a graph construction submodel and a fundus type prediction submodel. The entity relationship submodel is used to perform entity data recognition on the target detection text to obtain target entity data and target entity relationship data. Then, the graph construction submodel is used to perform graph construction on the target fundus region image, the target entity data and the target entity relationship data to obtain a target subgraph to be matched. The fundus type prediction submodel and a pre-constructed fundus knowledge graph are used to perform graph matching on the target subgraph to be matched to determine a fundus matching subgraph. Finally, the fundus matching subgraph is used to determine a target fundus detection type of the target fundus detection image. The embodiments of the present application can effectively improve the prediction efficiency and prediction accuracy of the fundus image by introducing a knowledge graph to predict the type of the fundus image. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a first flowchart of the fundus image prediction method provided by the embodiments of the present application;

[0057] Figure 2 is a structural schematic diagram of a target fundus detection image provided by the embodiments of the present application;

[0058] Figure 3 is a second flowchart of the fundus image prediction method provided by the embodiment of the present application;

[0059] Figure 4 is Figure 3 is a flowchart of the specific method of step S310 in the embodiment of the present application;

[0060] Figure 5 is Figure 1 is a flowchart of the specific method of step S140 in the embodiment of the present application;

[0061] Figure 6 is a structural schematic diagram of the target subgraph to be matched provided by the embodiment of the present application;

[0062] Figure 7 is a third flowchart of the fundus image prediction method provided by the embodiment of the present application;

[0063] Figure 8 is Figure 1 is a flowchart of the specific method of step S160 in the embodiment of the present application;

[0064] Figure 9 is Figure 8 is a flowchart of the specific method of step S840 in the embodiment of the present application;

[0065] Figure 10 is a module structure block diagram of the fundus image prediction system provided by the embodiment of the present application;

[0066] Figure 11 is a hardware structure schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0068] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0070] Firstly, some terms involved in the present application are analyzed:

[0071] Artificial Intelligence (AI): It is a new technical science of researching and developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; Artificial intelligence is a branch of computer science, and artificial intelligence tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] At present, in order to screen out abnormal types of fundus images, type prediction is performed on the fundus images to help the target object to perform medical intervention in time when the eye disease is not ill or in the early stage of the disease. The fundus image prediction method in the related art usually screens a single disease type, and the prediction efficiency is low and the prediction accuracy cannot meet the current demand. Therefore, how to provide a fundus image prediction method capable of improving the prediction efficiency and prediction accuracy of the fundus image has become a technical problem to be solved.

[0073] Based on this, the embodiments of the present application provide a fundus image prediction method, a fundus image prediction system, a device and a storage medium, which can effectively improve the prediction efficiency and prediction accuracy of the fundus image.

[0074] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0075] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0076] The eye fundus image prediction method provided in the embodiments of the present application relates to the technical field of artificial intelligence. The eye fundus image prediction method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch or the like; the server can be a stand-alone server or a cloud server providing cloud services, a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content distribution network (CDN), and basic cloud computing services such as a big data and artificial intelligence platform; and the software can be an application for implementing the eye fundus image prediction method, but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] It should be noted that in each specific embodiment of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the object, such as object information, object behavior data, object voice data, object historical data and object location information, the permission or consent of the object is obtained first, and the collection, use and processing of the data are in compliance with relevant laws, regulations and standards of the country and region. In addition, when the embodiments of the present application need to obtain sensitive personal information of the object, the separate permission or separate consent of the object is obtained through a pop-up window or a jump to a confirmation page, and after the separate permission or separate consent of the object is obtained, the necessary object-related data for enabling the embodiments of the present application to normally operate is obtained.

[0079] Please refer to Figure 1 , Figure 1is an optional flowchart of the fundus image prediction method provided by the embodiment of the present application. In some embodiments of the present application, the fundus image prediction method proposed in the present application includes but is not limited to steps S110 to S170, which will be described below in combination with Figure 1 The seven steps will be described in detail.

[0080] Step S110, obtaining a target fundus detection image, the target fundus detection image including a target fundus region image and a target text region image;

[0081] Step S120, performing text recognition on the target text region image to obtain a target detection text;

[0082] Step S130, inputting the target fundus region image and the target detection text into a pre-trained fundus image prediction model, the fundus image prediction model including an entity relationship sub-model, a graph construction sub-model and a fundus type prediction sub-model;

[0083] Step S140, performing entity data recognition on the target detection text according to the entity relationship sub-model to obtain target entity data and target entity relationship data;

[0084] Step S150, performing graph construction on the target fundus region image, the target entity data and the target entity relationship data according to the graph construction sub-model to obtain a target to-be-matched sub-graph;

[0085] Step S160, performing graph matching on the target to-be-matched sub-graph according to the fundus type prediction sub-model and a pre-constructed fundus knowledge graph to determine a fundus matched sub-graph;

[0086] Step S170, determining a target fundus detection type of the target fundus detection image according to the fundus matched sub-graph.

[0087] In steps S110 to S170 of some embodiments, a target fundus detection image is acquired, which includes a target fundus region image and a target text region image. Text recognition is performed on the target text region image to obtain target detection text. Then, the target fundus region image and the target detection text are input into a pre-trained fundus image prediction model, which includes an entity relation sub-model, a graph construction sub-model, and a fundus type prediction sub-model. Entity data recognition is performed on the target detection text according to the entity relation sub-model to obtain target entity data and target entity relation data. Next, a graph is constructed from the target fundus region image, target entity data, and target entity relation data according to the graph construction sub-model to obtain a target matching sub-graph. Graph matching is performed on the target matching sub-graph according to the fundus type prediction sub-model and the pre-constructed fundus knowledge graph to determine the fundus matching sub-graph. Finally, the target fundus detection type of the target fundus detection image is determined based on the fundus matching sub-graph. This application's embodiments introduce a knowledge graph to predict the type of fundus images, which can effectively improve the prediction efficiency and accuracy of fundus images.

[0088] In step S110 of some embodiments, in order to predict the fundus image type corresponding to the fundus detection image, such as Figure 2 As shown, a target fundus detection image 210 is first acquired. This target fundus detection image 210 represents the fundus angiography report generated by a doctor when diagnosing fundus diseases in a target subject. The target fundus detection image 210 includes a target fundus region image 211 and a target text region image 212. The target fundus region image 211 may include at least one type of fundus fluorescence detection image, such as fundus color imaging (Mcolor), fundus near-infrared imaging (IR), or fundus fluorescence angiography (FFA). The target detection text identified in the target text region image is used to characterize the image analysis results of the fundus fluorescence detection image obtained by the fundus detection device. For example, an image analysis result could be "Diagnosis of surface A and surrounding blood vessels B with leakage C, unclear boundaries, seen early during left eye angiography," where A, B, and C can be used to characterize different structures in the fundus of the target subject. The target fundus detection type is used to characterize the fundus disease type predicted by the fundus image prediction method of this application corresponding to the target fundus detection image. The target fundus examination type can be any existing fundus disease type, such as type D, type E, etc., where D and E are specific fundus disease types.

[0089] It should be noted that in this embodiment, different types of fundus fluorescence detection images are extracted separately to facilitate model similarity comparison.

[0090] It should be noted that the fundus image prediction method of the present application can be used to assist in solving the problem of mismatch between existing target object disease prevention needs and medical resources, that is, through informationization, intelligentization and large-scale means, timely disease prevention is realized for a large number of people with eye discomfort, rather than simply moving the doctor's work from offline to online.

[0091] In step S120 of some embodiments, in order to improve the prediction accuracy of the fundus image, after the text recognition of the target text region image, the application embodiment can also use a preset automatic text correction algorithm to correct the recognized target detection text to update the target detection text. The text recognition method of the present application embodiment can use Convolutional Recurrent Neural Network (CRNN), Sequence2Sequence algorithm, etc., which is not limited here.

[0092] In step S130 of some embodiments, the fundus image prediction model is constructed according to the entity relationship sub-model, the graph construction sub-model and the fundus type prediction sub-model, and the model training of the fundus image prediction model is equivalent to the joint training of the three sub-models.

[0093] Please refer to Figure 3 , Figure 3 is another optional flowchart of the fundus image prediction method provided by the present application. In some embodiments of the present application, before step S130, the fundus image prediction method of the present application can also specifically include: training the fundus image prediction model, that is, jointly training the entity relationship sub-model, the graph construction sub-model and the fundus type prediction sub-model. Therefore, this step can specifically include but is not limited to steps S310 to S360, which will be described below in combination with Figure 3 The six steps will be described in detail.

[0094] Step S310, obtaining a first fundus region image, a first detection text of the first fundus region image and a first fundus detection type of the first fundus region image;

[0095] Step S320, inputting the first fundus region image and the first detection text into a preset initial image prediction model, the initial image prediction model including an entity relationship sub-model, a graph construction sub-model and a fundus type prediction sub-model;

[0096] Step S330, performing entity data recognition on the first detection text according to the entity relationship sub-model to obtain first entity data and first entity relationship data;

[0097] In step S340, the first fundus region image, the first entity data, and the first entity relationship data are subjected to graph construction according to the graph construction submodel to obtain an initial to-be-matched subgraph;

[0098] In step S350, the initial to-be-matched subgraph is subjected to graph matching according to the fundus type prediction submodel and the pre-constructed fundus knowledge graph to obtain a sample matched subgraph, and the predicted fundus detection type is determined according to the sample matched subgraph.

[0099] In step S360, the initial image prediction model is subjected to parameter adjustment according to the first fundus detection type and the predicted fundus detection type to obtain a fundus image prediction model.

[0100] In steps S310 to S360 of some embodiments, in order to improve the prediction accuracy and interpretability of the trained fundus image prediction model. Specifically, the entity relationship submodel is used to enable the trained model to have good migration and generalization ability between different tasks, especially to be applicable to small sample fine-tuning. The graph construction submodel can greatly improve the data visualization, and the knowledge graph is used to realize automatic identification of image types, which can quickly filter out the required key information. The fundus type prediction submodel is used to predict the image type according to the constructed fundus knowledge graph. The first fundus detection type is used to represent the fundus disease type after the doctor's diagnosis, which is used to help the model more accurately predict the fundus image type. The first fundus detection type includes at least one fundus disease type, for example, D type, E type, etc. The first detection text can refer to the processing process of the target detection text in the entity relationship submodel, which will not be described here.

[0101] It should be noted that a training sample image for training the fundus image prediction model is constructed, which can be an eye fundus angiography report generated by a doctor when diagnosing the fundus disease of a target object, and the report is subjected to image acquisition to obtain the training sample image. Through image region segmentation on the training sample image, the first fundus region image and the first detection text are obtained.

[0102] It should be noted that the model loss value is determined according to the first fundus detection type and the predicted fundus detection type, and the initial image prediction model is subjected to parameter adjustment according to the model loss value, and when the model after parameter adjustment satisfies the prediction end condition, the fundus image prediction model is obtained. The loss function used for solving the model loss value can be any one of L1 loss function, L2 loss function, etc., which is not limited here.

[0103] It should be noted that the model structure of the initial image prediction model and the model structure of the fundus image prediction model are the same.

[0104] It should be noted that the preset training end condition of the initial image prediction model for parameter adjustment in the embodiments of the present application can be that the prediction accuracy of the fundus detection report of the initial image prediction model is greater than or equal to a preset accuracy threshold; or, when the model loss value of the initial image prediction model is less than a preset loss threshold, such as a preset loss threshold of 0.05, etc., which is not limited here.

[0105] Please refer to Figure 4 , Figure 4 is a flowchart of the specific method of step S310 provided by the embodiments of the present application. In some embodiments of the present application, step S310 can specifically include but is not limited to steps S410 to S460, which will be described below in combination with Figure 4 The six steps will be described in detail.

[0106] Step S410, obtaining an initial detection image, and performing image contour recognition on the initial detection image to obtain a fundus detection region image;

[0107] Step S420, performing image blur detection on the fundus detection region image to obtain a blur detection result;

[0108] Step S430, performing image screening on the initial detection image according to the blur detection result to determine a candidate detection image;

[0109] Step S440, performing image division on the candidate detection image to obtain a first fundus region image and a sample text region image;

[0110] Step S450, performing fundus detection type marking on the first fundus region image to obtain a first fundus detection type;

[0111] Step S460, performing text recognition on the sample text region image to obtain a first detection text.

[0112] In step S410 of some embodiments, in order to improve the efficiency of the model in predicting the fundus image, after obtaining the fundus detection report, the fundus detection report is first data cleaned according to the diagnosis result of the doctor on the fundus detection report, that is, the detection report with no abnormality of the fundus is excluded. And the detection report with abnormality of the fundus is taken as the initial detection image, which is the training sample image. Then, in order to further improve the quality of the model training sample, the initial detection image is subjected to image contour recognition according to a preset image contour detection method to obtain a fundus detection region image and a fundus text region image. As Figure 2 shown, the region corresponding to the target fundus region image 211 is equivalent to the region corresponding to the first fundus region image, and the region corresponding to the target text region image 212 is equivalent to the region corresponding to the first detection text.

[0113] It should be noted that the preset image contour detection method can be image contour detection using opencv in the python toolkit of the related technology, or can be a contour detection method based on edge detection, that is, by defining low-level mutations of brightness, color and the like, image contour detection is completed by identifying points with obvious brightness changes in the image. The specific image contour detection method is not limited here.

[0114] In step S420 of some embodiments, the image blur detection can be performed by the method of calculating the edge blur degree value of the Laplacian operator, that is, the single channel of the fundus detection region image is convolved with the preset blur convolution kernel matrix to obtain the variance of the image. Then, the variance is compared with the preset image variance threshold to determine the model result of the fundus detection region image. When the variance of the image is lower than the preset image variance threshold, the image is blurred; when the variance is not lower than the preset image variance threshold, the image is not blurred. The specific image blur detection method is not limited here, and will not be described again.

[0115] It should be noted that the image blur detection can also be performed manually to eliminate invalid initial detection images.

[0116] In step S430 of some embodiments, when the blur detection result indicates that the fundus detection region image is blurred, the initial detection image is invalid, and the initial detection image is eliminated; when the blur detection result indicates that the fundus detection region image is not blurred, the initial detection image is valid, and the valid initial detection image is used as a candidate detection image.

[0117] In step S440 of some embodiments, the candidate detection image is divided into an image according to the image contour detection method to obtain a first fundus region image and a sample text region image. The sample text region image is used to represent the image corresponding to the region where the first detection text is located.

[0118] It should be noted that the image contour method used at this time can refer to the description of the image contour detection method in the above embodiments, which will not be described again here.

[0119] In steps S450 and S460 of some embodiments, in order to improve the accuracy of the model in predicting the type of fundus image, the first fundus region image is labeled with a fundus detection type to obtain a first fundus detection type. In addition, the sample text region image is processed by text recognition to identify the first detection text.

[0120] It should be noted that after the text recognition of the sample text region image, the application embodiment can also use a preset automatic text correction algorithm to perform text correction processing on the recognized first detection text to update the first detection text, thereby improving the prediction accuracy of the model for the fundus image type.

[0121] In step S140 of some embodiments, since the related art fundus image prediction method is mostly for single modality, single disease type screening diagnosis, the prediction ability of the model is weak. In order to solve the above problem, the application embodiment adopts a plurality of modalities of fundus images and different fundus disease descriptions as auxiliary training, so that the fundus image prediction model obtained by training is more accurate and has better interpretability. The target entity data is used to represent the specific entity text in the target detection text, and the target entity relationship data is used to represent the data connection between different target entity data.

[0122] It should be noted that the application embodiment can improve based on the BERT model structure to construct the entity relationship sub-model. The entity relationship sub-model can also be fine-tuned based on the UIE framework to realize unified modeling of entity extraction, relationship extraction, event extraction and other tasks, so that the model has good migration and generalization ability between different tasks. The target entity data is used to represent the key text data in the first detection text and the entity type corresponding to the key text data. The target entity relationship data is used to represent the association information between different entity types.

[0123] For example, assuming that a target detection text is "A surface and surrounding B blood vessels are dilated with C leakage during left eye angiography, and F shows strong fluorescence in the late stage". After inputting the target detection text into the entity relationship sub-model for entity relationship extraction, the target entity data includes: "abnormal site: A surface and surrounding B blood vessels", "abnormal phenomenon: dilatation", "abnormal phenomenon: C leakage", "abnormal site: F", and "abnormal phenomenon: strong fluorescence". Therefore, the corresponding entity types include "abnormal site" and "abnormal phenomenon", and the target entity data "abnormal site: A surface and surrounding B blood vessels" and the target entity data "abnormal phenomenon: dilatation" can constitute a mutually associated target entity relationship data. For another example, the target entity data "abnormal site: A surface and surrounding B blood vessels" and the target entity data "abnormal phenomenon: C leakage" can constitute another mutually associated target entity relationship data. However, the target entity data "abnormal site: A surface and surrounding B blood vessels" and the target entity data "abnormal phenomenon: strong fluorescence" cannot constitute associated target entity relationship data. By analogy, "abnormal site: F" and "abnormal phenomenon: strong fluorescence" can also constitute an associated target entity relationship data.

[0124] Please refer to Figure 5 , Figure 5 is a flowchart of a specific method of step S140 provided by the embodiments of the present application. In some embodiments of the present application, step S140 can specifically include but is not limited to steps S510 to S530, which will be described below in combination with Figure 5 The three steps will be described in detail.

[0125] Step S510, text labeling is performed on the target detection text to obtain fundus detection words, text labeling types of the fundus detection words, and position information of the fundus detection words;

[0126] Step S520, fundus detection words are screened according to the preset entity type and the text labeling type to obtain target entity data;

[0127] Step S530, entity relationship extraction is performed on the target entity data according to the position information to obtain target entity relationship data.

[0128] In steps S510 to S530 of some embodiments, in order to accurately extract the target entity data and the target entity relationship data, first, text labeling is performed on the target detection text, and the fundus detection word is used to represent the text word after the target detection text is divided. The text labeling type is used to represent the text type of the fundus detection word. The position information is used to represent the paragraph position of the text word in the target detection text, which is used to determine the entity relationship between different text words. Since the text type includes multiple types, such as nouns, adverbs, and verbs, and the text word of the adverb type has no effective meaning, for example, the fundus detection word is screened according to the preset entity type to determine the text word that has an impact on the model prediction, that is, to obtain the target entity data. The preset entity type can be a noun, a verb, etc. The entity relationship extraction is performed on the target entity data according to the position information, for example, the entity relationship between the target entity data that are far apart is weak and can not be considered.

[0129] In step S150 of some embodiments, since the knowledge graph method can greatly improve the data visualization and can realize automatic identification of image types, thereby quickly screening the required key information, the embodiments of the present application construct an ophthalmic multi-modal knowledge graph in combination with the knowledge graph method to improve the prediction accuracy of the fundus image. Among them, because many medical nouns are expressed in multiple ways, after obtaining the target entity data and the target entity relationship data, the extracted entities and relationships are cleaned and disambiguated to update the target entity data and the target entity relationship data. And according to the graph construction sub-model, the updated target fundus region image, the target entity data and the target entity relationship data are graph constructed, that is, the cleaned entity relationship is aligned and integrated with the image to obtain a target to-be-matched subgraph.

[0130] For example,Figure 6 As shown, Figure 6 is a structural schematic diagram of a target subgraph provided by an embodiment of the present application. The target subgraph includes two constituent elements: a predicted subgraph vertex and a predicted subgraph edge. The predicted subgraph vertex includes a subgraph number 611 and an abnormal phenomenon in target entity data, and the predicted subgraph edge is an abnormal part of the target entity data connected according to target entity relationship data. In this embodiment of the present application, the target fundus region image and the target entity data extracted from the target detection text are structured in a graph atlas manner according to the target entity relationship data, so as to be converted into a computer language convenient for computer reading and operation. The target fundus region image includes a first image 612 and a second image 613. For example, the target detection text is "A surface expansion is seen in the early stage and F shows strong fluorescence in the late stage during left eye angiography", and the target entity data obtained after entity data recognition includes abnormal phenomena "expansion" and "strong fluorescence", and further includes abnormal parts "A surface" and "F". The extracted target entity data and the corresponding target fundus region image are graphically constructed according to the target entity relationship data, and a target subgraph to be matched is obtained.

[0131] Please refer to Figure 7 , Figure 7 is another optional flowchart of the fundus image prediction method provided by an embodiment of the present application. In some embodiments of the present application, before step S160, the fundus image prediction method provided by the present application can further include the step of constructing a fundus knowledge graph, which can specifically include but is not limited to steps S710 to S750. The following will be described in combination with Figure 7 The five steps will be described in detail.

[0132] Step S710, acquiring a second fundus region image, a second detection text of the second fundus region image, and a second fundus detection type of the second fundus region image;

[0133] Step S720, inputting the second detection text into a preset entity extraction model to extract entity relationship, and obtaining second entity data and second entity relationship data;

[0134] Step S730, integrating data according to the second fundus region image, the second entity data, and the second entity relationship data, and obtaining sample fundus triple data;

[0135] Step S740, constructing a graph atlas according to the second fundus detection type and the sample fundus triple data, and obtaining an initial knowledge graph;

[0136] Step S750, inputting the initial knowledge graph into a preset graph reasoning model to perform graph completion processing, and obtaining a fundus knowledge graph.

[0137] In steps S710-S750 of some embodiments, in order to effectively improve the prediction efficiency and prediction accuracy of the fundus image, the embodiments of the present application predict the type of the fundus image by introducing a knowledge graph. First, a sample detection report is obtained, which is in the same form as the target fundus detection image and will not be described here. The sample detection report can be data obtained from a medical detection database, or data flexibly set according to actual needs. The image of the sample detection report includes a second fundus region image, a second detection text of the second fundus region image, and a second fundus detection type of the second fundus region image. The second fundus detection type is the pre-labeled fundus detection type to which the sample detection report belongs. The model structure of the pre-set entity extraction model and the entity relationship sub-model is the same, and will not be described in detail here. Then, the second fundus region image, the second entity data, and the second entity relationship data are integrated to obtain sample fundus triple data. The second fundus region image, the second entity data, and the second entity relationship data have a mapping relationship, and each sample fundus triple data is used to construct a sample sub-graph. The fundus knowledge graph includes at least one fundus knowledge sub-graph, and each fundus knowledge sub-graph is used to represent a graph with the second fundus detection type as the knowledge sub-graph main node. The sample fundus triple data under the same type is graphically constructed according to the second fundus detection type, i.e., the sample fundus triple data under the same type is connected to the node of the second fundus detection type. The embodiments of the present application integrate the sample fundus triple data under the same second fundus detection type together to construct a rich fundus knowledge graph including different second fundus detection types.

[0138] It should be noted that many sample detection reports obtained do not explicitly diagnose the disease type, i.e., the second fundus detection type cannot be determined. In order to perfect the data of the fundus knowledge graph to better assist in constructing the fundus image prediction model, after graphically constructing the sample fundus triple data according to the second fundus detection type, an initial knowledge graph is obtained. Since the sample fundus triple data cannot be determined, the node of the second fundus detection type connected thereto cannot be determined. Therefore, the initial knowledge graph is input into a pre-set graph reasoning model for graph completion processing, i.e., the sample fundus triple data not connected to the second fundus detection type is graphically completed to obtain the fundus knowledge graph.

[0139] It should be noted that the graph reasoning sub-model used for graph completion in the embodiments of the present application can use any one of a Translating Embedding (TransE) model, a Translating on relation space model, etc., and is not specifically limited here.

[0140] The embodiment of the application improves the data of the fundus knowledge graph through the graph reasoning model, so as to better assist in constructing the fundus image prediction model, and the entity extraction model can effectively improve the prediction efficiency and prediction accuracy of the fundus image.

[0141] In step S160 of some embodiments, in order to improve the prediction accuracy of the fundus image, the fundus type prediction sub-model of the embodiment of the application includes a text matching unit and an image matching unit. Specifically, when performing graph matching, first determine some sub-graphs similar to the text nodes of the target to-be-matched sub-graph from the fundus knowledge graph according to the text matching unit. Then, the target fundus region image and the images in the sub-graph similar to each text node are matched to accurately determine the sub-graph corresponding to the image with the highest similarity, that is, the fundus matching sub-graph. The embodiment of the application can effectively reduce the calculation amount of the model by the sub-graph matching scheme of first text matching and then image matching, that is, effectively improve the prediction efficiency and prediction accuracy of the model.

[0142] Please refer to Figure 8 , Figure 8 is a flowchart of the specific method of step S160 provided by the embodiment of the application. In some embodiments of the application, step S160 can specifically include but is not limited to steps S810 to S840, which will be described below in combination with Figure 8 The four steps will be described in detail.

[0143] Step S810, entity data extraction is performed on the fundus knowledge graph to obtain candidate entity data;

[0144] Step S820, entity similarity calculation is performed on the target entity data and the candidate entity data to obtain entity similarity data, and prediction entity data is determined according to the entity similarity data and a preset entity threshold;

[0145] Step S830, graph segmentation is performed on the fundus knowledge graph according to the prediction entity data to obtain candidate sub-graphs;

[0146] Step S840, sub-graph screening is performed on the candidate sub-graphs according to the target fundus region image to obtain a fundus matching sub-graph.

[0147] In steps S810 to S840 of some embodiments, since the fundus knowledge graph is constructed according to entity data. Specifically, the entity data in the fundus knowledge graph is extracted, and the extracted candidate entity data and the target entity data of the target to-be-matched subgraph are subjected to entity similarity calculation to obtain entity similarity data. The entity similarity data is used to represent the text similarity between the candidate entity data and the target entity data. The preset entity threshold is used to determine the predicted entity data that meets the entity similarity requirement, so as to improve the prediction accuracy of the model on the fundus image. The predicted entity data is used to represent the entity data whose entity similarity data is greater than or equal to the preset entity threshold. The fundus knowledge graph is subjected to graph segmentation according to the predicted entity data to obtain a candidate subgraph, and the candidate subgraph includes the predicted entity data and a candidate fundus image. Therefore, the subgraph matching is performed according to the target fundus region image and the candidate fundus image in the candidate subgraph, so as to screen and determine the fundus matching subgraph. The embodiment of the present application determines the candidate subgraph with similar entity data by text matching first. Then, the subgraph matching scheme of image matching is performed, which can effectively reduce the calculation amount of the model, and can effectively improve the prediction efficiency and prediction accuracy of the model through layer-by-layer screening.

[0148] It should be noted that the entity similarity calculation method can adopt a similarity transfer algorithm or a text similarity calculation method, which is not limited here.

[0149] Please refer to Figure 9 , Figure 9 is a flowchart of the specific method of step S840 provided by the embodiment of the present application. In some embodiments of the present application, step S840 can specifically include but is not limited to steps S910 and S920, which will be described below in combination with Figure 9 The two steps will be described in detail.

[0150] Step S910, image similarity calculation is performed on the target fundus region image and the candidate fundus image in the candidate subgraph to obtain image similarity data;

[0151] Step S920, subgraph screening is performed on the candidate subgraph according to the image similarity data to obtain the fundus matching subgraph.

[0152] In steps S910 and S920 of some embodiments, the embodiments of the present application use an image matching method to calculate the image similarity of the target fundus region image and the candidate fundus image of the candidate subgraph, and compare the obtained image similarity data with the preset image similarity threshold value. If the image similarity data is greater than or equal to the image similarity threshold value, the candidate subgraph corresponding to the candidate fundus image is determined as the fundus matching subgraph. If the image similarity data is less than the image similarity threshold value, the candidate subgraph is excluded, and the comparison of the next candidate subgraph is performed. The preset image similarity threshold value is used to determine the candidate fundus image that meets the image similarity requirement, so as to improve the prediction accuracy of the model on the fundus image.

[0153] It should be noted that the image similarity calculation method includes a cosine similarity calculation method, a hash algorithm, etc., which is not limited here.

[0154] In step S170 of some embodiments, the fundus detection type corresponding to the fundus matching subgraph is the target fundus detection type predicted for the target fundus detection image. In specific applications, for example, the fundus image prediction system proposed by the present application can be applied to a terminal, and the fundus image prediction model trained by the present application is deployed in the fundus image prediction system to execute the above-mentioned fundus image prediction method. When the target object needs to perform fundus image prediction in the terminal, the target object can upload the target fundus detection report as the target fundus detection image on the terminal page. Then, the terminal page can display a pop-up box to make the target object confirm the completion of uploading and start executing the fundus image prediction method to generate a fundus image prediction service request. The terminal sends the image prediction service request containing the target fundus detection image to the port executing the fundus image prediction method for fundus image prediction processing. Then, the target fundus detection type is received and returned.

[0155] Please refer to Figure 10 , Figure 10 is a schematic diagram of the module structure of the fundus image prediction system provided by the embodiments of the present application. In some embodiments of the present application, the fundus image prediction system includes an image acquisition module 1010, a text recognition module 1020, a model input module 1030, an entity data recognition module 1040, a graph construction module 1050, a graph matching module 1060, and an image prediction module 1070.

[0156] The image acquisition module 1010 is configured to acquire a target fundus detection image, wherein the target fundus detection image includes a target fundus region image and a target text region image.

[0157] The text recognition module 1020 is configured to perform text recognition on the target text region image to obtain a target detection text.

[0158] The model input module 1030 is used to input the target fundus region image and the target detection text into a pre-trained fundus image prediction model. The fundus image prediction model includes an entity relationship sub-model, a graph construction sub-model and a fundus type prediction sub-model.

[0159] The entity data recognition module 1040 is used to perform entity data recognition on the target detection text according to the entity relationship sub-model to obtain target entity data and target entity relationship data.

[0160] The atlas construction module 1050 is used to construct atlases from the target fundus region image, target entity data and target entity relationship data based on the atlas construction sub-model to obtain the target matching sub-graph.

[0161] The graph matching module 1060 is used to perform graph matching on the target subgraph to be matched based on the fundus type prediction submodel and the pre-built fundus knowledge graph to determine the matching fundus subgraph.

[0162] The image prediction module 1070 is used to determine the target fundus detection type of the target fundus detection image based on the fundus matching sub-image.

[0163] It should be noted that the fundus image prediction system of this application embodiment is used to execute the above fundus image prediction method, and the fundus image prediction system of this application embodiment corresponds to the aforementioned fundus image prediction method. For the specific training process, please refer to the aforementioned fundus image prediction method, which will not be described in detail here.

[0164] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the fundus image prediction method described in this application.

[0165] Electronic devices can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0166] The following is combined with Figure 11 The electronic devices described in the embodiments of this application will be described in detail.

[0167] Please refer to Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0168] The processor 1110 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0169] The memory 1120 can be implemented by a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1120 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1120 and are called and executed by the processor 1110 to implement the fundus image prediction method of the embodiments of the present application.

[0170] The input / output interface 1130 is configured to implement information input and output.

[0171] The communication interface 1140 is configured to implement the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0172] The bus 1150 is configured to transmit information between various components (for example, the processor 1110, the memory 1120, the input / output interface 1130, and the communication interface 1140) of the device.

[0173] The processor 1110, the memory 1120, the input / output interface 1130, and the communication interface 1140 are connected to each other by the bus 1150 to realize the communication connection between the device.

[0174] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the fundus image prediction method of the embodiments of the present application.

[0175] The embodiment of the application provides an eye fundus image prediction method, an eye fundus image prediction system, equipment and a storage medium. The type of the eye fundus image is predicted by introducing an eye fundus knowledge graph. The multi-modal eye fundus knowledge graph takes multi-modal eye fundus detection images and disease description texts as entities, and reflects the relationship between the eye fundus detection type and the specific lesion, so that the visualization of data can be greatly improved. Moreover, the embodiment of the application uses the eye fundus knowledge graph to realize automatic identification of the image type, and the required key information can be quickly screened out. In addition, the embodiment of the application perfects the data of the eye fundus knowledge graph through a graph reasoning model, so as to better assist in constructing an eye fundus image prediction model, and the prediction efficiency and prediction accuracy of the eye fundus image can be effectively improved in combination with an entity extraction model.

[0176] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0177] The embodiments described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0178] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the application, and can include more or fewer steps than the figures, or combine certain steps or different steps.

[0179] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0180] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the equipment can be implemented as software, firmware, hardware and their appropriate combinations.

[0181] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for descriptive purposes and not for pronouncing the limitations of the application described. For example, a first element, feature, structure, process, etc. brought out here can be later

[0182] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the association relationship of associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0184] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment of the present application.

[0185] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0186] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0187] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for predicting fundus images, characterized in that, The method includes: Acquire a target fundus detection image, the target fundus detection image including a target fundus region image and a target text region image; Perform text recognition on the target text region image to obtain the target detected text; Training a fundus image prediction model specifically includes: acquiring a first fundus region image, a first detection text of the first fundus region image, and a first fundus detection type of the first fundus region image; inputting the first fundus region image and the first detection text into a preset initial image prediction model, the initial image prediction model including an entity relation sub-model, a graph construction sub-model, and a fundus type prediction sub-model; performing entity data recognition on the first detection text according to the entity relation sub-model to obtain first entity data and first entity relation data; performing graph construction on the first fundus region image, the first entity data, and the first entity relation data according to the graph construction sub-model to obtain an initial sub-image to be matched; performing graph matching on the initial sub-image to be matched according to the fundus type prediction sub-model and a pre-constructed fundus knowledge graph to obtain a sample matching sub-image, and determining the predicted fundus detection type according to the sample matching sub-image; adjusting the parameters of the initial image prediction model according to the first fundus detection type and the predicted fundus detection type to obtain a fundus image prediction model; The target fundus region image and the target detection text are input into a pre-trained fundus image prediction model, which includes an entity relationship sub-model, a graph construction sub-model, and a fundus type prediction sub-model. Based on the entity relationship sub-model, entity data recognition is performed on the target detection text to obtain target entity data and target entity relationship data; Based on the atlas, a sub-model is constructed to perform atlas construction on the target fundus region image, the target entity data, and the target entity relationship data to obtain a target sub-map to be matched. The target subgraph to be matched is determined by graph matching based on the fundus type prediction submodel and the pre-constructed fundus knowledge graph. The target fundus detection type of the target fundus detection image is determined based on the fundus matching sub-image.

2. The method according to claim 1, characterized in that, Before performing graph matching on the target sub-image to be matched based on the fundus type prediction sub-model and the pre-constructed fundus knowledge graph to determine the matching fundus sub-image, the method further includes: Constructing the aforementioned fundus knowledge graph specifically includes: Acquire a second fundus region image, a second detection text of the second fundus region image, and a second fundus detection type of the second fundus region image; The second detected text is input into a preset entity extraction model to extract entity relationships, thereby obtaining second entity data and second entity relationship data; Data is integrated based on the second fundus region image, the second entity data, and the second entity relationship data to obtain sample fundus triad data. Based on the second fundus detection type and the three data points of the sample fundus, a graph is constructed to obtain an initial knowledge graph; The initial knowledge graph is input into a preset graph reasoning model for graph completion processing to obtain the fundus knowledge graph.

3. The method according to claim 1, characterized in that, The step of performing graph matching on the target sub-graph to be matched based on the fundus type prediction sub-model and the pre-constructed fundus knowledge graph to determine the matching fundus sub-graph includes: Entity data is extracted from the fundus knowledge graph to obtain candidate entity data; Entity similarity is calculated on the target entity data and the candidate entity data to obtain entity similarity data, and predicted entity data is determined based on the entity similarity data and a preset entity threshold. Based on the predicted entity data, the fundus knowledge graph is segmented to obtain candidate subgraphs; The candidate sub-images are filtered based on the target fundus region image to obtain the fundus matching sub-image.

4. The method according to claim 3, characterized in that, The step of filtering the candidate sub-images based on the target fundus region image to obtain the fundus matching sub-image includes: Image similarity data is obtained by calculating the image similarity between the target fundus region image and the candidate fundus images in the candidate sub-image. The candidate sub-images are filtered based on the image similarity data to obtain the fundus matching sub-image.

5. The method according to any one of claims 1 to 4, characterized in that, The step of performing entity data recognition on the target detected text based on the entity relationship sub-model to obtain target entity data and target entity relationship data includes: Text annotation is performed on the target detection text to obtain fundus detection words, the text annotation type of the fundus detection words, and the location information of the fundus detection words; The target entity data is obtained by filtering the fundus detection words according to the preset entity type and the text annotation type. Based on the location information, entity relationships are extracted from the target entity data to obtain the target entity relationship data.

6. The method according to claim 1, characterized in that, The acquisition of the first fundus region image, the first detection text of the first fundus region image, and the first fundus detection type of the first fundus region image includes: An initial detection image is acquired, and image contour recognition is performed on the initial detection image to obtain an image of the fundus detection area; Image blur detection is performed on the image of the fundus detection area to obtain blur detection results; Based on the fuzzy detection results, the initial detection image is filtered to determine candidate detection images; The candidate detection image is segmented to obtain the first fundus region image and the sample text region image; The fundus detection type is labeled on the first fundus region image to obtain the first fundus detection type; The first detected text is obtained by performing text recognition on the sample text region image.

7. A fundus image prediction system, characterized in that, The system includes: The image acquisition module is used to acquire a target fundus detection image, which includes a target fundus region image and a target text region image; The text recognition module is used to perform text recognition on the target text region image to obtain the target detected text; The model input module is used to train a fundus image prediction model, specifically including: acquiring a first fundus region image, a first detection text of the first fundus region image, and a first fundus detection type of the first fundus region image; inputting the first fundus region image and the first detection text into a preset initial image prediction model, wherein the initial image prediction model includes an entity relation sub-model, a graph construction sub-model, and a fundus type prediction sub-model; performing entity data recognition on the first detection text according to the entity relation sub-model to obtain first entity data and first entity relation data; and performing entity data recognition on the first fundus region image, the first entity data, and the first entity relation data according to the graph construction sub-model. The system constructs a graph from entity relation data to obtain an initial subgraph to be matched; it then performs graph matching on the initial subgraph to be matched based on the fundus type prediction submodel and the pre-constructed fundus knowledge graph to obtain a sample matching subgraph, and determines the predicted fundus detection type based on the sample matching subgraph; the initial image prediction model is then adjusted according to the first fundus detection type and the predicted fundus detection type to obtain a fundus image prediction model; the model input module is further used to input the target fundus region image and the target detection text into the pre-trained fundus image prediction model, which includes an entity relation submodel, a graph construction submodel, and a fundus type prediction submodel; The entity data recognition module is used to perform entity data recognition on the target detection text according to the entity relationship sub-model to obtain target entity data and target entity relationship data. The atlas construction module is used to construct an atlas based on the atlas construction sub-model for the target fundus region image, the target entity data and the target entity relationship data, to obtain a target sub-graph to be matched. The graph matching module is used to perform graph matching on the target subgraph to be matched based on the fundus type prediction submodel and the pre-constructed fundus knowledge graph to determine the fundus matching subgraph; An image prediction module is used to determine the target fundus detection type of the target fundus detection image based on the fundus matching sub-image.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images