Diabetic retinopathy whole fundus dynamic labeling system

Abstract

The invention provides a diabetic retinopathy total fundus dynamic labeling system, and relates to the technical field of medical image processing. The system comprises a data acquisition module, an image labeling module, a multi-mode integration display module and a user interaction module. According to the system, the diabetic retinopathy marking model is utilized to realize lesion area identification and lesion type marking on various types of fundus image data, so that a doctor can clearly and quickly know the progress of an illness state without artificial experience; the full fundus image data marked with the lesion area and the lesion type, the image data of the posterior pole and the macular area and the retinal blood vessel image data are fused, the lesion area and the lesion type of any area of the full fundus of the user to be detected can be displayed on a display interface, a visual blind area is eliminated, and the detection accuracy is improved. The user can view any interested area of the whole fundus, so that comprehensive analysis of the diabetic retinopathy is realized, and the diagnosis and treatment efficiency of the diabetic retinopathy is comprehensively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a diabetic retinopathy full fundus dynamic labeling system. Background Art

[0002] Diabetic retinopathy (DR) is a common and serious complication of diabetes, characterized by different fundus manifestations including microaneurysms, hard exudates, soft exudates, hemorrhage, retinal neovascularization, etc.

[0003] In the related technology, doctors mainly rely on viewing fundus photographs of patients with diabetic retinopathy and evaluating the patient's lesion condition based on clinical experience. However, fundus photographs have blind spots, and the above method cannot fully present the details of the lesions of the entire fundus; and the evaluation of the lesion condition based solely on the doctor's clinical experience cannot meet the needs of accurate and rapid diagnosis and treatment of diabetic retinopathy.

[0004] Therefore, how to achieve comprehensive, accurate and rapid diagnosis and treatment of diabetic retinopathy is an urgent problem to be solved. Summary of the invention

[0005] In view of the problems existing in the prior art, an embodiment of the present invention provides a diabetic retinopathy full fundus dynamic labeling system.

[0006] The present invention provides a diabetic retinopathy whole fundus dynamic marking system, comprising: A data acquisition module, connected to the wide-angle fundus camera, the OCT device, the OCTA device and the FFA device through standard interfaces, for acquiring an initial multimodal fundus image data set of the user to be examined; the multimodal fundus image data set includes full fundus image data, posterior pole and macular area image data, and retinal vascular image data; An image annotation module connected to the data acquisition module is used to input the initial multimodal fundus image data set into a diabetic retinopathy annotation model to obtain a predicted annotation result output by the diabetic retinopathy annotation model; the predicted annotation result includes fundus image data annotated with lesion areas and lesion types; the diabetic retinopathy annotation model is trained based on full fundus image sample data, posterior pole and macular area image sample data, and retinal vascular image sample data of multiple users, and corresponding sample annotation results; the diabetic retinopathy annotation model is used to perform lesion annotation on multiple types of fundus image data in the multimodal fundus image data set; a multimodal integrated display module connected to the data acquisition module and the image annotation module respectively, for fusing different types of fundus image data in the initial multimodal fundus image data set based on an image fusion algorithm to obtain a target multimodal fundus image data set; A user interaction module connected to the image annotation module and the multimodal integrated display module is used to display the fundus image data of any area of ​​the entire fundus of the user to be tested and the corresponding predicted annotation results on the display interface of the user terminal based on the target multimodal fundus image data set.

[0007] Optionally, the system further comprises: The lesion comparison and analysis module connected to the image annotation module is used to use a feature point matching algorithm to compare the predicted annotation results with the historical annotation results of the historical follow-up images of the user to be inspected to obtain lesion change information; the lesion change information includes the change information of the lesion area and lesion type in the predicted annotation results and the historical annotation results; based on the lesion change information, the characteristic parameters of the lesion change information are calculated using statistical and machine learning algorithms; based on the characteristic parameters of the lesion area, a follow-up report of the user to be inspected is generated.

[0008] Optionally, the lesion comparison and analysis module is specifically used to: Segmenting the predicted annotation results and the historical follow-up images into a plurality of sub-regions of the same size and number; For each sub-region, a feature point matching algorithm is used to identify the boundary of the lesion region of the predicted annotation result and the historical annotation result in the sub-region, and obtain a first boundary corresponding to the lesion region of the predicted annotation result and a second boundary corresponding to the lesion region of the historical annotation result in the sub-region; The lesion change information is determined based on the predicted labeling result and the lesion type corresponding to the historical follow-up image, and the coordinate information corresponding to the first boundary and the second boundary.

[0009] Optionally, the characteristic parameters include: the change values ​​of the size, quantity and position coordinates of the lesion area between the predicted annotation result and the historical annotation result; The lesion comparison and analysis module is specifically used for: The follow-up report is generated according to each of the change values; the follow-up report includes a lesion trend graph of the user to be examined, which is used to reflect the change trend of the lesion area and lesion type.

[0010] Optionally, the user interaction module is further used to: When the change value of at least one of the size, number and position coordinates of the lesion area exceeds a preset threshold, a prompt message is displayed on the display interface; the prompt message is used to reflect that at least one of the size, number and position of the lesion area has improved or worsened.

[0011] Optionally, the user interaction module is connected to the lesion comparison and analysis module; The user interaction module is also used to display the follow-up report on the display interface.

[0012] Optionally, the user interaction module is further used to: In response to a first operation instruction of the user on the display interface, the predicted annotation result is modified and / or an annotation result is added.

[0013] Optionally, the user interaction module is further used to: In response to a second operation instruction of the user on the display interface for the fundus image data of any area of ​​the entire fundus, the predicted labeling result and the historical labeling result corresponding to the fundus image data of any area of ​​the entire fundus are displayed.

[0014] Optionally, the user interaction module is further used to: In response to a third operation instruction of the user on the display interface, displaying a normal full fundus image atlas on the display interface; In response to a fourth operation instruction of the user on the display interface for the normal whole fundus image atlas, a manual annotation result is added to the normal whole fundus image atlas; the manual annotation result includes a whole fundus image atlas annotated with lesion areas and lesion types.

[0015] Optionally, the lesion type includes at least one of the following: Microaneurysm, hard exudates, soft exudates, retinal hemorrhage, tortuosity and dilatation of retinal veins, neovascularization, vitreous hemorrhage, macular edema, epiretinal membrane, retinal detachment, hard exudate rings, optic disc neovascularization, retinal arteriosclerosis, fibrovascular proliferation and diabetic maculopathy.

[0016] In the diabetic retinopathy full fundus dynamic labeling system provided by the present invention, a data acquisition module is connected to a wide-angle fundus camera, an OCT device, an OCTA device and an FFA device respectively through a standard interface to collect an initial multimodal fundus image data set of a user to be tested, including full fundus image data, posterior pole and macular area image data, and retinal vascular image data; then the initial multimodal fundus image data set is input into a diabetic retinopathy labeling model by using an image labeling module, and the diabetic retinopathy labeling model can identify the lesion area of ​​multiple types of fundus image data in the multimodal fundus image data set and label the corresponding lesion type, thereby obtaining a predicted labeling result; a multimodal integrated display module fuses different types of fundus image data in the initial multimodal fundus image data set based on an image fusion algorithm to obtain a target multimodal fundus image data set; a user interaction module, based on the target multimodal fundus image data set, displays the fundus image data of any area of ​​the full fundus of the user to be tested and the corresponding predicted labeling result on the display interface of the user terminal. Through the above system, the diabetic retinopathy annotation model is used to realize the lesion area identification and lesion type annotation of various types of fundus image data, so that doctors can clearly and quickly understand the progression of the disease without manual experience; in addition, the whole fundus image data, posterior pole and macular area image data, and retinal vascular image data marked with lesion areas and lesion types are fused, and the lesion area and lesion type of any area of ​​the whole fundus of the user to be examined can be displayed on the display interface of the user terminal, eliminating visual blind spots. Users can view any area of ​​interest in the whole fundus, thereby realizing a comprehensive analysis of diabetic retinopathy and comprehensively improving the diagnosis and treatment efficiency of diabetic retinopathy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the structural schematic diagrams of the diabetic retinopathy full fundus dynamic annotation system provided by the present invention; Figure 2 is a schematic diagram of a display interface of a user terminal provided by the present invention; Figure 3 This is the second structural schematic diagram of the whole fundus dynamic marking system for diabetic retinopathy provided by the present invention; Figure 4 It is a schematic diagram of the flow of the method for dynamic marking of the entire fundus of diabetic retinopathy provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In order to facilitate a clearer understanding of the embodiments of the present application, some relevant knowledge is first introduced as follows.

[0021] First, the professional terms involved in the present invention are explained: 1. Wide-angle fundus camera: usually used to record the entire fundus lesions, but doctors can only judge the progression of the disease by comparing the images taken each time with the naked eye, and cannot systematically and intelligently compare the changes in lesions; 2. Optical coherence tomography (OCT) and optical coherence tomography angiography (OCTA): They are mainly used to observe lesions in the posterior pole and macular area, but they cannot fully present the details of lesions in the entire fundus (especially the peripheral area), which limits the comprehensive analysis of diabetic retinopathy. 3. Fluorescein Fundus Angiography (FFA) device: It can provide detailed images of retinal blood vessels, but is limited by its invasiveness and unsuitability for high-frequency follow-up.

[0022] In order to meet the demand for comprehensive, accurate and dynamic diagnosis and treatment of diabetic retinopathy and overcome the shortcomings of the prior art in the diagnosis and follow-up of diabetic retinopathy, the present invention provides a diabetic retinopathy full-fundus dynamic annotation system.

[0023] Combine the following Figures 1 to 3 The whole fundus dynamic labeling system for diabetic retinopathy provided by the present invention is described in detail. Figure 1 This is one of the structural schematic diagrams of the diabetic retinopathy full fundus dynamic annotation system provided by the present invention, see Figure 1 As shown, the system includes: a data acquisition module 101, an image annotation module 102, a multi-modal integrated display module 103 and a user interaction module 104; wherein: The data acquisition module 101 is connected to the wide-angle fundus camera, OCT device, OCTA device and FFA device respectively through standard interfaces, and is used to collect the initial multimodal fundus image data set of the user to be tested; the multimodal fundus image data set includes full fundus image data, posterior pole and macular area image data, and retinal vascular image data.

[0024] In the embodiment of the present invention, the data acquisition module 101 achieves seamless connection with various clinical devices through a standard interface, ensuring the integrity and high resolution of the initial multimodal fundus image data set, and ensuring the stability and efficiency of data acquisition.

[0025] In practical applications, the user to be tested refers to a patient with diabetic retinopathy. The data acquisition module 101 acquires an initial multimodal fundus image dataset of the patient with diabetic retinopathy to provide data support for the subsequent generation of a target multimodal fundus image dataset.

[0026] The image annotation module 102 connected to the data acquisition module 101 is used to input the initial multimodal fundus image data set into the diabetic retinopathy annotation model to obtain the predicted annotation results output by the diabetic retinopathy annotation model; the predicted annotation results include fundus image data annotated with lesion areas and lesion types; the diabetic retinopathy annotation model is trained based on full fundus image sample data, posterior pole and macular area image sample data, and retinal vascular image sample data of multiple users, and corresponding sample annotation results; the diabetic retinopathy annotation model is used to annotate lesions of multiple types of fundus image data in the multimodal fundus image data set.

[0027] In an embodiment of the present invention, the diabetic retinopathy annotation model can identify at least one lesion area of ​​each type of fundus image data in the initial multimodal fundus image data set, and the lesion type corresponding to each lesion area.

[0028] The image annotation module 102 can dynamically annotate the lesions (such as microaneurysms, exudation, hemorrhage, etc.) in the initial multimodal fundus image data set through the diabetic retinopathy annotation model, which can assist doctors in clearly understanding the progression or changes of the disease.

[0029] Optionally, the lesion type includes at least one of the following: A. Microaneurysms; B. Hard Exudates; C. Soft Exudates / Cotton Wool Spots; D. Retinal Hemorrhages; E. Venous Beading; F. Neovascularization; G. Vitreous hemorrhage; H. Macular Edema; I. Epiretinal Membrane; J. Retinal Detachment; K, Circinate Exudates; L, Neovascularization of the Disc (NVD); M, Retinal Arteriosclerosis; N, Fibrovascular Proliferation; O. Diabetic Maculopathy.

[0030] These lesion types may exist alone or simultaneously. Observing the development and changes of lesion types is a key factor in the clinical diagnosis and treatment of diabetic retinopathy.

[0031] The structure of the diabetic retinopathy annotation model is as follows: The diabetic retinopathy labeling model uses nnU-Net (No New-Net) as the low-level network topology, and is composed of six symmetric encoder-decoder layers with skip connections to provide detailed features from encoder to decoder.

[0032] In practical applications, a patch with 3 channels (each image data in the initial multimodal fundus image dataset) is first input into a 3×3 convolution with a stride of 1 to obtain a low-level feature map with 32 channels. In the encoder, each layer contains two 3×3 convolutions with a stride of 1, followed by a 3×3 down-convolution with a stride of 2. In the decoder, each layer contains a 2×2 up-convolution with a stride of 2, followed by two 3×3 up-convolutions. 3 convolutions with a stride of 1. Finally, the feature map of the last decoder layer is input into a 1×1 convolution with a stride of 1 to output a segmentation map, which is the lesion area.

[0033] After the lesion area is identified, the diabetic retinopathy labeling model identifies the lesion type of the lesion area, and then labels the identified lesion type to the corresponding lesion area.

[0034] The multimodal integrated display module 103, which is respectively connected to the data acquisition module 101 and the image annotation module 102, is used to fuse different types of fundus image data in the initial multimodal fundus image data set based on an image fusion algorithm to obtain a target multimodal fundus image data set.

[0035] In the embodiment of the present invention, the multimodal integrated display module 103 can integrate image data from a variety of detection devices such as a wide-angle fundus camera, an OCT device, an OCTA device, and an FFA device, and synchronously present image data of different modalities, thus breaking through the regional limitations of traditional examinations.

[0036] In practical applications, the fusion of different types of fundus image data in the initial multimodal fundus image dataset can be achieved in the following ways: Identify the position coordinates of each lesion area in each image data; For each lesion area, the position coordinates of the lesion area are associated with the position coordinates of each image data to obtain the target multimodal fundus image data set, so that the fundus image data of any area of ​​the entire fundus of the user to be tested and the corresponding prediction and annotation results can be displayed on the display interface of the user terminal based on the target multimodal fundus image data set. The position coordinates of the lesion area are the same as the position coordinates of each image data.

[0037] In practical applications, the number of clock points can be used to indicate the location of the lesion.

[0038] The above method can be used to fuse image data of different modalities and then present them synchronously on the display interface of the user terminal. The user can choose to display the lesion area in any area of ​​the entire fundus (including the posterior pole and peripheral part), breaking the limitation of the area and image singularity of traditional examination.

[0039] The user interaction module 104 connected to the image annotation module 102 and the multimodal integrated display module 103 is used to display the fundus image data of any area of ​​the entire fundus of the user to be tested and the corresponding predicted annotation results on the display interface of the user terminal based on the target multimodal fundus image data set.

[0040] In the embodiment of the present invention, the user can click on the fundus image data of any area of ​​the entire fundus on the display interface, and the lesion area and lesion type contained in the area can be displayed on the display interface.

[0041] It should be noted that the user interaction module 104 can also be directly connected to the data acquisition module 101, and the user can separately display at least one of the full fundus image data, posterior pole and macular area image data and retinal vascular image data of the user to be examined on the display interface of the user terminal.

[0042] Figure 2 Schematic diagram of the display interface of the user terminal provided by the present invention. Figure 2 As shown, the display interface includes a full fundus image display area (indicated by ①), a full fundus image data display area (indicated by ②), a posterior pole and macular area image data display area (indicated by ③), and a retinal vascular image data display area (indicated by ④).

[0043] The whole fundus image is shown in display area ①, in which there are multiple lesion areas. If the user clicks on lesion area A, the image of lesion area A and its surrounding area in the whole fundus image data is displayed in display area ②, the image of lesion area A and its surrounding area in the posterior pole and macular area image data is displayed in display area ③, and the image of lesion area A and its surrounding area in the retinal vascular image data is displayed in display area ④.

[0044] In some possible implementations, if a certain display area cannot display the image of the lesion area A due to reasons such as the data acquisition module 101 not acquiring relevant image data, then the image may not be displayed in the display area.

[0045] It is understandable that the user can also click on the non-lesion area, and the whole fundus image data, posterior pole and macular area image data, and retinal vascular image data corresponding to the non-lesion area can also be displayed.

[0046] In the diabetic retinopathy full fundus dynamic labeling system provided by the present invention, a data acquisition module is connected to a wide-angle fundus camera, an OCT device, an OCTA device and an FFA device respectively through a standard interface to collect an initial multimodal fundus image data set of a user to be tested, including full fundus image data, posterior pole and macular area image data, and retinal vascular image data; then the initial multimodal fundus image data set is input into a diabetic retinopathy labeling model by using an image labeling module, and the diabetic retinopathy labeling model can identify the lesion area of ​​multiple types of fundus image data in the multimodal fundus image data set and label the corresponding lesion type, thereby obtaining a predicted labeling result; a multimodal integrated display module fuses different types of fundus image data in the initial multimodal fundus image data set based on an image fusion algorithm to obtain a target multimodal fundus image data set; a user interaction module, based on the target multimodal fundus image data set, displays the fundus image data of any area of ​​the full fundus of the user to be tested and the corresponding predicted labeling result on the display interface of the user terminal. Through the above system, the diabetic retinopathy annotation model is used to realize the lesion area identification and lesion type annotation of various types of fundus image data, so that doctors can clearly and quickly understand the progression of the disease without manual experience; in addition, the whole fundus image data, posterior pole and macular area image data, and retinal vascular image data marked with lesion areas and lesion types are fused, and the lesion area and lesion type of any area of ​​the whole fundus of the user to be examined can be displayed on the display interface of the user terminal, eliminating visual blind spots. Users can view any area of ​​interest in the whole fundus, thereby realizing a comprehensive analysis of diabetic retinopathy and comprehensively improving the diagnosis and treatment efficiency of diabetic retinopathy.

[0047] Figure 3 This is the second structural schematic diagram of the diabetic retinopathy full fundus dynamic marking system provided by the present invention, see Figure 3 As shown, in Figure 1 Based on the system structure shown, Figure 3 The system shown also includes: a lesion comparison and analysis module 105 connected to the image annotation module 102, which is used to use a feature point matching algorithm to compare the predicted annotation results with the historical annotation results of the historical follow-up images of the user to be inspected to obtain lesion change information; the lesion change information includes the change information of the lesion area and lesion type in the predicted annotation results and the historical annotation results; based on the lesion change information, the characteristic parameters of the lesion change information are calculated using statistical and machine learning algorithms; based on the characteristic parameters of the lesion area, a follow-up report of the user to be inspected is generated.

[0048] In the embodiment of the present invention, the lesion comparison and analysis module 105 can dynamically compare the follow-up images of different periods, and realize the accurate positioning and trend analysis of lesion changes through the feature point matching algorithm. That is, by comparing the predicted annotation results with the historical annotation results of the historical follow-up images, according to the changes of the patient's lesion area and lesion type, a trend chart and a quantitative follow-up report are generated, and each examination result is automatically recorded and intelligently analyzed, providing a quantitative disease change report to assist clinical decision-making.

[0049] Specifically, by comparing the predicted annotation results with the historical annotation results, it can be determined whether the size, position, etc. of the lesion area in the predicted annotation results have changed compared with the historical annotation results, and whether the lesion type has changed. For example, the size of the lesion area A changes from 1 cm 3 Enlarged to 2cm 3 For example, in the historical annotation results, a certain area B has no lesions, but in the predicted annotation results, a certain area B has lesions, and the size of the lesion in the area B is 1cm 3 Based on these key lesion change information, the change trend of the lesion can be determined. The present invention realizes automatic comparison of dynamic changes of lesions through the lesion comparison and analysis module 105, thereby improving the accuracy and efficiency of disease condition change assessment.

[0050] Optionally, the lesion comparison and analysis module 105 uses a feature point matching algorithm to compare the predicted annotation result with the historical annotation result of the historical follow-up image of the user to be examined to obtain lesion change information, which is specifically implemented in the following manner: Step 1), dividing the predicted annotation results and the historical follow-up images into multiple sub-regions of the same size and number; Step 2), for each sub-region, use a feature point matching algorithm to identify the boundaries of the lesion area of ​​the predicted annotation result and the historical annotation result in the sub-region, and obtain a first boundary corresponding to the lesion area of ​​the predicted annotation result in the sub-region and a second boundary corresponding to the lesion area of ​​the historical annotation result; Step 3), based on the predicted labeling result and the lesion type corresponding to the historical follow-up image, and the coordinate information corresponding to the first boundary and the second boundary, determine the lesion change information.

[0051] In the embodiment of the present invention, the fundus image data corresponding to the predicted labeling result includes the whole fundus image data, the posterior pole and macular image data, and the retinal vascular image data. The above fundus image data are divided into a plurality of sub-areas respectively. Similarly, the historical whole fundus image data, the historical posterior pole and macular image data, and the historical retinal vascular image data included in the historical follow-up images are also divided into a plurality of sub-areas of the same size and quantity.

[0052] For each type of image data (taking full fundus image data and historical full fundus image data as examples), for sub-regions at the same position in the full fundus image data and the historical full fundus image data, a feature point matching algorithm is used to identify the boundaries of the diseased areas in the two sub-regions, and the first boundary corresponding to the diseased area of ​​the full fundus image data and the second boundary corresponding to the diseased area of ​​the historical full fundus image data are obtained.

[0053] By comparing the coordinate information of the first boundary with the coordinate information of the second boundary, the lesion change information such as the position, size, and number of the lesion areas in the two sub-areas can be determined.

[0054] Optionally, the characteristic parameters include: the change values ​​of the size, quantity and position coordinates of the lesion area between the predicted annotation result and the historical annotation result; The lesion comparison and analysis module 105 generates a follow-up report of the user to be inspected according to the characteristic parameters of the lesion area, which is specifically implemented in the following manner: The follow-up report is generated according to each of the change values; the follow-up report includes a lesion trend graph of the user to be examined, which is used to reflect the change trend of the lesion area and lesion type.

[0055] In the embodiment of the present invention, the lesion comparison and analysis module 105 can calculate key parameters such as the size and quantity changes of the lesion area in real time, and generate a follow-up report to assist doctors in judging the progression of the disease.

[0056] Optionally, the user interaction module 104 is connected to the lesion comparison and analysis module 105; the user interaction module 104 is also used to display the follow-up report on the display interface.

[0057] Optionally, the user interaction module 104 is further configured to: When the change value of at least one of the size, number and position coordinates of the lesion area exceeds a preset threshold, a prompt message is displayed on the display interface; the prompt message is used to reflect that at least one of the size, number and position of the lesion area has improved or worsened.

[0058] In the embodiment of the present invention, the change value increases beyond a preset threshold, for example, the area increases from 1 cm 3 Change to 3cm 3 , indicating that the lesion area has significantly deteriorated; the change value has decreased and exceeded the preset threshold, for example, the area has increased from 3cm 3 Change to 1cm 3 , indicating that the lesion area has improved significantly.

[0059] In the above embodiments, the system provided by the present invention has an intelligent reminder function, which can automatically remind the user when the lesion changes significantly, and assist clinical decision-making. This function can help doctors timely discover the trend of worsening or improvement of the disease and effectively intervene in the right time, but the existing technology lacks such intelligent decision-making assistance function.

[0060] Optionally, the user interaction module 104 is further configured to: In response to a first operation instruction of the user on the display interface, the predicted annotation result is modified and / or an annotation result is added.

[0061] In an embodiment of the present invention, a user-friendly operation interface is provided, allowing the user to perform manual corrections on the basis of automatic annotation, that is, allowing manual annotation to be combined with system automatic annotation, combining artificial intelligence with clinical experience, and further improving the accuracy and clinical applicability of annotations. This combination of manual and automatic annotations greatly makes up for the limitations of a single method. It supports manual annotation adjustment, dynamic display of images, and convenient switching of various types of imaging information. The module is designed with intuitive interactive tools to facilitate physicians to operate and correct annotation results.

[0062] Optionally, the user interaction module 104 is further configured to: In response to a second operation instruction of the user on the display interface for the fundus image data of any area of ​​the entire fundus, the predicted labeling result and the historical labeling result corresponding to the fundus image data of any area of ​​the entire fundus are displayed.

[0063] In the embodiment of the present invention, the user can click on any area of ​​the entire fundus in the fundus image data, and not only the predicted annotation result of the area can be displayed, but also the historical annotation result of the area can be displayed.

[0064] That is, in the above implementation, it includes overall follow-up function and detailed follow-up. The overall follow-up can retrieve the patient's current and past fundus lesion images at any time; the detailed follow-up, that is, clicking on a certain lesion area, can simultaneously display the previous lesion conditions at the lesion site.

[0065] Optionally, the user interaction module 104 is further configured to: In response to a third operation instruction of the user on the display interface, displaying a normal full fundus image atlas on the display interface; In response to a fourth operation instruction of the user on the display interface for the normal whole fundus image atlas, a manual annotation result is added to the normal whole fundus image atlas; the manual annotation result includes a whole fundus image atlas annotated with lesion areas and lesion types.

[0066] In an embodiment of the present invention, a doctor can manually add manual annotation results to a normal full fundus image atlas according to the patient's lesion condition and the fundus retinal lesion record.

[0067] It should be noted that a toolbar is provided in the display interface, and different lesion types are provided in the toolbar. The user can click on the lesion area and then click on the lesion type in the toolbar to mark the lesion area.

[0068] In addition, the toolbar also has a text box for text annotation. Each lesion annotation on the atlas can be used to add text to add special instructions for the lesion. The number of points on the clock indicates the location of the lesion.

[0069] In summary, the diabetic retinopathy full fundus dynamic annotation system provided by the present invention has the following advantages: 1. Accurate dynamic lesion annotation and comparison: The system provided by the present invention adopts intelligent dynamic annotation technology, which can automatically identify and annotate various lesion forms of diabetic retinopathy (such as microaneurysms, exudation, hemorrhage, etc.), and automatically compare and analyze the images at different time points during the follow-up process. Compared with the existing technology that relies on the naked eye judgment of doctors, the present invention greatly improves the accuracy of lesion annotation and the efficiency of comparison, and reduces human errors.

[0070] 2. Multimodal image integration and comprehensive presentation: The system provided by the present invention can integrate a variety of imaging data such as wide-angle fundus photography, OCT, OCTA and FFA, and supports the selection and display of any area of ​​the entire fundus (including the posterior pole and peripheral part). This multimodal image integration breaks through the limitation of traditional methods that are limited to a single detection method and area, and provides more comprehensive and three-dimensional fundus lesion information, which is helpful for comprehensive assessment of the condition.

[0071] 3. Personalized follow-up management and quantitative analysis: The system provided by the present invention can generate a personalized follow-up plan based on the patient's specific lesion conditions, automatically record and analyze follow-up data, and provide quantitative disease change reports and trend analysis. This function not only improves the systematic and targeted nature of follow-up, but also provides doctors with a scientific basis for decision-making. The existing technology lacks such systematic and personalized follow-up management capabilities.

[0072] 4. Intelligent change reminder and auxiliary decision support: The system provided by the present invention has an intelligent reminder function, which can automatically remind doctors when the lesion changes significantly and assist clinical decision-making. This function can help doctors to timely discover the trend of worsening or improvement of the disease.

[0073] 5. Combining manual and automatic annotation to improve annotation accuracy: The system provided by the present invention allows doctors to perform manual corrections based on automatic annotation, combining artificial intelligence with clinical experience to further improve the accuracy and clinical applicability of annotation. This combination of manual and automatic annotation greatly makes up for the possible limitations of a single method.

[0074] 6. Reduce the workload of doctors and improve clinical efficiency: Through intelligent image processing and annotation comparison, the system provided by the present invention significantly reduces the time doctors spend on image interpretation and disease analysis, improves the efficiency of clinical diagnosis and follow-up, and reduces the workload of doctors.

[0075] In summary, the present invention provides advantages for the diagnosis and management of diabetic retinopathy and significantly improves its clinical application value through intelligent integration of multimodal images, accurate lesion annotation and dynamic comparison, personalized follow-up management, and intelligent decision-making support functions.

[0076] The whole fundus dynamic labeling method for diabetic retinopathy provided by the present invention is described below. The whole fundus dynamic labeling method for diabetic retinopathy described below and the whole fundus dynamic labeling system for diabetic retinopathy described above can be referenced to each other. Figure 4 FIG. 1 is a flow chart of a method for dynamic marking of the whole fundus of diabetic retinopathy provided by the present invention, such as Figure 4 As shown, the method for dynamic marking of the whole fundus of diabetic retinopathy includes the following steps: Step 401 : collecting an initial multimodal fundus image data set of the user to be tested; wherein the multimodal fundus image data set includes full fundus image data, posterior pole and macular area image data, and retinal vascular image data.

[0077] Step 402: input the initial multimodal fundus image data set into the diabetic retinopathy annotation model to obtain the predicted annotation results output by the diabetic retinopathy annotation model; wherein the predicted annotation results include fundus image data annotated with lesion areas and lesion types; the diabetic retinopathy annotation model is trained based on full fundus image sample data, posterior pole and macular area image sample data, and retinal vascular image sample data of multiple users, and corresponding sample annotation results; the diabetic retinopathy annotation model is used to annotate lesions of various types of fundus image data in the multimodal fundus image data set.

[0078] Step 403: fuse different types of fundus image data in the initial multimodal fundus image data set based on an image fusion algorithm to obtain a target multimodal fundus image data set.

[0079] Step 404: Based on the target multimodal fundus image data set, the fundus image data of any area of ​​the entire fundus of the user to be tested and the corresponding prediction and annotation results are displayed on the display interface of the user terminal.

[0080] The method for dynamic labeling of the entire fundus of diabetic retinopathy provided by the present invention utilizes a diabetic retinopathy labeling model to realize lesion area identification and lesion type labeling for various types of fundus image data, so that doctors can clearly and quickly understand the progression of the disease without relying on manual experience; and, the entire fundus image data, posterior pole and macular area image data, and retinal vascular image data labeled with lesion areas and lesion types are fused, and the lesion area and lesion type of any area of ​​the entire fundus of the user to be examined can be displayed on the display interface of the user terminal, eliminating visual blind spots, and the user can view any area of ​​interest in the entire fundus, thereby realizing a comprehensive analysis of diabetic retinopathy and comprehensively improving the diagnosis and treatment efficiency of diabetic retinopathy.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A diabetic retinopathy full fundus dynamic labeling system, characterized in that: The system comprises: A data acquisition module, connected to the wide-angle fundus camera, the OCT device, the OCTA device and the FFA device through standard interfaces, for acquiring an initial multimodal fundus image data set of the user to be examined; the multimodal fundus image data set includes full fundus image data, posterior pole and macular area image data, and retinal vascular image data; An image annotation module connected to the data acquisition module is used to input the initial multimodal fundus image data set into a diabetic retinopathy annotation model to obtain a predicted annotation result output by the diabetic retinopathy annotation model; the predicted annotation result includes fundus image data annotated with lesion areas and lesion types; the diabetic retinopathy annotation model is trained based on full fundus image sample data, posterior pole and macular area image sample data, and retinal vascular image sample data of multiple users, and corresponding sample annotation results; the diabetic retinopathy annotation model is used to perform lesion annotation on multiple types of fundus image data in the multimodal fundus image data set; a multimodal integrated display module connected to the data acquisition module and the image annotation module respectively, for fusing different types of fundus image data in the initial multimodal fundus image data set based on an image fusion algorithm to obtain a target multimodal fundus image data set; A user interaction module connected to the image annotation module and the multimodal integrated display module is used to display the fundus image data of any area of ​​the entire fundus of the user to be tested and the corresponding predicted annotation results on the display interface of the user terminal based on the target multimodal fundus image data set.

2. The diabetic retinopathy whole fundus dynamic marking system according to claim 1 is characterized in that: The system further comprises: The lesion comparison and analysis module connected to the image annotation module is used to use a feature point matching algorithm to compare the predicted annotation results with the historical annotation results of the historical follow-up images of the user to be inspected to obtain lesion change information; the lesion change information includes the change information of the lesion area and lesion type in the predicted annotation results and the historical annotation results; based on the lesion change information, the characteristic parameters of the lesion change information are calculated using statistical and machine learning algorithms; based on the characteristic parameters of the lesion area, a follow-up report of the user to be inspected is generated.

3. The diabetic retinopathy full fundus dynamic annotation system according to claim 2, characterized in that: The lesion comparison and analysis module is specifically used for: Segmenting the predicted annotation results and the historical follow-up images into a plurality of sub-regions of the same size and number; For each sub-region, a feature point matching algorithm is used to identify the boundary of the lesion region of the predicted annotation result and the historical annotation result in the sub-region, and obtain a first boundary corresponding to the lesion region of the predicted annotation result and a second boundary corresponding to the lesion region of the historical annotation result in the sub-region; The lesion change information is determined based on the predicted labeling result and the lesion type corresponding to the historical follow-up image, and the coordinate information corresponding to the first boundary and the second boundary.

4. The diabetic retinopathy full fundus dynamic annotation system according to claim 2, characterized in that: The characteristic parameters include: the change value of the size, quantity and position coordinates of the lesion area compared with the predicted annotation result and the historical annotation result; The lesion comparison and analysis module is specifically used for: The follow-up report is generated according to each of the change values; the follow-up report includes a lesion trend graph of the user to be examined, which is used to reflect the change trend of the lesion area and lesion type.

5. The diabetic retinopathy whole fundus dynamic marking system according to claim 4, characterized in that: The user interaction module is also used for: When the change value of at least one of the size, number and position coordinates of the lesion area exceeds a preset threshold, a prompt message is displayed on the display interface; the prompt message is used to reflect that at least one of the size, number and position of the lesion area has improved or worsened.

6. The diabetic retinopathy whole fundus dynamic annotation system according to any one of claims 2 to 4, characterized in that: The user interaction module is connected to the lesion comparison and analysis module; The user interaction module is also used to display the follow-up report on the display interface.

7. The diabetic retinopathy whole fundus dynamic annotation system according to claim 1, characterized in that: The user interaction module is further used for: In response to a first operation instruction of the user on the display interface, the predicted annotation result is modified and / or an annotation result is added.

8. The diabetic retinopathy whole fundus dynamic annotation system according to claim 1, characterized in that: The user interaction module is further used for: In response to a second operation instruction of the user on the display interface for the fundus image data of any area of ​​the entire fundus, the predicted labeling result and the historical labeling result corresponding to the fundus image data of any area of ​​the entire fundus are displayed.

9. The diabetic retinopathy full fundus dynamic annotation system according to claim 1, characterized in that: The user interaction module is further used for: In response to a third operation instruction of the user on the display interface, displaying a normal full fundus image atlas on the display interface; In response to a fourth operation instruction of the user on the display interface for the normal whole fundus image atlas, a manual annotation result is added to the normal whole fundus image atlas; the manual annotation result includes a whole fundus image atlas annotated with lesion areas and lesion types.

10. The diabetic retinopathy whole fundus dynamic annotation system according to any one of claims 1 to 5 and 7 to 9, characterized in that: The lesion type includes at least one of the following: Microaneurysm, hard exudates, soft exudates, retinal hemorrhage, tortuosity and dilatation of retinal veins, neovascularization, vitreous hemorrhage, macular edema, epiretinal membrane, retinal detachment, hard exudate rings, optic disc neovascularization, retinal arteriosclerosis, fibrovascular proliferation and diabetic maculopathy.

Images