Method for obtaining retinal topography and eye disease diagnosis device

Abstract

Embodiments of the present application provide a method for obtaining a retinal topography map and an apparatus for diagnosing eye diseases. The method comprises: obtaining a retinal fundus image; inputting the retinal fundus image into a target image generator, and obtaining a retinal topography map via the target image generator; and providing the retinal topography map. The method of the embodiments of the present application can provide doctors with more retinal topography maps for diagnosing eye diseases in patients. The apparatus of the embodiments of the present application can employ software methods to generate topography maps that facilitate subsequent diagnosis by doctors in combination with other eye images. In other words, the topography maps obtained in some embodiments of the present application can be used to assist doctors in diagnosing and treating diseases.

Description

Technical Field

[0001] The present application relates to the field of eye image acquisition. Specifically, embodiments of the present application relate to a method for acquiring retinal topography and an eye disease diagnosis device. Background Art

[0002] Macular edema (ME) is one of the most common retinal diseases and can be caused by many eye diseases, such as diabetic retinopathy (DR), retinal vein occlusion (RVO), and epiretinal membrane (ERM). Patients with ME often experience varying degrees of visual distortion and decreased central vision, which seriously affects their quality of life. ME is very harmful to patients, and early intervention can effectively control the progression of the disease. However, the diagnosis of ME mainly relies on expensive equipment such as optical coherence tomography (OCT) and fundus fluorescein angiography (FFA), which is too difficult to implement in primary hospitals. Compared with OCT and FFA, fundus photography has excellent equipment in primary hospitals and even community hospitals in China, but for ophthalmologists, judging the presence of ME based solely on fundus photography requires a high level of professionalism.

[0003] Therefore, how to enrich the types and quantity of fundus photographs has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method for obtaining retinal topography maps and an eye disease diagnosis device. The method of the embodiments of the present application can provide doctors with more retinal topography maps for diagnosing eye diseases in patients. The device of the embodiments of the present application can combine the synthesized retinal topography map and retinal eye map to output preliminary diagnostic results for eye diseases, which can be used to assist doctors in diagnosing and treating diseases.

[0005] In a first aspect, an embodiment of the present application provides a method for obtaining a retinal topography map, the method comprising: obtaining a retinal fundus image; inputting the retinal fundus image into a target image generator, and obtaining a retinal topography map through the target image generator; and providing the retinal topography map.

[0006] Some embodiments of the present application generate a retinal topography map based on a retinal eye map through an image generator, which can effectively overcome the problem of being unable to obtain a retinal topography map.

[0007] In some embodiments, the target image generator is obtained by training a generative adversarial network.

[0008] In some embodiments of the present application, a target image generator is obtained by training a generative adversarial network.

[0009] In some embodiments, the generative adversarial network includes a generative model and an adversarial model, wherein the feature extraction network of the generative model adopts ResNet50, the activation function of the adversarial model is the deep learning activation function Mish function, and the optimizer of the adversarial model is the deep learning optimizer Ranger optimizer.

[0010] Some embodiments of the present application combine the above-mentioned feature extraction network, activation function and optimizer, which can reduce the amount of calculation and make feature extraction richer, model training faster, and the final synthesized retinal topography more realistic, making model training more stable.

[0011] In some embodiments, the generative adversarial network is trained using training data processed by the following strategy: the same area is extracted from a real retinal fundus image and a real retinal topography map, set to the same size, and the background is uniformly adjusted to white to obtain a preliminary preprocessed image; the image is rotated multiple times to obtain the training data; wherein the target image generator is obtained by training the generative adversarial network based on the training data.

[0012] Some embodiments of the present application improve the quality and quantity of images in a training dataset by preprocessing and expanding the data, thereby being able to train a target image generator with better performance.

[0013] In a second aspect, some embodiments of the present application provide an eye disease diagnosis device, comprising: a retinal topography map acquisition module, configured to acquire a retinal topography map using the methods of the embodiments of the first aspect described above; and a disease diagnosis result acquisition module, configured to acquire a diagnosis result based on the retinal topography map and a retinal eye map paired with the retinal topography map.

[0014] In some embodiments, the disease diagnosis result acquisition module obtains the diagnosis result based on a deep learning network, wherein the deep learning network.

[0015] In some embodiments, the eye disease diagnosis apparatus further comprises: an output template configured to output the diagnosis result.

[0016] In a third aspect, some embodiments of the present application provide a device for obtaining a retinal topography map, the device comprising: a retinal fundus image acquisition module, configured to obtain a retinal fundus image; a retinal topography map acquisition module, configured to input the retinal fundus image into a target generator and obtain a retinal topography map through the target generator; and an output module, configured to provide the retinal topography map.

[0017] In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect above.

[0018] In a fifth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor can implement the method described in the embodiment of the first aspect when executing the program. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of the structure and training process of a generative adversarial network provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a method for obtaining a retinal topography map provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the network architecture of the method for obtaining retinal topography provided in an embodiment of the present application;

[0023] Figure 4 A block diagram of the eye disease diagnosis device provided in an embodiment of the present application;

[0024] Figure 5 A block diagram of the apparatus for obtaining retinal topography provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0027] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0028] The retinal topography map in the embodiments of the present application refers to an image used to characterize the topography of different regions on the retinal surface. For example, in related art, a specific light source can be used to perform a three-dimensional scan of the retina. By quantitatively analyzing the retinal scan image, the relative height values ​​of the retinal surface are obtained, thereby depicting the topography of different regions on the retinal surface to obtain a retinal topography map.

[0029] Please see Figure 1 , Figure 1 This is a structural diagram of the generative adversarial network 100 provided in the embodiment of the present application, combined with Figure 1 It is not difficult to find that the generative adversarial network 100 includes a generative network 110 (also called a generator) and a discriminative network 120 (also called a discriminator).

[0030] It is understandable that in order to obtain a target image generator capable of synthesizing a retinal topography map according to a retinal fundus map, it is necessary to Figure 1 The generative adversarial network 100 is trained. During the training process, it is necessary to obtain the real retinal fundus image and the real retinal topography map in advance, and then input these images into the generative adversarial network to train the network model. When the training is completed, the target image generator is obtained.

[0031] The following is an example of the process of training a generative adversarial network.

[0032] First, the dataset (image) and its preprocessing are used to obtain the training dataset.

[0033] The generative adversarial network is trained using training data processed by the following strategy: the same area is extracted from a real retinal fundus image and a real retinal topography map, set to the same size, and the background is uniformly adjusted to white to obtain a preliminary preprocessed image; the image is rotated multiple times to obtain the training data; wherein the target image generator is obtained by training the generative adversarial network based on the training data.

[0034] For example, based on the framing part of the retinal fundus map generated in the device, the same area is extracted from the original retinal fundus map (i.e., the real retinal eye map obtained by real shooting) and the retinal topography map (i.e., the real retinal topography map obtained by real shooting), set to the same size, and the background is uniformly adjusted to white. Then data expansion is performed, and the retinal fundus map and the topography map are rotated by the same angle at the same time through multiple angle rotations. The retinal fundus map and retinal topography map after data expansion are used as training data sets, without annotation, and are input into the GAN model (i.e. Figure 1 In the generative adversarial network 100, as Figure 1 The GAN model shown includes a generation network G and a discriminant model D (or called a discriminative network).

[0035] A real retinal fundus image (a retinal fundus image in the training set obtained after the above preprocessing) is randomly input into the generation network 110, and the generation network 110 will output a fake sample set. That is to say, during the training process, the real sample set retinal fundus image and retinal topography map are simultaneously input into the GAN model to obtain true and false data sets. Then, the true and false data sets are input into the discriminant model D for training. The model is similar to a supervised two-classification problem. It judges the retinal topography sample input into the discriminant model D to determine whether it is real (true sample) or generated by the generation network D (false sample). After the training of the discriminant model D, the model continuously improves the ability of the generation model G to generate images similar to the real retinal topography. Then, the generation network G is connected in series with the trained judgment network D, and a random input is continued to be input to the generation network. The loss function is to determine whether the output of the network D is true (whether it is similar to the real retinal topography). According to the loss function, the parameters of the generation network D are updated, and at the same time, the parameters of the judgment network D are not updated. After the training of the generation network G, the model is randomly input again to obtain a new false data set, and the latest true and false retinal topography map samples are input into the judgment network D for training, thereby completing the training of the judgment network D again.

[0036] Repeat the above process until the number of training times is reached, the model training is completed, and the target image generator is obtained.

[0037] The new retinal fundus image is then pre-processed and fed into the previously trained GAN model (specifically, the target image generator obtained by training the generative network 110) to generate a retinal topography map that matches (or is paired with) the input retinal fundus image. It is understood that combining the generated retinal topography map with the matched original retinal fundus image can assist doctors in diagnosing eye diseases in patients.

[0038] It should be noted that Figure 1 The real retinal eye map is input into the generative network, which obtains a predicted retinal topography map based on the input image during training. The real retinal topography map and the predicted retinal topography map corresponding to the input real retinal fundus map are then input into the discriminative network 120 to obtain a similarity value. The parameters of the generative network 110 are then adjusted according to the similarity value until the difference between the predicted retinal topography map and the real retinal topography map is as small as possible (i.e., the loss function is satisfied). The target image generator is obtained. In some embodiments of the present application, the activation function of the discriminative network 120 is the Mish function and the optimizer is the Ranger optimizer, which is different from the Relu activation function and Adam optimizer in the related art.

[0039] Some embodiments of the present application use the Mish function activation to ensure that there is no saturation area, so there will be no problem of gradient disappearance during training. The smooth activation function allows better information to penetrate deep into the neural network, resulting in better accuracy and generalization. The Mish activation function is boundless (positive value) and avoids saturation due to capping. The slight tolerance for negative values ​​theoretically allows better gradient flow, rather than a hard zero boundary like in ReLU. Using Mish activation makes model training more stable. In addition, some embodiments of the present application use the Ranger optimizer because this optimizer combined with the corresponding selected activation function makes the network more stable and robust. For example, the Ranger optimizer combines two new developments (RAdam+Lookahead) into a single optimizer. RAdam provides the best foundation for the optimizer when starting training. RAdam uses a dynamic rectifier to adjust the adaptive momentum of Adam according to changes, effectively providing an automatic warm-up for the current dataset to ensure a solid training start. Lookahead reduces variance during training, significantly reduces sensitivity to suboptimal hyperparameters, and reduces the need for extensive hyperparameter tuning, enabling faster convergence during training.

[0040] The following combination Figure 2 and Figure 3 Exemplary description of some embodiments of the present application according to the target image generator (ie using Figure 1 The process of obtaining the retinal topography map by training the generative adversarial network in a retinal manner (the network corresponding to the generative network is obtained).

[0041] like Figure 2 As shown, an embodiment of the present application provides a method for obtaining a retinal topography map, the method comprising: S101, obtaining a retinal fundus image; S102, inputting the retinal fundus image into a target image generator, and obtaining a retinal topography map through the target image generator; and S103, providing the retinal topography map.

[0042] That is to say, if Figure 3 As shown, the acquired retinal fundus image is input into the target image generator 10 to obtain and output a retinal topography map that is paired with or matches the input image. Some embodiments of the present application generate a retinal topography map based on the retinal eye map by an image generator, which can effectively overcome the problem of being unable to obtain a retinal topography map.

[0043] It is understandable that the target image generator 10 uses the training set to Figure 1The generative network is obtained by training the generative adversarial network of the generative model. It should be noted that, unlike the related technical solutions, the feature extraction network included in the target generator 10 of some embodiments of the present application adopts ResNet50. As described above, in order to increase the training speed as much as possible and obtain high-quality retinal topography, the generative adversarial network of some embodiments of the present application includes a generative model and an adversarial model, wherein the feature extraction network of the generative model adopts ResNet50, the activation function of the adversarial model is the deep learning activation function Mish function, and the optimizer of the adversarial model is the deep learning optimizer Ranger optimizer.

[0044] Some embodiments of the present application combine the above-mentioned feature extraction network, activation function and optimizer, which can reduce the amount of computation while also making feature extraction richer, model training faster, and the resulting synthesized retinal topography more realistic, making model training more stable. In other words, some embodiments of the present application do not use the VGG16 in the GAN network for feature extraction, but instead use the ResNet50 model to extract features from the input retinal eye map. This can reduce the amount of computation while also making feature extraction richer, model training faster, and the resulting synthesized retinal topography more realistic.

[0045] like Figure 4 As shown, some embodiments of the present application provide an eye disease diagnosis device, which includes: a retinal topography acquisition module 101 and a disease diagnosis result acquisition module 102.

[0046] The retinal topography acquisition module 101 is configured to adopt Figure 2 The method described in the examples obtains retinal topography.

[0047] The disease diagnosis result acquisition module 102 is configured to acquire a diagnosis result based on the retinal topography map obtained by the retinal topography map acquisition module 101 and a retinal eye map paired with the retinal topography map.

[0048] In order to assist doctors in diagnosing and treating diseases, in some embodiments of the present application, Figure 4 The eye disease diagnosis device further includes: an output template 103, which is configured to output the diagnosis result.

[0049] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0050] Please refer to Figure 5 , Figure 5The apparatus for obtaining retinal topography provided by the embodiment of the present application is shown. It should be understood that the apparatus is similar to the above-mentioned apparatus. Figure 2 The method embodiment corresponds to the method embodiment and can execute each step involved in the above method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, a detailed description is omitted here. The device includes at least one software functional module that can be stored in a memory in the form of software or firmware or embedded in the device's operating system. The device for obtaining a retinal topography map includes: a retinal fundus image acquisition module 201, a retinal topography map acquisition module 202, and an output module 203.

[0051] The retinal fundus image acquisition module 201 is configured to acquire a retinal fundus image.

[0052] The retinal topography map acquisition module 202 is configured to input the retinal fundus image into a target generator, and obtain a retinal topography map through the target generator.

[0053] The output module 203 is configured to provide the retinal topography map.

[0054] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0055] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which can implement the above-mentioned Figure 2 The method described in any embodiment of the method.

[0056] like Figure 6 Some embodiments of the present application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 via a bus 530 and executes the program to achieve the above Figure 2 The method described in the corresponding embodiment.

[0057] Processor 520 can process digital signals and can include various computing architectures, such as a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0058] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the present disclosure may be used to execute the instructions in the memory 510 to implement Figure 2 The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0059] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0060] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0061] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0062] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0063] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0064] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A method for obtaining a retinal topography map, characterized in that: The method comprises: Acquire retinal fundus images; Inputting the retinal fundus image into a target image generator, and obtaining a retinal topography map through the target image generator; providing the retinal topography map; Among them, the target image generator is obtained by training a generative adversarial network, including: extracting the same area from a real retinal fundus image and a real retinal topography map, and setting the extracted same area to the same size and uniformly adjusting the background to white to obtain a preliminary preprocessed image; performing multiple angle rotations on the preliminary preprocessed image to obtain training data; training the generative adversarial network based on the training data to obtain the target image generator, the generative adversarial network includes a generative model and an adversarial model, the feature extraction network of the generative model adopts ResNet50, the activation function of the adversarial model is a deep learning activation function Mish function, and the optimizer of the adversarial model is a deep learning optimizer Ranger optimizer.

2. An eye disease diagnosis device, characterized in that: The eye disease diagnosis device comprises: a retinal topography acquisition module, configured to acquire a retinal topography using the method according to claim 1; The disease diagnosis result acquisition module is configured to acquire a diagnosis result based on the retinal topography map and a retinal eye map paired with the retinal topography map.

3. The eye disease diagnosis device according to claim 2, wherein: The eye disease diagnosis device further includes an output module configured to output the diagnosis result.

4. A device for obtaining retinal topography, characterized in that: The device comprises: a retinal fundus image acquisition module, configured to acquire a retinal fundus image; a retinal topography acquisition module, configured to input the retinal fundus image into a target generator and obtain a retinal topography map through the target generator; an output module configured to provide the retinal topography map; Among them, the target image generator is obtained by training a generative adversarial network, including: extracting the same area from a real retinal fundus image and a real retinal topography map, and setting the extracted same area to the same size and uniformly adjusting the background to white to obtain a preliminary preprocessed image; performing multiple angle rotations on the preliminary preprocessed image to obtain training data; training the generative adversarial network based on the training data to obtain the target image generator, the generative adversarial network includes a generative model and an adversarial model, the feature extraction network of the generative model adopts ResNet50, the activation function of the adversarial model is a deep learning activation function Mish function, and the optimizer of the adversarial model is a deep learning optimizer Ranger optimizer.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to claim 1 can be implemented.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to claim 1 can be implemented.

Images