Method and device for supporting reading fundus images, and system including the device

Abstract

The present application provides a method for supporting reading fundus images and a device utilizing the same, the method comprising the following steps: based on a machine learning model, extracting first attribute information from a first fundus image of an object, mapping an adversarial factor to the first fundus image to change the first attribute information, generating a second fundus image with second attribute information, and providing the first fundus image with the first attribute information and the second fundus image with the second attribute information to an external entity.

Description

Technical Field

[0001] The present invention relates to a method and device for supporting reading of a fundus image of a subject, and a system for supporting reading of a fundus image comprising the same. Background Art

[0002] With the recent development of artificial intelligence learning models, many machine learning models are used when reading medical images. For example, learning models such as Convolutional Neural Networks, Deep Neural Networks, Recurrent Neural Networks, and Deep Belief Networks are used in detection, classification, and feature learning of medical images.

[0003] In the medical field, the fundus field also uses the above-mentioned learning model to support the reading (finding / diagnosis) of images to predict diseases of the subject.

[0004] As a prior art, Korean Patent No. 10-1858321 (hereinafter referred to as "the prior art patent") describes a method for supporting reading a fundus image of a subject and a computing device using the same. More specifically, a fundus image of the subject is acquired, attribute information is extracted based on a machine learning model and provided to a diagnostician, and in this case, the attribute information includes a variety of information contained in the fundus image.

[0005] However, even if the attribute information of the image is extracted based on the learning model as described above, when there are differences in the learning information input into the learning model, wrong information will be provided to entities such as medical staff. For example, the difference in learning information can be the lack of learning data input into the learning model, the difference in image shooting environment (for example, physical examination center, personal ophthalmology hospital, ophthalmology specialty comprehensive hospital), the difference in group (for example, only normal people, only abnormal people, a mixture of normal people and abnormal people), the difference in image shooting equipment, etc. The above-mentioned various reasons may lead to the wrong prediction of the disease of the person being tested.

[0006] Therefore, there is a need for a method and apparatus that can more accurately predict a disease on a fundus image and provide an explanation that can support the disease prediction even if there are differences in the learning information.

[0007] [Prior art literature]

[0008] [Patent Literature]

[0009] Korean Patent No. 10-1848321 Summary of the invention

[0010] Embodiments of the present invention intend to provide a method for supporting reading of fundus images, an apparatus for supporting reading of fundus images, and a system for supporting reading of fundus images comprising the same, which can utilize a machine learning model for support so as to enable more reliable fundus image reading of fundus images obtained using the machine learning model.

[0011] Embodiments of the present invention intend to provide a method for supporting reading of fundus images, an apparatus for supporting reading of fundus images, and a system for supporting reading of fundus images comprising the same, which can provide position information of the main organs of the eyeball based on the anatomical structure of the eyeball displayed in the fundus image.

[0012] The subject of the present application is not limited to the subject mentioned above, and other subjects not mentioned yet can be clearly understood by those skilled in the art through the following description.

[0013] According to one aspect of the present invention, a method for supporting reading of a fundus image related to an object includes: a step of extracting first attribute information from a first fundus image of the object based on a machine learning model; a step of mapping an adversarial factor to the first fundus image to change the first attribute information and generate a second fundus image with second attribute information; and a step of providing a first fundus image with the first attribute information and a second fundus image with the second attribute information to an external entity.

[0014] According to other aspects of the present invention, a device supporting reading of a fundus image related to an object includes: a memory unit storing a first fundus image of the object; and a processor extracting first attribute information from the first fundus image based on a machine learning model, mapping an adversarial factor to the first fundus image to change the first attribute information, and generating a second fundus image having second attribute information.

[0015] The first attribute information of the first fundus image may be at least one of C / D ratio (Cup-to-Diskratio) information contained in the fundus image, thickness change information related to disc rim thinning, brightness information related to retinal nerve fiber layer defect, and position information related to retinal hemorrhage.

[0016] The countermeasure factor may be at least one of a value for adjusting the gradation level of R, G, B pixels representing the first fundus image, a value for adjusting the color change of R, G, B pixels, and a value for adjusting the local contrast ratio in the first fundus image.

[0017] According to the embodiments of the present invention, it is possible to support reading of interpretable fundus images when reading fundus images based on a machine learning model.

[0018] In addition, even if there are differences in the learning information used for the machine learning model, it is possible to use comparable fundus images for more accurate reading, thereby enabling reliable fundus images to be read.

[0019] The effects of the present application are not limited to the effects mentioned above, and other effects not mentioned yet can be clearly understood by ordinary technicians through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 FIG. 1 is a diagram for explaining a method of supporting reading of a fundus image according to an embodiment of the present invention.

[0021] Figure 2 FIG. 1 is a diagram schematically showing an exemplary structure of an apparatus for supporting reading of a fundus image that executes a method for supporting reading of a fundus image related to a subject according to an embodiment of the present invention.

[0022] Figure 3 FIG. 1 is a diagram schematically showing a system for supporting reading of a fundus image that executes a method for supporting reading of a fundus image related to a subject according to an embodiment of the present invention.

[0023] Figure 4 This is a diagram for explaining a method of supporting comparative reading of C / D ratio (Cup-to-Disk ratio) information as attribute information included in a fundus image according to an embodiment of the apparatus for supporting reading of a fundus image of the present invention.

[0024] Figure 5 The present invention is a diagram showing a visualized image in which adversarial noise is mapped to a fundus image having information of a C / D ratio (Cup-to-Disk ratio) as attribute information in an embodiment of the apparatus for supporting reading of a fundus image according to the present invention.

[0025] Figure 6This is a diagram for explaining a method for supporting the comparative reading of information related to disc rim thinning as attribute information included in a fundus image in one embodiment of the method for supporting the reading of a fundus image according to the present invention.

[0026] Figure 7 This is a diagram for explaining a method for supporting comparative reading of light and dark information related to retinal nerve fiber layer defect (RNLD) as attribute information included in a fundus image in one embodiment of the method for supporting reading of a fundus image according to the present invention.

[0027] Figure 8 This is a diagram for explaining a method of supporting the reading of a fundus image in one embodiment of the present invention so as to enable comparative reading of position information related to retinal hemorrhage as attribute information included in a fundus image.

[0028] Fig. 9A as well as Fig. 9B is a diagram for illustrating a method for generating a fundus image for a machine learning model in one embodiment of an apparatus for supporting reading a fundus image according to the present invention.

[0029] Fig.10 is a flowchart exemplarily illustrating a method for supporting reading of fundus images according to an embodiment of the present invention.

[0030] Reference numerals:

[0031] 10: first fundus image, 20: second fundus image, 30: first attribute information, 40: second attribute information, 110: computing device, 111: processor, 113: memory unit, 115: storage device, 117: input / output interface, 118: network adapter, 119: display adapter, 130: display device. DETAILED DESCRIPTION

[0032] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, anyone with ordinary knowledge in the technical field should easily understand that the accompanying drawings are only for the purpose of more easily disclosing the contents of the present invention, and the scope of the present invention is not limited by the scope of the accompanying drawings.

[0033] In addition, the terms used in the detailed description and claims of the present invention are used only to describe specific embodiments and are not intended to limit the present invention. Unless otherwise specified in the context, a singular expression includes a plural expression.

[0034] In the detailed description and claims of the present invention, it should be understood that terms such as "including" or "having" are used to specify the existence of features, numbers, steps, actions, constituent elements, parts or combinations thereof recorded in the specification, and do not preclude the existence or additional possibilities of one or more other features, or numbers, steps, actions, constituent elements, parts or combinations thereof.

[0035] In the detailed description and claims of the present invention, “learning” refers to a term for performing machine learning by computing in steps, and therefore it should be understood that it is not intended to refer to a mental effect such as an educational activity.

[0036] Further, the present invention includes all possible combinations of the embodiments shown in this specification. It should be understood that the various embodiments of the present invention are different from each other, but there is no need for exclusivity between each other. For example, in one embodiment, the specific shapes, structures and characteristics recorded herein can be embodied by other embodiments without departing from the concept and scope of the present invention. In addition, it should be understood that the position or configuration of individual constituent elements within each disclosed embodiment can be changed without departing from the concept and scope of the present invention. Therefore, the detailed description described below is not adopted as a limiting meaning. As long as it is properly explained, the scope of the present invention is based on all the scopes of equality claimed by the claim and is limited only by the attached claims. In the drawings, similar figure symbols refer to the same or similar functions in many aspects.

[0037] Figure 1 FIG. 1 is a diagram for explaining a method of supporting reading of a fundus image according to an embodiment of the present invention.

[0038] Usually, when a fundus image is acquired from a subject, attribute information of a disease such as glaucoma is derived from the fundus image using a machine learning model. However, as described above, reliable attribute information of the fundus image cannot be derived due to differences in learning information input into the machine learning model. In addition, even when the attribute information of the fundus image is derived for the first time, there is a lack of explanations that can support it in the process of predicting and judging the disease, and therefore, it is not possible to reliably support the reading of the fundus image.

[0039] According to an embodiment of the present invention, in order to solve the problem of lack of reliability in fundus image reading support, the derived fundus image attribute information has reliability, and can automatically generate fundus image attribute information that can support medical staff to accurately read. That is, the device 100 for supporting fundus image reading according to the present invention can firstly help understand the attribute information from the first fundus image 10 obtained by photographing the fundus of the object through a camera or the like, and secondly can help read or explain whether the fundus image of the object including the attribute information is actually a reliable fundus image.

[0040] In order to generate the attribute information of the fundus image as described above, the apparatus 100 supporting reading fundus images may extract the first attribute information 30 of the first fundus image 10 from the captured first fundus image 10 using a machine learning model, and the extracted first attribute information 30 of the first fundus image 10 is stored in the system memory unit 113 or the storage device 115 described later. The machine learning model may be input to the processor 111 for execution, and although not shown in the figure, may be input to the computer.

[0041] In addition, the machine learning model can be input into the memory unit 113 or the storage device 115 and executed by the processor 111.

[0042] Furthermore, the device 100 supporting reading fundus images can further store clinical information of an object, such as an object (patient), in a memory unit 113 or a storage device 115 described later. The processor 111 can utilize the clinical information of the object stored in the memory or storage device to extract the first attribute information of the first fundus image based on the machine learning model. The clinical information may be the age, gender, medical history, outpatient information, test values, exercise habits, eating habits, family medical history related to the medical history, the patient's alcohol consumption, and whether or not he smokes, etc. of the object (patient), and is not limited thereto. Outpatient information may include neurological clinics that doctors can perform on patients, which, unlike medical records, may refer to the ideal opinion currently observed. In addition, the test values ​​may consider, for example, values ​​of intraocular pressure, blood pressure, blood sugar values, etc.

[0043] The first attribute information 30 may include information that can support entities such as medical personnel in predicting and diagnosing disease names. For example, when predicting and diagnosing glaucoma in a fundus image of a subject, the first attribute information may include information on the enhanced C / D ratio (Cup-to-Disk ratio) contained in the fundus image, thickness change information related to disc rim thinning, brightness information related to retinal nerve fiber layer defect, and location information related to retinal hemorrhage. In addition, when predicting and diagnosing diabetic retinopathy in the fundus image of the object, the first attribute information may include location information related to drusen, information related to retinal pigment epithelium change, information related to chorioretinal scar / atrophy, information related to periapillary atrophy, information related to hard exudate, information related to cotton-wool patch, information related to hemorrhage, information related to vascular abnormality, information related to laser scar, information related to subretinal fluid, information related to myelinated nerve fiber, information related to epiretinal membrane, etc. In addition, the first attribute information may include: optic nerve and blood vessel information contained in the fundus image that can understand the information of the main organs of the eyeball in the fundus image, two-eye distinction information showing whether the fundus image is a left eye image or a right eye image, position information showing at least one position of the macula and the optic nerve head contained in the fundus image, regional distinction information of the area showing the fundus image, etc.

[0044] At this time, the first attribute information may be represented by whether there is a finding indicating whether there is a disease or not (Finding) or a prediction value (Prediction Value) that can confirm whether there is a finding. The prediction value is a percentage and may be represented by a number between 0 and 1, but is not limited thereto.

[0045] On the one hand, GAN (Generative Adversarial Networks) as a machine learning model is also used to generate similar images (data) close to actual images (data). However, as mentioned above, since the learning model generates similar images (data) close to actual images using images (images) similar to the original images instead of the original images, it cannot actually accurately reflect the attribute information of the original images. As a result, the reliability of the generated images (data) is reduced.

[0046] According to an embodiment of the present invention, the method for supporting reading fundus images is different from the learning model of GAN. Instead of using similar images, the original image, that is, the captured first fundus image 10, is mapped with an adversarial factor 11 to generate a second fundus image 20 including second attribute information 40. The second attribute information 40 is obtained by changing the first attribute information of the first fundus image. At this time, mapping the adversarial factor 11 can be performed by the processor 111. The second fundus image including the second attribute information 40 can be stored in the memory unit 113 or the storage device 115 described later together with the first fundus image 10.

[0047] The adversarial factor 11 may be an adversarial noise (AdversarialNoise, N0…Nn) applied to the first fundus image. For example, the adversarial noise may include a value for adjusting the gradient level of the R, G, and B pixels representing the first fundus image, a value for adjusting the color (Colar) change of the R, G, and B pixels, a value for adjusting the local contrast (Contrast Ratio) in the first fundus image, etc., and is not limited thereto. As long as it is a factor that can cause a change in the first attribute information of the first fundus image, all of them may be included here.

[0048] The second fundus image 20 including the second attribute information may be generated by mapping the countermeasure factor once, or may be generated by repeatedly mapping so that the prediction value (Prediction Value) displayed for the second attribute information 40 obtained based on the machine learning model converges to the set value (Set Value).

[0049] The mapping method is as shown in the figure. If the set value is 1, the predicted value of the first fundus image obtained based on the machine learning model is 0.58. After mapping the adversarial factor to the first fundus image, it is repeatedly mapped so that the predicted values ​​of the second fundus image obtained based on the same machine learning model, 0.60, 0.66, 0.68,... converge to the set value 1.

[0050] Although not shown in the figure, if the set value is 0 and the predicted value of the first fundus image obtained based on the machine learning model is 0.58, the adversarial factor is repeatedly mapped to the first fundus image, thereby repeatedly mapping so that the predicted values ​​of the second fundus image, 0.56, 0.54, 0.52, ... converge to the set value of 0.

[0051] At this time, the setting value of 1 means that the fundus image is almost abnormal, and may be a fundus image that has a view that a disease (for example, glaucoma, etc.) can be predicted and diagnosed from the fundus image, and the setting value of 0 means that the fundus image is almost normal, and may be a fundus image that has no view that a disease (for example, glaucoma, etc.) can be predicted and diagnosed from the fundus image. Therefore, the value of the second fundus image can be determined according to the value of the setting value.

[0052] As described above, the first fundus image having first attribute information obtained based on a machine learning model and the second fundus image having second attribute information obtained by mapping an adversarial factor can be provided to an external entity or to a remote computing device or other apparatus linked to the computing device 110 via an Internet network through a transmission module such as the network adapter 118 or the display adapter 119 described later.

[0053] Since the second fundus image with the provided second attribute information is close to a normal fundus image or close to an abnormal fundus image, when the entity reads the first fundus image, it is easy to compare and read it with the second fundus image, and the reading result can be explained with a comparison standard, thereby improving the reliability of reading the first fundus image with the first attribute information.

[0054] In addition, the difference between the first attribute information of the first fundus image and the second attribute information of the second fundus image can be displayed and provided to the entity. A specific comparative reading method will be described later.

[0055] Figure 2 FIG. 1 is a diagram schematically showing an exemplary structure of an apparatus for supporting reading of a fundus image that executes a method for supporting comparison of fundus images related to a subject according to an embodiment of the present invention.

[0056] refer to Figure 2According to an embodiment of the present invention, a device 100 supporting reading of fundus images may include a computing device 110 and a display device 130. The computing device 110 may include a processor 111, a memory unit 113, a storage device 115, an input / output interface 117, a network adapter 118, a display adapter 119, and a system bus 112 that connects various system components including the processor to the memory unit 113, but is not limited thereto. In addition, the device supporting reading of fundus images may include not only a bus 112 for transmitting information, but also other communication mechanisms.

[0057] A bus or other communication mechanism enables the processor, a memory as a computer-readable recording medium, a short-range communication module (for example, Bluetooth or NFC (near field communication)), a network adapter including a network interface or a mobile communication module, a display device (for example, a CRT (cathode ray tube) or an LCD (liquid crystal display)), an input device (for example, a keyboard, a keypad, a virtual keyboard, a mouse, a trackball, a stylus, a touch sensing unit, etc.) and / or a lower system to be connected to each other.

[0058] The processor 111 may be a processing module that automatically processes using the machine learning model 13 , and may be a CPU (central processing unit), an AP (application processor), a microcontroller, etc., but is not limited thereto.

[0059] The processor 111 may communicate with a hardware controller for the display device, such as a display adapter 119 , so as to display the actions of the apparatus supporting reading fundus images and a user interface on the display device 130 .

[0060] The processor 111 communicates with the memory unit 113 to execute one or more sequences of instructions or logic stored in the memory unit, thereby controlling the operation of the apparatus for supporting fundus image reading according to an embodiment of the present invention described later.

[0061] The instructions may also be read in a memory by a static storage unit or other computer-readable recording medium such as a disk drive. In other embodiments, a circuit unit (hard-wired circuitry) stored in hardware instead of or in combination with software instructions for embodying the present disclosure may also be used. Logic may also refer to any medium that participates in providing instructions through a processor, and may also be loaded into the memory unit 113.

[0062] System bus 112 represents one or more of a plurality of possible types of bus architectures, including a storage bus, a peripheral device bus, an accelerated graphics port, and a processor or local bus. For example, the architectures may include an ISA (Industry Standard Architecture) bus, an MCA (Micro Channel Architecture) bus, an EISA (Enhanced ISA) bus, a VESA (Video Electronics Standard Association) local bus, an AGP (Accelerated Graphics Port) bus, and PCI (Peripheral Component Interconnects), PCI-Express bus, PCMCIA (Personal Computer Memory Card Industry Association), USB (Universal Serial Bus), and the like.

[0063] The system bus 112 can be implemented by connecting to a wireless network. The subsystems including the processor 111, the mass storage device, the operating system, the imaging software, the imaging data, the network adapter, the system memory, the input / output interface, the display adapter, and the display device can be included in one or more remote computing devices 200, 300, 400 at physically separate locations, and can be connected through the bus of the above-mentioned form when the distributed system is effectively implemented.

[0064] Transmission media including the wires of a bus may include coaxial cables, copper wires, and optical fibers. In one example, transmission media may also take the form of acoustic or light waves generated during radio wave communications or infrared data communications.

[0065] The apparatus 100 for supporting fundus image reading according to the embodiment of the present invention can also send and receive instructions including text messages, data, information, and one or more programs (ie, application codes) through a network link and a network adapter (Network Adapter) 118 .

[0066] The network adapter 118 may also include a separate or combined antenna to be able to send and receive signals through a network link. The network adapter 118 can be connected to a network to communicate with a remote computing device 200, 300, 400 such as a device that remotely supports reading fundus images. The network may include at least one of a LAN (local area network), a WLAN (wireless local area network), a PSTN (public switched telephone network), and a cellular phone network, but is not limited thereto.

[0067] The network adapter 118 may include at least one of a network interface and a mobile communication module to connect to the network. The mobile communication module may connect to mobile communication networks of various eras (eg, 2G to 5G mobile communication networks).

[0068] The program code may also be executed by the processor 111 when received and / or may be stored in a disk drive of the memory unit 113 or a non-volatile memory other than a disk drive for execution.

[0069] The computing device 110 may include a variety of computer-readable recording media. The readable medium may be any of a variety of media accessible by the computing device, for example, it may include volatile or non-volatile media, removable media, fixed media, but not limited thereto. The memory unit 113 may store the operating system, driver, application, data, and database required for the operation of the device supporting reading fundus images according to an embodiment of the present invention, but not limited thereto. In addition, the memory unit 113 may include computer-readable media in the form of volatile memory such as RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory such as flash memory. In addition, it may include a disk drive, such as a hard disk drive, a solid state drive, an optical disk drive, etc., but not limited thereto. In addition, the memory unit 113 and the storage device 115 may each include data such as imaging data 113a, 115a of a typical fundus image of a subject, imaging software 113b, 115b that can be immediately accessed to be executed by the processor 111, and program modules such as operating systems 113c, 115c.

[0070] The machine learning model 13 may be inserted into the processor, the memory unit 113 or the storage device 115. The machine learning model at this time may include a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc., which are one of the machine learning algorithms, but is not limited thereto.

[0071] A deep neural network (DNN) is an artificial neural network (ANN) composed of multiple hidden layers between the input layer and the output layer. Like general artificial neural networks, deep neural networks (DNN) can model complex non-linear relationships. For example, in a deep neural network structure used for object recognition models, each object can be represented by a layer structure of basic factors of an image. At this time, additional layers can aggregate the features of the gradually accumulated lower layers. The above features of deep neural networks can model complex data with only a smaller number of units (nodes) than similarly executed artificial neural networks.

[0072] Convolutional Neural Network (CNN) is a type of multilayer perceptron designed to use minimal preprocessing. Convolutional Neural Network (CNN) consists of one or more convolutional layers and a general artificial neural network layer on top of it, further utilizing weights and pooling layers. Due to the structure, Convolutional Neural Network (CNN) can fully utilize the two-dimensional structure of input data. Compared with other deep learning structures, Convolutional Neural Network (CNN) has excellent performance in both image and speech fields. Convolutional Neural Network (CNN) can also be trained by standard back propagation. Compared with other feedforward artificial neural network technologies, Convolutional Neural Network (CNN) has the advantages of being easy to train and using a small number of parameters.

[0073] In deep learning, a convolutional deep belief network (CDBN) has been developed, which is very similar in structure to the existing convolutional neural network (CNN), so it can make good use of the two-dimensional structure. Similarly, it can also take advantage of the prior training in the deep belief network (CDBN). The convolutional deep belief network (CDBN) provides a general structure that can be used in a variety of image and signal processing techniques, and is used in multiple benchmark results related to standard image data such as CIFAR.

[0074] A recurrent neural network (RNN) is a neural network in which the connections between the units that make up the artificial neural network form a directed cycle. In order to process any input, the recurrent neural network can make use of the memory inside the neural network. Due to this characteristic, recurrent neural networks are used in fields such as handwriting recognition and show high recognition rates.

[0075] Figure 3 FIG. 1 is a diagram schematically showing a system for supporting reading of a fundus image that executes a method for supporting reading of a fundus image related to a subject according to an embodiment of the present invention.

[0076] refer to Figure 3 , a system 500 supporting fundus image reading according to an embodiment of the present invention includes the constituent elements of the device 100 supporting fundus image reading according to an embodiment of the present invention, and further includes a camera unit (Camera) 150. Therefore, the actions and descriptions related to the constituent elements of the device supporting fundus image reading that have been described are omitted. The camera unit 15 includes an image sensor (not shown) and captures a fundus image of the object. The image sensor captures the image of the object and converts the image into an image signal using photoelectrons. The captured fundus image of the object is provided to the processor 111 through the input / output interface 117 and processed based on the machine learning model.

[0077] Figure 4 This is a diagram for explaining a method of supporting comparative reading of C / D ratio (Cup-to-Disk ratio) information as attribute information included in a fundus image according to an embodiment of the apparatus for supporting reading of a fundus image of the present invention.

[0078] The C / D ratio is the value obtained by dividing the maximum distance in the up-down direction (superior and inferior in the direction defined by the fundus image) of the area of ​​the optic cup (Cup) of the fundus image by the maximum distance in the up-down direction of the area of ​​the optic disc (Disc). The C / D ratio information is a criterion for predicting glaucoma in fundus images.

[0079] refer to Figure 4, (a) is a first fundus image obtained from an object based on a machine learning model by a device supporting reading fundus images according to the present invention, and (a') is an image of an optic disc area magnified in the first fundus image of (a). (b) is a second fundus image nearly normal obtained by mapping a negative antagonistic factor to the first fundus image, and (b') is an image of an optic disc area magnified in the second fundus image of (b). (c) is a second fundus image nearly abnormal obtained by mapping a positive antagonistic factor to the first fundus image, and (c') is an image of an optic disc area magnified in the second fundus image of (c). The first attribute information of the first fundus image is information on the C / D ratio, and the second attribute information of the second fundus image is information related to information on the C / D ratio that changes by mapping the antagonistic factor. As described above, the first attribute information and the second attribute information are respectively reflected in the first fundus image and the second fundus image and shown. The mapping method of the antagonistic factor has been described, so the description thereof is omitted.

[0080] Specifically, in the case of the first fundus image (a) obtained based on the machine learning model, if it is based on the information of the C / D ratio as the first attribute information, it is a fundus image on which an entity such as a medical staff gives an opinion that the fundus image has a disease such as glaucoma. However, even if there is an opinion on the first fundus image based on the machine learning model as described above, when there is no fundus image that can actually be compared, there is a lack of explanation on the reliability of the first fundus image or the confirmation of the opinion.

[0081] According to the device supporting reading fundus images of the present invention, a second fundus image can be generated to make the C / D ratio close to that of a normal fundus image (as shown in (b)), or a second fundus image can be generated to make the C / D ratio close to that of an abnormal fundus image (as shown in (c)), and provided to an entity, so that when reading the first fundus image, it can be compared with the second fundus image, and the reliability of the reading of the first fundus image and the basis for explaining the reading can be obtained.

[0082] Figure 5 The present invention is a diagram showing a visualized image in which adversarial noise is mapped to a fundus image having information of a C / D ratio (Cup-to-Disk ratio) as attribute information in an embodiment of the apparatus for supporting reading of a fundus image according to the present invention.

[0083] refer to Figure 5 , (a) and (b) are each Figure 4In the images (b) and (c) that visualize the countermeasure noise (red or blue dots) mapped on the second fundus image, as indicated by the arrows, it can be confirmed that the countermeasure noise is mapped centered on the optic disc area where the C / D ratio information can be confirmed in the fundus image.

[0084] Specifically, when the C / D ratio is close to normal (a), it can be confirmed that the countermeasure noise is more concentratedly mapped in the optic disc area, and when the C / D ratio is close to abnormal (b), it can be confirmed that the countermeasure noise is not only in the optic disc area, but also widely mapped in the entire fundus image.

[0085] As described above, the apparatus for supporting reading of fundus images according to the present invention can confirm at which position of the fundus image attribute information such as C / D ratio information changes by visualizing the anti-noise, thereby enabling reading of more reliable fundus images.

[0086] Figure 6 This is a diagram for explaining a method for supporting the comparative reading of information related to disc rim thinning as attribute information included in a fundus image in one embodiment of the method for supporting the reading of a fundus image according to the present invention.

[0087] Disc rim thinning refers to the degree to which the thickness of the rim defined in the superior rim (Stemp), superior, inferior rim (Inferiotempal), and inferior rim directions defined in the fundus image becomes narrower, excluding the area of ​​the optic cup (Cup) in the optic disc (Disc) area of ​​the fundus image. Generally, when any of the values ​​of the rim thickness in the superior rim, superior, inferior rim, and inferior rim directions divided by the thickness of the rim in the temporal (Temp) direction is approximately 1.6 or less, or is approximately 0.1 times or less of the maximum length of the optic disc (Disc), it is considered that glaucoma is present.

[0088] refer to Figure 6, (a) is a first fundus image obtained from an object based on a machine learning model by a device supporting fundus image reading according to the present invention, and (a') is an image of an optic disc area magnified in the first fundus image of (a). (b') is an image of an optic disc area magnified in a second fundus image almost close to normal obtained by mapping a negative antagonistic factor to the first fundus image. (c') is an image of an optic disc area magnified in a second fundus image almost close to abnormal obtained by mapping a positive antagonistic factor to the first fundus image. The first attribute information of the first fundus image is thickness change information related to disc rim thinning, and the second attribute information of the second fundus image is information related to information related to disc rim thinning that changes due to mapping of the antagonistic factor. As described above, the first attribute information and the second attribute information are respectively reflected in the first fundus image and the second fundus image and shown.

[0089] Specifically, the case (a') obtained based on the machine learning model is that, if it is based on the thickness change information related to disc rim thinning as the first attribute information, the ratio of the thickness R of the rim (Rim) in the upward outer edge direction (Stemp Direction) to the thickness D in the outer edge direction (Temp Direction) is equivalent to about 0.56, which is a fundus image that indicates the presence of glaucoma. In this regard, the device supporting reading of fundus images according to the present invention can generate a second fundus image so that the ratio of the thickness R2 of the rim (Rim) in the upward outer edge direction (Stemp Direction) to the thickness D in the outer edge direction (Temp Direction) is close to a normal fundus image (as shown in (b')), or generate a second fundus image so that the ratio of the thickness R1 of the rim (Rim) in the upward outer edge direction (Stemp Direction) to the thickness D in the outer edge direction (Temp Direction) is close to an abnormal fundus image (as shown in (c')), and provide it to the entity. In addition, although not shown in the figure, the device supporting reading of fundus images according to the present invention can generate a fundus image having information such as a C / D ratio, and map advancer noise to the fundus image having thickness change information related to disc rim thinning as attribute information, thereby visualizing it.

[0090] Figure 7This is a diagram for explaining a method for supporting comparative reading of light and dark information related to retinal nerve fiber layer defect (RNLD) as attribute information included in a fundus image in one embodiment of the method for supporting reading of a fundus image according to the present invention.

[0091] Retinal nerve fiber layer defect (RNFL Defect) refers to the situation where the retinal nerve fiber layer is displayed in a fan-shaped form starting from the optic disc (Disc) in the fundus image, and the nerve fibers are defective. When the retinal nerve fiber layer defect occurs, the corresponding part in the fundus image becomes dark, and the pattern of bright nerve fiber types is lost, so it is considered to be the presence of glaucoma.

[0092] refer to Figure 7 , (a) is a first fundus image obtained from an object based on a machine learning model by a device supporting fundus image reading according to the present invention, and (a') is an image of an area of ​​the retinal nerve fiber layer enlarged in the first fundus image of (a). (b') is an image of the retinal nerve fiber layer enlarged in a second fundus image almost close to normal obtained by mapping a negative antagonist factor to the first fundus image. (c') is an image of the retinal nerve fiber layer enlarged in a second fundus image almost close to abnormal obtained by mapping a positive antagonist factor to the first fundus image. The first attribute information of the first fundus image is light and dark information related to retinal nerve fiber layer defect, and the second attribute information of the second fundus image is information related to light and dark information related to retinal nerve fiber layer defect that changes due to the mapping of the antagonist factor. As described above, the first attribute information and the second attribute information are respectively reflected in the first fundus image and the second fundus image and shown.

[0093] Specifically, in the case of (a') obtained based on the machine learning model, if it is based on the brightness information related to the retinal nerve fiber layer defect (RNFLD) as the first attribute information, there is a wedge-shaped focal RNFL defect (Wedge shaped Focal RNFLDefect) in the green dotted line area. In other words, (a') is a fundus image that does not show the fiber layer equivalent to the retinal nerve fiber, and based on this, it is considered that there may be glaucoma.

[0094] In this regard, the device for supporting reading of fundus images according to the present invention can reduce the area where retinal nerve fibers are not displayed (orange dotted line area) and display the retinal nerve fibers indicated by arrows in the green dotted line area, thereby generating a second fundus image close to a normal fundus image (as shown in (b')), or increase the area where retinal nerve fibers are not displayed (red dotted line area), and make the corresponding area darker in order to make the retinal nerve fibers in the original green dotted line area not displayed, thereby generating a second fundus image close to an abnormal fundus image (as shown in (c')), and providing it to the entity. In addition, although not shown in the figure, the device for supporting reading of fundus images according to the present invention can generate a fundus image such as one having information of a C / D ratio, and map an adversarial noise to a fundus image having light and dark information related to a retinal nerve fiber layer defect as attribute information, thereby visualizing it.

[0095] Figure 8 This is a diagram for explaining a method of supporting the reading of a fundus image in one embodiment of the present invention so as to enable comparative reading of position information related to retinal hemorrhage as attribute information included in a fundus image.

[0096] Retinal hemorrhage is caused by the rupture of blood vessels distributed on the retina, resulting in darkening or red spots in specific areas of the fundus image.

[0097] refer to Figure 8 , (a) is a first fundus image obtained from an object based on a machine learning model by a device supporting fundus image reading according to the present invention, and (a') is an image of an area in the first fundus image that is magnified to show retinal hemorrhage. (b') is an image that is magnified to not show retinal hemorrhage in a second fundus image that is almost normal and obtained by mapping a negative antagonist factor to the first fundus image. (c') is an image that is magnified to show retinal hemorrhage in a second fundus image that is almost abnormal and obtained by mapping a positive antagonist factor to the first fundus image. The first attribute information of the first fundus image is position information related to retinal hemorrhage, and the second attribute information of the second fundus image is information related to position information related to retinal hemorrhage that has changed due to the mapping of the antagonist factor. As described above, the first attribute information and the second attribute information are respectively reflected in the first fundus image and the second fundus image and shown.

[0098] Specifically, in the case of (a') obtained based on the machine learning model, if it is based on the position information related to retinal hemorrhage as the first attribute information, there is a bleeding point in the part indicated by the arrow.

[0099] In this regard, the device for supporting reading of fundus images according to the present invention can generate a second fundus image close to a normal fundus image so that no bleeding spots appear in the same area of ​​(a') (the enlarged image is shown in (b')), or generate a second fundus image close to an abnormal fundus image so that the color of the bleeding spots is more clear in the same area of ​​(a') or bleeding spots appear further in other areas (the enlarged image is shown in (c')), and provide it to the entity. In addition, although not shown in the figure, the device for supporting reading of fundus images according to the present invention can generate a fundus image having information such as a C / D ratio, and map an adversarial noise to a fundus image having position information related to retinal hemorrhage as attribute information, thereby visualizing it.

[0100] In addition, the device supporting reading of fundus images according to the present invention can compare and read position information related to drusen, information related to retinal pigment epithelium change, information related to chorioretinal scar / atrophy, information related to periapillary atrophy, information related to hard exudate, information related to cotton-wool patch, information related to hemorrhage, information related to vascular abnormality, information related to laser scar, information related to subretinal fluid, information related to myelinated nerve fiber, information related to epiretinal membrane, etc. The comparative reading method can utilize the same method as the embodiment described above, so the description thereof is omitted.

[0101] Fig. 9A as well as Fig. 9B is a diagram for illustrating a method for generating a fundus image for a machine learning model in one embodiment of an apparatus for supporting reading a fundus image according to the present invention.

[0102] refer to Fig. 9A For a first fundus image (a) of a subject captured by a camera or stored, in order to focus on confirming a specific area of ​​the first fundus image, such as information on the optic nerve head area including retinal blood vessels, image preprocessing is performed to generate a fundus image (b) to enlarge the corresponding area. The fundus image predicts whether there is a fundus disease based on attribute information obtained through a machine learning model.

[0103] In addition, if Fig. 9B As shown, for the image (a) enlarged by image preprocessing, as described in (b), it can be locally divided to generate multiple fundus images (c). At this time, the multiple fundus images can be generated by locally overlapping the fundus parts. The multiple fundus images are also based on the attribute information obtained by the machine learning model to predict whether there is a fundus disease.

[0104] As described above, a new fundus image is generated from the initial first fundus image of the subject through a variety of methods and input into a machine learning model, thereby improving the prediction accuracy of fundus diseases.

[0105] On the one hand, Fig. 9A as well as Fig. 9B In the method, a new fundus image for the machine learning model is generated by enlarging a specific area or performing image preprocessing by local division. However, it is not limited to this, and a variety of new fundus images can be generated. For example, the fundus image can be generated by rotating around a specific reference axis of the first fundus image, or adjusting the contrast (Contrast) or brightness (Brightness) of the first fundus image to generate the fundus image, or flipping (flipping) around the horizontal or vertical axis of the first fundus image to generate the fundus image. In addition, the above methods can be combined in multiple ways to generate new fundus images.

[0106] Fig.10 is a flowchart exemplarily illustrating a method for supporting reading of fundus images according to an embodiment of the present invention.

[0107] refer to Fig.10, a first fundus image of the object is obtained from an external device, such as a fundus camera or other device (not shown) linked to the computing device 110, and the computing device 110 extracts first attribute information from the first fundus image based on the machine learning model S910. The first fundus image can be stored in the memory unit 113 or the storage device 115. Then, the computing device 110 maps the adversarial noise to the first fundus image. The mapping method of the adversarial noise is the same as the method described above, so its description is omitted. The first fundus image mapped with the adversarial noise is based on the result learned by the same machine learning model, and the attribute information changes, thereby generating a second fundus image with second attribute information with changed attribute information S920. In this case, an image visualized by mapping the adversarial noise to the first fundus image can be further generated. The second fundus image can also be stored in the memory unit 113 or the storage device 115. The second fundus image with the first attribute information and the second attribute information is provided to the external entity through a transmission module such as the network adapter 118 and the display adapter 119 of the computing device S930.

[0108] As described above, the device and method for supporting reading fundus images according to the present invention can support reading of fundus images that can be explained when reading fundus images based on machine learning models, thereby enabling medical staff to read more reliable fundus images.

[0109] As described in the above embodiments, it should be clearly understood that the present invention can be achieved by a combination of software and hardware or by hardware alone. The object of the technical method of the present invention or the part that contributes to the prior art can be embodied in the form of program instructions that can be executed by a variety of computer components, thereby recorded in a machine-readable recording medium. The machine-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded in the machine-readable recording medium may also be specially designed and constructed for the present invention, or may be known and usable to a general technician in the field of computer software.

[0110] Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs (Compact Disc Read Only Drives), DVDs, magneto-optical media such as floppy disks, and specially constructed hardware devices such as ROMs, RAMs, and flash memories to store and execute program instructions. Examples of program instructions include not only machine language codes such as those obtained by compilation, but also high-level language codes that can be executed by a computer using an interpreter.

[0111] In order to perform the processing of the present invention, the hardware device can be configured to act as one or more software modules, and its functions are the same. The hardware device may include a processor such as a CPU or a GPU, and may include a communication unit that can send and receive signals with an external device, wherein the processor such as the CPU or the GPU is configured to be combined with a memory such as a ROM / RAM for storing program instructions to execute the instructions stored in the memory. Furthermore, the hardware device may include a keyboard, a mouse, and other external input devices to transmit and obtain instructions written by the developer.

[0112] As described above, after observing the embodiments according to the present invention, it should be clear to those with common knowledge in the technical field that in addition to the embodiments described above, the present invention can also be embodied in other specific forms without departing from the subject matter or scope. Therefore, it should be understood that the above embodiments are not intended to be limiting but to be used as examples, and thus, the present invention is not limited to the above description and can also be changed within the scope of the attached claims and their equivalents.

Claims

1. A method for supporting reading of a fundus image, which is a method for supporting reading of a fundus image related to an object, the method include: (a) extracting first attribute information from a first fundus image of the subject based on a machine learning model; (b) mapping an antagonistic factor to the first fundus image to change the first attribute information and generate a second fundus image having second attribute information, wherein the second fundus image is generated by repeatedly mapping the antagonistic factor to the first fundus image at least once, so that a predicted value of the second attribute information obtained based on the machine learning model converges to a set value; and (c) A step of providing a first fundus image having the first attribute information and a second fundus image having the second attribute information to an external entity.

2. The method for supporting reading fundus images according to claim 1, in, The countermeasure factor includes at least one of: adjusting a value representing a gradient level of R, G, and B pixels of the first fundus image, a value capable of adjusting a color change of R, G, and B pixels, and a value adjusting a local contrast in the first fundus image.

3. The method for supporting reading fundus images according to claim 1, in, The value of the second fundus image is determined according to the value of the set value.

4. The method for supporting reading fundus images according to claim 1, in, The machine learning models include deep neural networks, convolutional neural networks, and recurrent neural networks.

5. The method for supporting reading fundus images according to claim 1, in, The first attribute information of the first fundus image is: At least one of the cup-to-disc ratio information, thickness variation information related to disc rim thinning, brightness information related to retinal nerve fiber layer defects, and position information related to retinal hemorrhage contained in the fundus image.

6. The method for supporting reading fundus images according to claim 1, in, The first attribute information of the first fundus image is: At least one of location information related to drusen, information related to retinitis pigmentosa, information related to chorioretinal scar / atrophy, information related to peripapillary atrophy, information related to hard exudates, information related to cotton-wool spots, information related to hemorrhage, information related to abnormal vascular diseases, information related to laser scars, information related to subretinal fluid, information related to myelinated nerve fibers, and information related to epiretinal membrane.

7. The method for supporting reading fundus images according to claim 1, further comprising: include: The step of mapping a countermeasure factor to the first fundus image to generate a visualized image. 8 . The method for supporting fundus image reading according to claim 1 , wherein the first attribute information of the first fundus image is extracted by utilizing clinical information of the subject.

9. The method for supporting reading fundus images according to claim 8, in, The clinical information includes at least one of the subject's age, gender, medical history, outpatient information, test values, exercise habits, eating habits, smoking, and drinking habits. 10 . A computer program stored in a machine-readable recording medium, comprising instructions configured to execute the method of claim 1 .

11. A device for supporting reading fundus images, the device include: a memory unit storing a first fundus image of the subject; as well as A processor, which extracts first attribute information from the first fundus image based on a machine learning model, maps an antagonistic factor to the first fundus image to change the first attribute information, and generates a second fundus image with second attribute information, wherein the second fundus image is generated by repeatedly mapping the antagonistic factor to the first fundus image at least once, so that the predicted value of the second attribute information obtained based on the machine learning model converges to a set value.

12. The device for supporting reading fundus images according to claim 11, in, The first attribute information of the first fundus image is: At least one of the cup-to-disc ratio information, thickness variation information related to disc rim thinning, brightness information related to retinal nerve fiber layer defects, and position information related to retinal hemorrhage contained in the fundus image.

13. The device for supporting reading fundus images according to claim 11, in, The antagonistic factors include: At least one of a value representing a gradation level of R, G, B pixels of the first fundus image, a value capable of adjusting a color change of the R, G, B pixels, and a value adjusting a local contrast in the first fundus image is adjusted.

14. A system for supporting reading fundus images, the system include: a camera unit that captures a first fundus image of a subject; a memory unit storing the first fundus image; as well as A processor, which extracts first attribute information from the first fundus image based on a machine learning model, maps an antagonistic factor to the first fundus image to change the first attribute information, and generates a second fundus image with second attribute information, wherein the second fundus image is generated by repeatedly mapping the antagonistic factor to the first fundus image at least once, so that the predicted value of the second attribute information obtained based on the machine learning model converges to a set value.

Images