IMAGE PROCESSING METHOD, PROGRAM, IMAGE PROCESSING APPARATUS, AND OPHTHALMOLOGY SYSTEM

The image processing method enhances fundus image analysis by extracting and emphasizing abnormal regions, improving diagnostic accuracy and reducing oversight in ophthalmology.

JP7579889B2Active Publication Date: 2024-11-08NIKON CORP +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2022576259
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-01-19
Publication Date
2024-11-08
Estimated Expiration
2041-01-19

AI Technical Summary

Technical Problem

Conventional methods for detailed observation of fundus images in ophthalmology lack efficient processes for identifying and highlighting abnormal regions, making it difficult for medical professionals to accurately diagnose and manage eye conditions.

Method used

An image processing method that includes acquiring a wide-angle fundus image, extracting partial images with abnormal findings, selecting a first partial image showing an abnormal region, and displaying it with emphasis, using a neural network or image processing algorithm to enhance abnormal areas and assist in diagnosis.

Benefits of technology

Facilitates easy recognition of abnormal areas in fundus images, aiding medical professionals in accurate diagnosis and reducing the likelihood of overlooking abnormalities, while improving the detection model's accuracy through relearning with annotated data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579889000001
    Figure 0007579889000001
  • Figure 0007579889000002
    Figure 0007579889000002
  • Figure 0007579889000003
    Figure 0007579889000003
Patent Text Reader

Abstract

This image processing method comprises: a process of acquiring a wide-angle fundus image obtained by imaging with an ophthalmic apparatus; a process of extracting a plurality of partial images from the wide-angle fundus image to which an abnormal finding is added; a selection process of selecting a first partial image showing an abnormal region that provides a reason of the abnormal finding from among the plurality of partial images; and a display process of displaying the first partial image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image processing method, a program, an image processing device, and an ophthalmologic system. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a method for displaying an enlarged image in response to a user's operation in order to observe details of a fundus image is known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2020 / 0069175 Summary of the Invention

[0004] The technique of the present disclosure provides a novel image processing method. [Means for solving the problem]

[0005] One embodiment of the present invention is an image processing method including a process for acquiring a wide-angle fundus image captured by an ophthalmic device, a process for extracting a plurality of partial images from the wide-angle fundus image having an abnormal finding attached thereto, a selection process for selecting a first partial image from the plurality of partial images that shows an abnormal area that is the basis of the abnormal finding, and a display process for displaying the first partial image. [Brief description of the drawings]

[0006] [Figure 1] 1 is an overall configuration diagram of an information processing system according to a first embodiment. [Diagram 2] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. [Diagram 3] 1 is a diagram showing a configuration of an ophthalmologic apparatus according to a first embodiment. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of a server according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing the configuration of a detection unit according to the first embodiment. [Figure 6] 4 is a flowchart showing an outline of processing according to the first embodiment. [Figure 7] 4 is a flowchart showing an estimation process according to the first embodiment. [Figure 8] 5 is a flowchart showing an output process according to the first embodiment. [Figure 9] 6 is a flowchart showing an addition process according to the first embodiment. [Figure 10] 13A and 13B are diagrams showing an example of setting a partial region of a wide-angle fundus image and an example of a GUI using the generated partial image. [Figure 11] 13A and 13B are diagrams showing an example of setting a partial region of a wide-angle fundus image and an example of a GUI using the generated partial image. [Figure 12] FIG. 13 illustrates an example of a GUI. [Figure 13] FIG. 13 illustrates an example of a GUI. [Figure 14] FIG. 13 illustrates an example of a GUI. [Figure 15] FIG. 11 is a diagram illustrating a functional configuration of a server according to the second embodiment. [Figure 16] 10 is a flowchart showing an estimation process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] First Embodiment Hereinafter, the present invention will be described based on a first embodiment, which is one embodiment of the present invention, with reference to the drawings.

[0008] 〔composition〕 1 shows a configuration of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 includes a server 10, a terminal 20, and an ophthalmic device 30. The server 10, the terminal 20, and the ophthalmic device 30 are connected to each other via a network 5 so as to be able to transmit and receive data to and from each other.

[0009] The network 5 is a wireless or wired communication means, such as the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a public communication network, a dedicated line, etc. Although the information processing system 1 according to this embodiment is composed of a plurality of information management devices, the present invention does not limit the number of these devices. Therefore, the information processing system 1 can be composed of one or more devices as long as they have the following functions.

[0010] The server 10 and the terminal 20 are information processing devices installed in medical institutions such as hospitals and clinics, and are mainly operated by medical institution staff to acquire images taken by the ophthalmic device 30, as well as edit and analyze the images.

[0011] The ophthalmic apparatus 30 is an apparatus that performs SLO (Scanning Laser Ophthalmoscope) and OCT (Optical Coherence Tomography) (FIG. 3). The ophthalmic apparatus 30 includes a control device 31 and an imaging device 32.

[0012] 2 shows an example of hardware (hereinafter referred to as "information processing device 100") used to realize the server 10, the terminal 20, and the control device 31 of the ophthalmic device 30. As shown in the figure, the information processing device 100 includes a processor 101, a main memory device 102, an auxiliary memory device 103, an input device 104, an output device 105, and a communication device 106. These are connected to each other so as to be able to communicate with each other via communication means such as a bus (not shown).

[0013] In addition, all of the configurations of the information processing device 100 do not necessarily need to be realized by hardware, and all or part of the configurations may be realized by virtual resources such as a cloud server of a cloud system.

[0014] The processor 101 is configured using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The processor 101 reads and executes a program stored in a main memory device 102, thereby realizing the functions of the server 10, the terminal 20, and the control device 31.

[0015] The main memory device 102 is a device for storing programs and data, and is a read only memory (ROM), a random access memory (RAM), a non-volatile semiconductor memory (Non Volatile RAM (NVRAM)), or the like.

[0016] The auxiliary storage device 103 is, for example, various non-volatile memories (NVRAM: Non-volatile memory) such as SSD (Solid State Drive) and SD memory card, a hard disk drive, an optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a storage area of ​​a cloud server, etc. The programs and data stored in the auxiliary storage device 103 are loaded into the main storage device 102 as needed.

[0017] The input device 104 is an interface that accepts input of information, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device (such as a microphone), a voice recognition device, etc. The information processing device 100 may be configured to accept input of information between it and another device via the communication device 106.

[0018] The output device 105 is an interface that outputs various types of information, such as a screen display device (such as a liquid crystal monitor, LCD (Liquid Crystal Display), a graphic card, a printer, etc.), a voice output device (such as a speaker), a voice synthesizer, etc. The information processing device 100 may be configured to output information to other devices via the communication device 106. The output device 105 corresponds to the display unit in the present invention.

[0019] The communication device 106 is a wired or wireless communication interface that realizes communication with other devices via the network 5, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, etc.

[0020] 3 shows the configuration of the ophthalmic apparatus 30. The ophthalmic apparatus 30 includes an image capturing device 32 and a control device 31. The control device 31 may be provided in the same housing as the image capturing device 32, or may be separate from the image capturing device 32.

[0021] The photographing device 32 operates under the control of the control device 31. The photographing device 32 includes an SLO unit 33, a photographing optical system 34, and an OCT unit 35. The photographing optical system 34 includes an optical scanner 341 and a wide-angle optical system 342. The photographing device 32 photographs an image of the subject's eye. The photographing device 32 photographs, for example, the fundus of the subject's eye, and obtains a fundus image and a tomographic image (OCT image) described below.

[0022] The SLO unit 18 acquires an image of the fundus 12A of the subject's eye 12. The OCT unit 20 acquires a tomographic image of the subject's eye 12. Hereinafter, a front-view image of the retina created based on the SLO data acquired by the SLO unit 18 will be referred to as an SLO image, and a tomographic image or a front image (en-face image) of the retina created based on the OCT data acquired by the OCT unit 20 will be referred to as an OCT image. The SLO image may also be referred to as a two-dimensional fundus image. The OCT image may also be referred to as a fundus tomographic image, a posterior segment tomographic image, or an anterior segment tomographic image depending on the part of the subject's eye 12 that is photographed.

[0023] Hereinafter, the image captured by the ophthalmic apparatus 30 may be referred to as an image of the subject's eye. In addition, when the image of the subject's eye is an image of the fundus captured using a wide-angle optical system as described below, the image may be referred to as a wide-angle fundus image P.

[0024] The optical scanner 341 scans the light emitted from the SLO unit 33 in the X direction and the Y direction. The optical scanner 341 may be an optical element capable of deflecting a light beam, and may be, for example, a polygon mirror or a galvanometer mirror.

[0025] The wide-angle optical system 342 includes an objective optical system. The wide-angle optical system 342 provides a wide-angle field of view at the fundus. The SLO system is realized by the control device 31, the SLO unit 33, and the photographing optical system 34 shown in FIG. 3. The SLO system includes the wide-angle optical system 342, and thus allows observation of the fundus in a wide field of view (FOV). The FOV indicates a range that can be photographed by the photographing device 32. The FOV can be expressed as a field of view. In this embodiment, the field of view can be defined by an internal irradiation angle and an external irradiation angle. The external irradiation angle is an irradiation angle that defines the irradiation angle of the light beam irradiated from the UWF ophthalmic device 110 to the subject's eye based on the pupil. The internal irradiation angle is an irradiation angle that defines the irradiation angle of the light beam irradiated to the fundus 12A based on the eyeball center O. The external irradiation angle and the internal irradiation angle are in a corresponding relationship. For example, when the external irradiation angle is 120 degrees, the internal irradiation angle corresponds to approximately 160 degrees. In this embodiment, the internal illumination angle is set to 200 degrees. Here, an SLO fundus image captured at an internal illumination angle and an imaging angle of view of 160 degrees or more is referred to as a UWF-SLO fundus image.

[0026] The wide-angle optical system 342 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric system combining a concave mirror and a lens. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it is possible to photograph the retina in the peripheral part of the fundus as well as the center part of the fundus. When using a system including an elliptical mirror, a configuration using a system using an elliptical mirror described in International Publication WO2016 / 103484 or International Publication WO2016 / 103489 may be used. The disclosures of International Publication WO2016 / 103484 and International Publication WO2016 / 103489 are each incorporated herein by reference in their entirety.

[0027] The SLO unit 33 includes a blue light (B light) light source 331B, a green light (G light) light source 331G, a red light (R light) light source 331R, and a light source 331IR of infrared light (IR light) such as near-infrared light, and an optical system 335 that reflects or transmits the light from these light sources and guides it to one optical path. The SLO unit 33 also includes a beam splitter 332, and detection elements 333B, 333G, 333R, and 333IR that detect the B light, G light, R light, and IR light, respectively.

[0028] The SLO unit 33 is capable of switching between a light source to be emitted or a combination of light sources to be emitted, such as a mode for emitting B light, R light, and G light, and a mode for emitting IR light.

[0029] The beam splitter 332 has a function of splitting the reflected light from the fundus into B light, R light, G light, and IR light, and reflecting each of the lights toward the detection elements 333B, 333G, 333R, and 333IR.

[0030] The detection elements 333B, 333G, 333R, and 333IR can detect B light, R light, G light, and IR light, respectively.

[0031] The light incident on the photographing optical system 34 from the SLO unit 33 is scanned in the X and Y directions by the optical scanner 341. The scanning light is irradiated onto the fundus via the wide-angle optical system 342. The light reflected by the fundus is incident on the SLO unit 33 via the wide-angle optical system 342 and the optical scanner 341.

[0032] The light reflected by the fundus enters the SLO unit 33 via the wide-angle optical system 342 and the optical scanner 341, and is split into B light, R light, G light, and IR light by the beam splitter 332. These lights are detected by the detection elements 333B, 333G, 333R, and 333IR, respectively.

[0033] By collecting information on the B light, R light, G light, and IR light detected by the detection elements 333B, 333G, 333R, and 333IR, the processor 101 of the control device 31 can generate an SLO fundus image.

[0034] The OCT system includes a control device 31, an OCT unit 35, and an imaging optical system 34 shown in Fig. 3. The OCT unit 35 includes a light source 351, a sensor 352, a first optical coupler 353, a reference optical system 354, a collimator lens 355, and a second optical coupler 356.

[0035] Light emitted from the light source 351 is branched by a first optical coupler 353. One of the branched lights is collimated by a collimating lens 355 as measurement light and enters the photographing optical system 34. The light that enters the photographing optical system 34 is scanned in the X and Y directions by an optical scanner 341. The scanning light passes through a wide-angle optical system 342 and is irradiated onto the fundus. The measurement light reflected by the fundus enters the OCT unit 35 via the wide-angle optical system 342, and enters the second optical coupler 356 via the collimating lens 355 and the first optical coupler 353.

[0036] The other light emitted by the light source 351 and branched by the first optical coupler 353 passes through a reference optical system 354 and enters a second optical coupler 356 as a reference light.

[0037] The reference light and the measurement light reflected by the fundus are interfered with by the second optical coupler 356 to generate interference light. The interference light is received by the sensor 352. The control device 31 receives a signal from the sensor 352 and generates a tomographic image. Note that, hereinafter, the act of photographing using the OCT system and the image obtained by photographing may be simply referred to as OCT photographing and OCT image, respectively.

[0038] [Software configuration] The main functions (functional configuration) of the server 10 are shown in Fig. 4. As shown in the figure, the server 10 has the functions of a database 114 and a management unit 120. The management unit 120 has the functions of an image processing unit 116 and a detection unit 118 in particular. The database 114 is stored in the main memory device 102 of the server 10. Each function of the management unit 120 is realized by the processor 101 of the server 10 reading and executing a program stored in the main memory device 102 of the server 10.

[0039] In addition to the above functions, the server 10 also has functions such as an operating system, a file system, a device driver, and a DBMS (DataBase Management System).

[0040] The management unit 120 performs processes executed by the server 10, such as acquisition and management of images. The images acquired and managed by the management unit include images captured by the ophthalmic device 30. The image processing unit 116 mainly generates a GUI and processes images captured by the ophthalmic device 30.

[0041] The detection unit 118 has a function of estimating the presence or absence of a finding indicating an abnormality such as bleeding or retinal detachment in the fundus including the retina or choroid (hereinafter referred to as an "abnormal finding") and details thereof from an image captured by the ophthalmologic apparatus 30. In the detection unit 118, the presence or absence of an abnormal finding and details thereof are estimated based on an abnormal region in the image of the subject's eye. In this embodiment, the detection unit 118 is a trained model generated by machine learning.

[0042] In detail, the detection unit 118 is a model that performs deep learning to learn image feature amounts of abnormal regions in an image of the subject's eye. The detection unit 118 constructs a neural network that outputs information indicating an estimated result of the presence or absence of abnormal findings for the input image of the subject's eye. For example, the neural network is a deep neural network (DNN).

[0043] The detection unit 118 has an input layer that receives input of an image of the subject's eye, an output layer that outputs an estimation result of the presence or absence of abnormal findings, and an intermediate layer that extracts image features of the image of the subject's eye (FIG. 5). Each of the input layer, output layer, and intermediate layer has nodes (indicated by white circles in the figure), and the nodes of each of these layers are connected by edges (indicated by arrows in the figure). Note that the configuration of the detection unit 118 shown in FIG. 5 is an example, and the numbers of nodes and edges, the number of intermediate layers, etc. can be changed as appropriate.

[0044] When the detection unit 118 is a CNN (Convolutional Neural Network), the intermediate layer has a convolution layer that convolves the pixel value of each pixel in the test eye image input from the input layer, and a pooling layer that maps the pixel value, and these layers are used to extract the feature amount of the test eye image. The output layer has one or more neurons that output the result of estimating abnormal findings in the input test eye image.

[0045] The detection unit 118 can also output the likelihood of the inference result together with the inference result. The likelihood is, for example, a probability value output from the output layer of the detection unit 118, and is indicated, for example, by a value from "0" to "1" indicating the degree of reliability of the inferred abnormal finding. By notifying the user of the likelihood, the user can know how accurate the inference result is.

[0046] The detection unit 118 also outputs the severity of the abnormal findings. The severity can be explained as the severity of the symptoms, the grade of the symptoms, the rate of progression, the impact of the symptoms on the human body, and the like. When the symptoms are bleeding, for example, the detection unit 118 estimates the severity of the bleeding from the size and amount of bleeding. Similarly, the detection unit 118 outputs the severity for other abnormal findings such as retinal detachment and neovascularization.

[0047] In particular, in this embodiment, the detection unit 118 is described as being a CNN, but the detection unit 118 is not limited to a CNN and may be a neural network other than a CNN or a trained model constructed using another learning algorithm.

[0048] The database 114 stores images obtained by photographing fundus tissues including the retina and choroid, such as wide-angle fundus images P and OCT images. Annotations can also be added to the image of the subject's eye by medical professionals, and the database 114 can store the image of the subject's eye and the portion of the image to which annotations have been added and the contents of the annotations in association with an image showing a part of the image of the subject's eye. The stored data is used for learning and re-learning of the detection unit 118. The annotations include information showing an abnormal area of ​​the fundus and details of abnormal findings such as macular degeneration added to the abnormal area. Therefore, when used for learning of the detection unit 118, the annotations function as data showing the correct answer to the abnormal findings and abnormal areas of the input image. In addition, the database 114 stores patient medical records such as electronic medical records including past records of the image of the subject's eye, patient IDs, and other data. The database 118 also stores information on findings obtained by the detection unit 118, which will be described later, in association with the image.

[0049] A data set including a large number of images including abnormal regions such as bleeding and neovascularization, and annotations attached to these images, is used for the deep learning of the detection unit 118. The deep learning of the detection unit 118 is performed by making the detection unit 118 learn or re-learn this data set.

[0050] 〔process〕 An example of the processing executed by the information processing system 1 will be described below with reference to the flowcharts of Figures 6 to 9. A program stored in the main memory device 102 of the server 10 is started, and the processing of the information processing system 1 is executed by the management unit 120 of the server 10 as follows. Note that, hereinafter, the processing executed by the management unit 120 of the server 10 may be simply described as being executed by the "server 10".

[0051] The process performed by the server 10 is roughly divided into four steps, as shown in FIG. 6. First, the server 10 acquires an image of the subject's eye obtained by the ophthalmic apparatus 30 capturing an image of the retina (S1). In this example, a wide-angle fundus image P is used as the image captured by the ophthalmic apparatus 30. As described above, the wide-angle fundus image P is captured by the SLO system of the ophthalmic apparatus 30. The management unit 120 acquires the wide-angle fundus image P stored in the ophthalmic apparatus 30 via the network 5.

[0052] Next, the management unit 120 estimates the presence or absence of abnormal findings in the acquired wide-angle fundus image P (S3). After that, the management unit 120 performs an output process of the subject's eye image for which the presence or absence of abnormal findings has been estimated (S5).

[0053] Next, the management unit 120 performs additional processing (S7). The additional processing includes writing to the electronic medical chart, relearning, etc., and is mainly executed in response to instructions from the user.

[0054] (Estimation process) Details of the estimation process (S3) are shown in FIG.

[0055] Next, the management unit 120 acquires an estimation result of the detection unit 118 regarding the presence or absence of an abnormality (S33). When it is estimated that there is an abnormality, the output of the detection unit 118 includes information for identifying the type of abnormal finding and the abnormal region. An abnormal region is a region where a difference occurs from a normal eye. Examples of abnormal regions captured in the image of the subject's eye include bleeding points, neovascular regions, retinal detachment regions, and nonperfusion regions in the retina or choroid. Findings that are added based on abnormal regions are abnormal findings, and types of abnormal findings include the presence or absence and degree of bleeding, neovascularization, retinal detachment, and nonperfusion regions in the retina or choroid. FIG. 10 shows a wide-angle fundus image P including the macula M as an example of an abnormal finding. In the estimation result, an image region estimated to include an abnormal region and an identification result that identifies the type of abnormal finding are acquired.

[0056] The detection unit 118 outputs the likelihood of the estimation result and the severity of the abnormal finding together. Then, the management unit 120 generates abnormal finding information including an image region estimated to include an abnormal region, a discrimination result for discriminating the type of the abnormal finding, the likelihood of the estimation result, and the severity of the abnormal finding.

[0057] As a result of the above, when an abnormal finding with sufficient certainty is output, the abnormal finding information is associated with and stored in the wide-angle fundus image P. The wide-angle fundus image P associated with the abnormal finding information is generated by the estimation process.

[0058] (Output processing) Details of the output process (S5) will be described below with reference to Figs. 8, 10, and 11. Fig. 10 shows an example of a wide-angle fundus image P to which an abnormal finding has been added by the estimation process (S3). In the following, bleeding will be used as an example of an abnormal finding for explanation. In the wide-angle fundus image P, a bleeding point B is displayed as an abnormal area.

[0059] The management unit 120 uses the image processing unit 116 to extract partial images showing partial regions in the wide-angle fundus image P (S51). The management unit 120 sets partial regions in the wide-angle fundus image P to generate partial images. A plurality of partial images are extracted, and the plurality of partial images may have overlapping regions, or may not overlap each other. When an abnormal finding is given, the region of the partial image is determined so that the abnormal region is included in one of the plurality of partial images. When the plurality of abnormal regions are located apart from each other, the region of the partial image may be determined so that one partial image includes all the abnormal regions, or the region of the partial image may be determined so that the plurality of partial images include the abnormal regions.

[0060] 10 and 11, frames F1, F2, F3, and F4 shown superimposed on the wide-angle fundus image P indicate four partial regions extracted from the wide-angle fundus image P. Also shown on the right side of each figure are four partial images D1, D2, D3, and D4 generated as a result of extracting each partial region from the wide-angle fundus image P. The partial images D1, D2, D3, and D4 correspond to the regions indicated by the frames F1, F2, F3, and F4, respectively.

[0061] The image processor 116 extracts partial images from the wide-angle fundus image P so that any of the partial regions includes the bleeding point B, which is an abnormal region. In the examples of Figs. 10 and 11, the region of the frame F2 includes the bleeding point B, and the other regions (F1, F3, F4) do not include the bleeding point B. As shown as an example in Fig. 10, the image processor 116 determines the size and arrangement of the regions so that the partial images D1, D2, D3, and D4 all include the entire macula M. At the same time, the image processor 116 determines the size and arrangement of the regions so that the bleeding point B is included in any of the regions, in other words, so that the bleeding point B is displayed in any of the partial images. As a result, as shown on the right side of Fig. 10, the macula M is displayed in all of the partial images D1, D2, D3, and D4. In addition, the bleeding point B is displayed in the partial image D2.

[0062] When characteristic structures of the fundus, such as the macula and optic disc, are displayed in each partial image as in this embodiment, the position of each partial image and abnormal region in the wide-angle fundus image P is easily understood by the user.

[0063] In the partial image extraction example in FIG. 11, the image processing unit 116 determines the division positions so that there is no overlap between the frames F1 to F4, i.e., the regions corresponding to the partial images D1 to D4, with the macula M at the center. The image processing unit 116 also adjusts the arrangement and size of the partial images so that the bleeding point B is displayed in one of the partial images. As a result, the bleeding point B is displayed in the partial image D1.

[0064] If there is no abnormal region, the method of setting the partial images is determined according to a predetermined method. For example, the layout and size of each image may be determined so that the frames F1 to F4, i.e., the partial images D1 to D4, all include the entire macula M (FIG. 10). Alternatively, the division positions may be determined so that the frames F1 to F4, i.e., the partial images D1 to D4, are centered on the macula M and do not overlap with each other (FIG. 11).

[0065] The image processing unit 116 extracts partial images from the wide-angle fundus image P as described above, and creates partial images D1, D2, D3, and D4.

[0066] Next, the image processing unit 116 performs image processing to clarify the abnormal area (S53). In the example of Fig. 10 and Fig. 11, in the partial image D2 displaying the bleeding point B, processing is performed to change the pixel values ​​of the bleeding point B and other areas so that the bleeding point B is easily visible. In addition, the image processing unit 116 removes the reflection of eyelashes and other artifacts from the partial images D1, D2, D3, D4 and the wide-angle fundus image P to make the images easier to view.

[0067] The image processing unit 116 further performs a process of setting a method of highlighting the abnormal region (S55). The highlighting method is appropriately selected according to the type, severity, and likelihood of the abnormal finding. For example, when abnormal findings are found in multiple locations, a partial image including an abnormal region with a high severity is set to be preferentially displayed in the subsequent display process (S59). The image processing unit 116 also performs a process of setting a display method for displaying partial images in which abnormal regions are displayed among multiple partial images in a manner that distinguishes them from partial images in which abnormal regions are not displayed. Specifically, the image processing unit 116 selects the magnification ratio and display order of partial images in which abnormal regions are displayed according to the type or likelihood of the abnormal finding. Furthermore, the image processing unit 116 selects an image such as a frame or an icon to be displayed over the portion to be highlighted according to the type or likelihood of the abnormal finding, and sets it to be displayed as highlighted display in the subsequent display process (S59). Note that these highlighting methods may be set by accepting a user's input.

[0068] The image processing unit 116 creates a GUI (Graphical User Interface) (S57) according to the partial image extraction method, abnormal area highlighting method, and highlighting selected up to step S55. This GUI is transmitted to the terminal 20 via the network 5 and displayed on the output device 105 of the terminal 20 (S59). The GUI accepts instructions from the user operating the terminal 20, changes the display in response to these instructions, and also adds and displays images such as icons and frames.

[0069] Examples of the GUI are shown in Figs. 10 to 14. As shown in each figure, in the GUI, a wide-angle fundus image P is arranged on the left side, and four partial images D1 to D4 are arranged to the right of the wide-angle fundus image P. Frames F1, F2, F3, and F4 corresponding to the partial images D1, D2, D3, and D4 are displayed superimposed on the wide-angle fundus image P. The display of the frames F1 to F4 makes it easy to understand which areas of the wide-angle fundus image P each of the partial images D1 to D4 indicates.

[0070] In FIG. 10, partial image D2 in which bleeding point B, which is an abnormal region, is displayed at the largest size with priority.

[0071] 12 shows the GUI when an emphasis method (see S55) is selected to further enlarge and display partial images including abnormal regions. Moreover, all of the partial images D1 to D4 are set to display the macula M (see S51). As shown in the figure, the magnification of the partial image D2, in which the bleeding point B is displayed, is larger than those of the other partial images D1, D3, and D4. This makes it easier for the user to confirm the bleeding point B.

[0072] In the GUI of FIG. 13, the bleeding point B displayed in the partial image D2 is highlighted by a small frame S (see S55). In response to a user instruction, the image processing unit 116 further enlarges the image within the small frame S and displays it as an enlarged image L. This allows the user to easily confirm the bleeding point B. The user's instruction is given by, for example, clicking the image of the small frame S with the mouse or tapping on the display screen.

[0073] It is possible to display not only one enlarged image L, but also multiple images simultaneously. When there are multiple abnormal regions, the enlarged images L may be displayed in sequence in response to a user instruction. All enlarged images L may be arranged in sequence and displayed as thumbnails. In this case, it is desirable to arrange the enlarged images L based on the type, severity, or likelihood of the abnormal findings.

[0074] The magnification ratio or angle of view of the enlarged image L is set arbitrarily. For example, the magnification ratio is set to 5 times and the angle of view is set to 30 degrees, which are easy for a doctor to view. The magnification ratio or angle of view may be changed according to a user's instruction. Alternatively, the magnification ratio or angle of view may be set in advance by a doctor. Furthermore, the user can change the positions of the partial images D1 to D4 or the range of enlargement by manipulating the positions and sizes of the frames F1 to F4.

[0075] In the GUI of FIG. 14, highlighting is performed using icons IC1 to IC4. These highlighting methods differ depending on the type of abnormal finding. In FIG. 14, icons IC1 to IC4 that are displayed differently are used for four types of abnormal findings: an area that a doctor judges to be abnormal and adds a display, an abnormal area output by AI, an area that a doctor and AI judge to be abnormal, and an area that a doctor judges not to be abnormal and instructs to change the display on the GUI. In this way, the highlighting method is differentiated depending on the type of abnormal finding. The icons IC1 to IC4 are displayed differently from each other by, for example, changing the line type, shape, color, etc.

[0076] Furthermore, when there are multiple abnormal regions, it is possible to display the enlarged images in order of the severity of the abnormal findings. Furthermore, the display order of the partial images D1 to D4 or the display mode of the enlarged image L is changed according to the severity.

[0077] The highlighting method also changes depending on the severity. For example, an abnormal area with a higher severity is highlighted with a brighter or more saturated color.

[0078] 10 to 14 are not dependent on the image division method or the GUI display mode, and any type of emphasis can be selected. It is also possible to combine a plurality of emphasis methods. For example, in a GUI display in which a partial image D2 is enlarged as shown in FIG. 11, a small frame S and an enlarged image L may be displayed superimposed on the partial image D2.

[0079] The image processor 116 can also perform OCT imaging using an OCT system in cooperation with the ophthalmologic apparatus 30 to obtain a tomographic image (S61). An instruction for OCT imaging can also be received via a GUI in the display process (S59). For example, the user can specify an abnormal area for which OCT imaging is to be performed by an operation such as clicking the mouse.

[0080] When there is an instruction from the user, the image processor 116 identifies the fundus position where the abnormal area is located, and instructs the ophthalmologic apparatus 30 to capture an image via the network 5. Based on this instruction, the ophthalmologic apparatus 30 performs OCT imaging of the abnormal area using the OCT system. The tomographic image of the retina acquired by OCT imaging is acquired by the server 10, and is further displayed on the output device 105 of the terminal 20. At that time, it may be displayed on the GUI together with the wide-angle fundus image P or the partial images D1 to D4.

[0081] OCT photography may be performed without a user's instruction. For example, when an abnormal finding is found, the image processor 116 identifies the fundus position where the abnormal area is located, and instructs the ophthalmic device 30 to perform photography via the network 5. Based on this instruction, the ophthalmic device 30 performs OCT photography of the abnormal area using the OCT system.

[0082] During OCT imaging, imaging may be performed according to patient information. For example, if a diagnosis such as macular degeneration is obtained in the electronic medical record stored in the database 114, the server 10 instructs the ophthalmic device 30 to perform OCT imaging of the vicinity of the macula M. Based on this instruction, the ophthalmic device 30 performs OCT imaging of the abnormal area.

[0083] (Additional Processing) The additional process will be described with reference to the flow chart in Fig. 9. The user can confirm the abnormal findings and their regions in the display process (S59), and then add annotations to the estimation results and save them (S71). The saved annotations are stored in the database 114, and are saved in a state in which they are associated with the wide-angle fundus image P and the estimation results.

[0084] The saved annotations and wide-angle fundus image P are used for re-learning by the detection unit 118 (S73). The detection unit 118 can improve the ability to estimate abnormal findings by learning the wide-angle fundus image P together with annotations indicating the type of abnormal findings and the correct abnormal area, for example.

[0085] <Second embodiment> In the first embodiment described above, the detection unit 118 uses a trained model having a CNN. As an embodiment different from the first embodiment, an image processing algorithm may be used instead of a trained model that has undergone machine learning to estimate abnormal findings. This will be described below as a second embodiment.

[0086] The software configuration of the server 10 according to the second embodiment is shown in Fig. 15. In the second embodiment, the server 10 includes a detection unit 119 configured by an image processing algorithm.

[0087] In the second embodiment, the configurations of the server 10, the terminal 20, and the ophthalmologic apparatus 30 are the same as those of the first embodiment, except for the detection unit 119. The same reference numbers as those used in the first embodiment are used for the same configurations as those in the first embodiment, and the description thereof will be omitted.

[0088] The estimation process (S3) by the detection unit 119 in the second embodiment will be described below. In this example, an algorithm for estimating the presence or absence of an abnormal finding such as bleeding in the wide-angle fundus image P will be described.

[0089] In step S35, the detection unit 119 extracts a blood region. Here, the blood region is a region including a blood vessel region indicating a blood vessel and a bleeding region formed by bleeding from the blood vessel in the wide-angle fundus image P. As a result of the processing in step S35, an image in which the blood region is extracted from the wide-angle fundus image P is obtained.

[0090] In step S36, the detection unit 119 extracts only the blood vessel region.

[0091] A general image analysis method is applied to extract the regions in steps S35 and S36. For example, the wide-angle fundus image P is displayed in 256 gradations, and the gradation value of each pixel is compared with a threshold value to convert it into a binary image.

[0092] When binarizing, in step S35, a threshold is set so as to distinguish between blood regions and other regions. In step S36, a threshold is set so as to distinguish between blood vessel regions and other regions. Various methods such as a mode method can be used to set the threshold.

[0093] In step S37, the detection unit 119 removes noise from the binarized image, particularly around the vascular region. In the noise removal process, for example, dilation and erosion are used. Through step S37, an image in which the vascular region is extracted from the wide-angle fundus image P is obtained.

[0094] In step S38, the detection unit 119 can obtain an image showing the bleeding region by taking the difference between the image showing the blood region and the image showing the blood vessel region.

[0095] The detection unit 119 estimates the presence or absence of bleeding based on the image thus acquired (S39). When the detection unit 119 determines that bleeding is present, it outputs an abnormal finding of bleeding and an image showing the bleeding area.

[0096] Although an example of the estimation process for bleeding has been shown above, the same process is performed for other abnormal findings such as neovascularization. In this case, the parameters used for extracting the region, such as the threshold value for binarization, are appropriately set according to the type of abnormal finding.

[0097] <Effects> The image processing in the above embodiment is an image processing method including a process (S1) of acquiring a wide-angle fundus image (P) obtained by photographing with an ophthalmic device, a process (S1) of extracting a plurality of partial images (D1-D4) from the wide-angle fundus image to which an abnormal finding has been added, a selection process (S51) of selecting a first partial image from the plurality of partial images, which indicates an abnormal area (B) that is the basis of the abnormal finding, and a display process (S59) of displaying the first partial image. Includes.

[0098] With the above configuration, it is possible to obtain a display that covers a wide range of abnormal findings in the fundus and allows the abnormal area to be easily visually recognized.

[0099] In the display process (S59), partial regions in which abnormal findings are observed are displayed so as to be distinguishable on the wide-angle fundus image P. Furthermore, among the partial images D1 to D4, partial images in which abnormal findings are observed can be enlarged and displayed.

[0100] With this configuration, the user can easily visually confirm details of abnormal findings and abnormal areas.

[0101] In addition, in the display process (S59), it is also possible to highlight the abnormal area. This highlighting is changed depending on the type of abnormal finding.

[0102] With the above configuration, the details of the abnormal findings and the abnormal area can be easily visually confirmed. Furthermore, even when there are multiple types of abnormal findings, the user can organize and confirm the abnormal findings by type.

[0103] The above embodiment includes an estimation process (S3) for estimating an abnormal finding in the wide-angle fundus image P. The estimation process includes a process (S31) for inputting the wide-angle fundus image P to the detection units 118 and 119 that estimate an abnormal finding, and a process for causing the detection units 118 and 119 to estimate the abnormal finding.

[0104] Such estimation processing can assist doctors and medical staff in making diagnoses and providing findings, and can also prevent abnormalities from being overlooked.

[0105] The above embodiment includes a process of accepting addition of an annotation regarding an abnormal finding to a wide-angle fundus image P (S71), and a process of re-learning the detection unit 118 using the annotation and the wide-angle fundus image P (S73).

[0106] Since the detection unit 118 re-learns using the annotated data, it is possible to improve the accuracy of the estimation process of the detection unit 118. In other words, it is possible to reduce the probability of false positives, false negatives, and the like being output.

[0107] In the estimation process (S3), the severity of the abnormal finding is estimated, and the display method of the first region is changed according to the estimated severity.

[0108] The above embodiment further includes a process of identifying the position of the abnormal region on the fundus (S61), and a process of performing OCT imaging on the identified position by the OCT system to obtain a tomographic image at the identified position (S61).

[0109] With the above configuration, it is possible to not only check the wide-angle fundus image P, but also to accurately and easily obtain a tomographic image when it is desired to further check for abnormal findings.

[0110] <Modification> In the above embodiment, the partial regions of the wide-angle fundus image P are set by rectangular regions, but the present invention is not limited to this. For example, the image may be divided by extending a dividing line radially, or may be divided into slits, and any shape may be adopted for the dividing line and the partial region.

[0111] The annotation may also include the shape of the partial region setting. For example, the annotation may include a partial region setting shape that the user determines to be optimal, and the detection unit 119 may learn and re-learn the shape. In this case, the detection unit 119 outputs an optimal partial region shape and partial region position in response to the abnormal findings estimated by the detection unit 119. In the partial image extraction process (S51), partial image extraction of the wide-angle fundus image P may be performed according to the partial image extraction method output by the detection unit 119.

[0112] The display mode in the display process (S59) and the like is a mode in which abnormal areas and the like are displayed by distinguishing them by color or shape using images, but the present invention is not limited to such a mode based on visual confirmation. For example, a mode in which display variations are set by sound may be used.

[0113] In the above embodiment, an ophthalmic apparatus 30 having an OCT system is used, but the present invention is not limited to this, and may be an ophthalmic apparatus that does not have an OCT system, or an ophthalmic apparatus that uses another system.

[0114] In the above embodiment, a plurality of terminals are connected to one server 10 to perform the above-mentioned functions. The present invention does not limit the number of servers or the number of terminals, and for example, the above-mentioned functions may be realized by only one device. The number of terminals or the number of servers may be further increased. Furthermore, each function does not necessarily have to be realized by the server 10 or the like, and the functions may be shared among a plurality of devices to realize the functions. In other words, the present invention does not limit the number of control units or devices, or the sharing of functions among devices. [Explanation of symbols]

[0115] 1. Information Processing Systems 10 Server 20 Terminals 30 Ophthalmological equipment

Claims

1. A process of acquiring a wide-angle fundus image captured by an ophthalmic device; A process of extracting a plurality of partial images from the wide-angle fundus image to which an abnormal finding has been added; a selection process for selecting a first partial image showing an abnormal region that is the basis of the abnormal finding from the plurality of partial images; a display process for displaying the first partial image; An image processing method comprising:

2. The image processing method according to claim 1 , wherein the display process is a process of displaying a second partial image, which does not show the abnormal region, among the plurality of partial images, simultaneously with the first partial image.

3. The image processing method according to claim 2 , wherein the display process displays the first partial image and the second partial image in different display forms.

4. The image processing method according to claim 3 , wherein the display process displays the first partial image in a size larger than that of the second partial image.

5. 5. The image processing method according to claim 3, wherein the display process extracts the abnormal area from the first partial image, and displays the abnormal area at a higher display magnification than the second partial image.

6. The image processing method according to claim 1 , wherein the display process is a process of displaying a position display image, which displays the position of the first partial image in the wide-angle fundus image, simultaneously with the first partial image.

7. The image processing method according to claim 1 , further comprising a process of highlighting the abnormal area in the first partial image.

8. The image processing method according to claim 7 , wherein the highlighting is changed depending on a type of the abnormal finding.

9. A process of inputting the wide-angle fundus image to an estimator that estimates abnormal findings; The image processing method according to claim 1 , further comprising an estimation process for causing the estimator to estimate the abnormal finding in the wide-angle fundus image.

10. The image processing method according to claim 9 , wherein the estimator is a trained model that estimates abnormal findings.

11. A process of receiving an annotation regarding an abnormal finding in the wide-angle fundus image; The image processing method according to claim 10 , further comprising a process of re-learning the trained model using the annotations and the wide-angle fundus image.

12. The estimation process further includes a process of causing the estimator to estimate a severity of the abnormal finding, The display process includes: The image processing method according to claim 9 , further comprising a step of changing a display method of the first partial image depending on the severity.

13. A process of performing an image capture process using an OCT system at a position of the abnormal area in the wide-angle fundus image and acquiring a tomographic image at the position; The image processing method according to claim 1 , further comprising:

14. A program causing a computer to execute the image processing method according to any one of claims 1 to 13.

15. An image processing device comprising a processing unit that executes the image processing method according to any one of claims 1 to 13.

16. An ophthalmologic system comprising a processing unit that executes the image processing method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Image processor and image processing method

    JP2014110884A

  • Image processing device, image processing method, and program

    JP2018020024A

  • Image processing device, image processing method, and program

    JP2019202229A

  • Image processing method, image processing device and image processing program

    JP2020058647A

  • Image processing device and image processing program

    US20200069175A1