Image diagnosis support device, image diagnosis support system, and image diagnosis support method

The image diagnosis support system uses machine learning to classify infectious and non-infectious eye diseases by analyzing anterior segment images, enhancing diagnostic accuracy and ensuring appropriate treatment strategies.

JP7594758B2Active Publication Date: 2024-12-05NAT UNIV CORP TOKAI NAT HIGHER EDUCATION & RES SYST +2
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022500485
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-02-12
Filing Date
2021-02-12
Publication Date
2024-12-05
Estimated Expiration
2041-02-12

AI Technical Summary

Technical Problem

Existing image diagnosis systems struggle to accurately classify infectious and non-infectious eye diseases based on anterior segment images, leading to potential misdiagnosis and inappropriate treatment, as they lack the capability to determine the pattern of opacity in corneal infiltrates.

Method used

An image diagnosis support system utilizing machine learning to analyze anterior segment images of the eye, calculating feature amounts and determining the likelihood of infectious, non-infectious, or normal states through a determination unit, which includes a learning unit and a determination unit to assess the image patterns.

Benefits of technology

The system provides accurate classification of eye diseases with high accuracy, offering a certainty in diagnosis by calculating likelihoods and evaluation values, aiding in appropriate treatment selection and avoiding adverse drug reactions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594758000007
    Figure 0007594758000007
  • Figure 0007594758000008
    Figure 0007594758000008
  • Figure 0007594758000009
    Figure 0007594758000009
Patent Text Reader

Abstract

Provided are a diagnostic imaging assistance device, a diagnostic imaging assistance system and a diagnostic imaging assistance method with which it is possible to determine a state of a captured image of an anterior eye part by machine learning. A diagnostic imaging assistance system 10 comprises an image processing determination device 12 having: an image database 30 for storing a learning image PL of an anterior eye part and information relating to a state of the learning image PL; and a determination unit 36 for determining the state of a captured subject image P by carrying out machine learning on the basis of the image database 30. This configuration makes it possible to determine the state of the subject image P, which is a captured image of the anterior eye part of the subject, by machine learning.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image diagnosis support device, an image diagnosis support system, and an image diagnosis support method that determine a state in a captured image by machine learning. [Background technology]

[0002] In recent years, software that diagnoses eye diseases based on captured images has been developed. For example, the technology described in Non-Patent Document 1 is one such technology. This software makes it possible to assist in diagnosing the presence or absence of diseases and the type of disease based on images of the subject's eyes.

[0003] However, in diagnosis using such images, images taken using special equipment such as OCT (Optical Coherence Tomography) or a fundus camera are used, and the subject must travel to a location where these devices are installed. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Clinically applicable deep learning for diagnosis and referral in rentinal disease, Jeffry De Fauw, ea al. (Deep Mind), Nature Medicine 24, 1342-1350,(2018) Summary of the Invention [Problem to be solved by the invention]

[0005] By the way, eye diseases can be largely classified into infectious diseases and non-infectious diseases, and the subsequent treatment methods vary greatly depending on the classification, so it is important to determine the classification. For example, if a therapeutic drug effective for non-infectious diseases is applied to infectious diseases, the symptoms may worsen, and vice versa. However, such a determination is difficult unless the doctor is an experienced specialist, and existing technologies such as the technology described in Non-Patent Document 1 above do not perform the above-mentioned classification determination.

[0006] For example, while both infectious and non-infectious corneal infiltration cause white opaque lesions on the clear cornea, it is said that there are characteristics between infectious and non-infectious infiltration in terms of the location, shape, number, etc. of the opacity. Ophthalmologists decide on the treatment method based on their experience from the findings of the opacity, but there are cases where it is difficult to make a decision.

[0007] On the other hand, the inventors have discovered that the appearance patterns of opacity and the like seen in the cornea of ​​the eye differ between infectious diseases and non-infectious diseases, and that diagnosis can be aided by evaluating whether the condition of the subject's eye resembles the appearance pattern of an infectious disease or the appearance pattern of a non-infectious disease.

[0008] The present invention is intended to solve such problems, and has an object to provide an image diagnosis support device, an image diagnosis support system, or an image diagnosis support method that can determine the state of a captured image of the anterior segment through machine learning. [Means for solving the problem]

[0009] In order to achieve this objective, the gist of the first invention of the present application is an image diagnosis support device having a learning unit that calculates, through machine learning, feature amounts for each of the infectious and non-infectious states for determining the state of a judgment image of the anterior eye based on a learning image of the anterior eye and information about the state in the learning image.

[0010] The gist of the second invention is an image diagnosis support device that has a judgment unit that judges the state of a judgment image of the anterior eye using features calculated by machine learning based on a learning image of the anterior eye and information about the state in the learning image, and the judgment unit calculates the likelihood that the state of the judgment image is infectious and the likelihood that it is non-infectious, respectively.

[0011] The gist of the third invention is an image diagnosis support device comprising an image storage unit that stores a learning image of the anterior eye and information on the state in the learning image, a learning unit that calculates features for judging the state of a determination image of the anterior eye by machine learning based on the image storage unit, and a judgment unit that judges the state of the determination image of the anterior eye using the features, wherein the judgment unit respectively calculates the likelihood that the state of the determination image is infectious and the likelihood that it is non-infectious. Effect of the Invention

[0012] According to the first invention, the learning unit calculates, through machine learning, features for each of the infectious and non-infectious states for determining the state of the judgment image of the anterior eye based on the learning image of the anterior eye and information about the state in the learning image.

[0013] According to the second invention, the determination unit determines the state of the determination image of the anterior eye using the feature amount calculated by machine learning. The determination unit also calculates the likelihood that the state of the determination image is infectious and the likelihood that the state is non-infectious. In this way, the likelihood is included in the determination result, so that the degree of confidence that the state is the determined result can be obtained.

[0014] According to the third invention, the state of the judgment image of the anterior eye can be judged by machine learning based on an image storage unit that stores a learning image of the anterior eye and information on the state in the learning image in advance. The judgment unit calculates the likelihood that the state of the judgment image is infectious and the likelihood that the state is non-infectious. In this way, the judgment result includes the likelihood, so that the degree of confidence that the state is the judgment result can be obtained.

[0015] Preferably, a fourth invention is characterized in that in the second or third invention, the judgment unit further calculates a likelihood that the state of the judgment image is normal. In this way, since the judgment result includes the likelihood, it is possible to obtain a degree of confidence that the state determined as the judgment result is the normal state.

[0016] Preferably, the fifth invention is characterized in that in any one of the second to fourth inventions, the judgment unit judges whether the state of the judgment image is an infectious disease, a non-infectious disease, or a normal state. In this way, it is possible to obtain a judgment as to whether the anterior eye in the captured image is an infectious disease, a non-infectious disease, or a normal state.

[0017] Preferably, the sixth invention is characterized in that, in any one of the second to fifth inventions, the judgment unit calculates an evaluation value for evaluating the judgment image based on the feature amount of the judgment image calculated for each of the infectious, non-infectious, and normal states. In this way, an evaluation value for evaluating the judgment image different from the feature amount can be calculated based on the feature amount. For example, a single evaluation value can be calculated based on multiple feature amounts, making the evaluation easy.

[0018] Preferably, the seventh invention is characterized in that, in any one of the second to fifth inventions, the likelihood that the state of the judgment image is infectious, non-infectious, or normal, or an evaluation value for evaluating the judgment image is calculated and output. In this way, machine learning is performed based on the learning image of the anterior eye and information on the state in the learning image, and the result of the machine learning can be used to obtain information on the likelihood or evaluation value of the state of the judgment image being infectious, non-infectious, or normal. By obtaining information on the state with the highest likelihood or evaluation value, not only is information for selecting a suitable treatment method provided, but information on states with low likelihood or evaluation value is also obtained, and when applied to a state with low likelihood or evaluation value, information necessary for judgment such as avoiding undesirable therapeutic drugs due to side effects is provided, which is useful.

[0019] Preferably, the eighth invention is characterized in that in the sixth invention, the judgment unit calculates the evaluation value by applying a Softmax function to the feature amounts of the judgment image calculated for each of the infectious, non-infectious, and normal states. In this way, the evaluation value can be calculated as a value between 0 and 1, making it easy to perform evaluation using the evaluation value.

[0020] Preferably, a ninth invention is characterized in that in the sixth or eighth invention, the judgment unit selects and outputs at least one of a plurality of sentences stored in advance according to the evaluation value. In this way, a sentence according to the evaluation value is output, so that the evaluation can be obtained intuitively.

[0021] Preferably, the tenth invention is characterized in that in any one of the second to ninth inventions, the determination unit estimates the name of a disease. In this way, information on the estimated name of a disease can be obtained in addition to information on the state of the anterior eye.

[0022] Preferably, an eleventh invention is characterized in that in any one of the third to tenth inventions, the learning unit re-executes the machine learning based on a confirmed state of a judgment made by the judgment unit in the past. In this way, the machine learning can be updated based on the judgment already made by the judgment unit and its validity.

[0023] Preferably, the 12th invention is characterized in that, in any one of the 2nd to 11th inventions, the device has an image cutting section that detects the position of the part to be determined from the image for determination and cuts out the image so as to include the position of the part to be determined, and the determination section judges the state of the anterior eye segment from the cut-out image cut out by the image cutting section. In this way, when an image is taken, it is sufficient to take the image so as to include the anterior eye segment, and the degree of freedom of the image that can be handled is increased.

[0024] Preferably, the thirteenth invention is characterized in that, in any one of the second to twelfth inventions, the device further comprises a camera for capturing an image including the anterior eye of the subject. In this way, an image including the anterior eye is captured by a camera included in an image diagnosis support device, so that the image capturing and the evaluation of the image can be performed by the same image diagnosis support device. More preferably, the camera is characterized in that it captures a visible image under visible light. In this way, it is possible to use a general-purpose camera device equipped in various devices.

[0025] Also preferably, the 14th invention is an image diagnosis support system including any one of the image diagnosis support devices of the 2nd to 12th inventions and a terminal device capable of information communication with the image diagnosis support device, the terminal device having a camera for taking an image including the anterior eye of a subject and a transmission unit for transmitting the image taken by the camera to the image diagnosis support device, the image diagnosis support device uses the image transmitted from the terminal device to determine the state of the anterior eye of the subject. In this way, the terminal device having the camera and the image diagnosis support device can be physically located apart, as long as they are capable of information communication. In other words, diagnosis support for an image taken at a separate location is possible. More preferably, the camera is characterized in that it takes a visible image under visible light. In this way, it is possible to use a general-purpose camera device equipped in various devices.

[0026] Also preferably, the fifteenth invention is an image diagnosis support method including a learning step of performing machine learning based on a learning image of the anterior eye and information on the state in the learning image, and a judgment step of judging the state of the judgment image using the learning result of the learning step, the judgment step calculating the likelihood that the state of the judgment image is infectious and the likelihood that the state of the judgment image is non-infectious. In this way, machine learning is performed based on the learning image of the anterior eye and information on the state in the learning image, and the state of the judgment image can be judged using the result of the machine learning. In addition, since the judgment result includes the likelihood, it is possible to obtain a degree of confidence that the state is the judgment result.

[0027] Also preferably, the 16th invention is an image diagnosis support method having a learning step of performing machine learning based on a learning image of the anterior eye and information on the state in the learning image, a likelihood calculation step of calculating the likelihood that the state of the judgment image is infectious and the likelihood that the state is non-infectious using the learning result of the learning step, and an output step of outputting the evaluation value or the likelihood of the judgment image calculated based on the likelihood for each of the infectious and non-infectious states. In this way, machine learning is performed based on the learning image of the anterior eye and information on the state in the learning image, and the result of the machine learning can be used to obtain information on the evaluation value or the likelihood that the state of the judgment image is either infectious or non-infectious. Since the treatment policy differs greatly depending on whether it is infectious or non-infectious, this information can be obtained by an ophthalmologist or the like and used as diagnostic support information for the pathological condition of the anterior eye. Knowing the likelihood information for the state with the highest likelihood not only provides information for selecting an appropriate treatment method, but also provides information for states with low likelihood, which, when applied to states with low likelihood, is useful in providing the necessary information for making decisions such as avoiding undesirable therapeutic drugs due to side effects, etc.

[0028] Preferably, the 17th invention is the image diagnosis support method according to the 16th invention, characterized in that the likelihood calculation step includes a step of calculating the likelihood that the state of the judgment image is normal, and the output step includes a step of further outputting the evaluation value or the likelihood of the judgment image calculated based on the likelihood for the normal state. In this way, machine learning is performed based on the learning image of the anterior eye and information on the state in the learning image, and the result of the machine learning can be used to obtain information on the likelihood that the state of the judgment image is normal. This information can be obtained by an ophthalmologist or the like and used as diagnostic support information for the pathology of the anterior eye.

[0029] Also preferably, the 18th invention is the image diagnosis support method of the 16th invention, characterized in that the likelihood calculation step includes a step of setting a plurality of judgment regions in the judgment image, and calculating the judgment region likelihood of an infectious state and the judgment region likelihood of a non-infectious state for each of the judgment regions, and the output step includes a step of outputting the evaluation value of the judgment image calculated based on the judgment region likelihood of the different judgment regions for the infectious state and the non-infectious state or the judgment region likelihood of the different judgment regions. In this way, machine learning is performed based on the learning image of the anterior eye and information on the state in the learning image, and the evaluation value or likelihood information that the state of the anterior eye is infectious and non-infectious is provided to the ophthalmologist using the result of the machine learning as unique information of the judgment image that does not depend on the selection of the judgment region.

[0030] Preferably, a 19th invention is characterized in that in the first invention, the learning unit further calculates, by the machine learning, a feature amount for a normal state for determining the state of the determination image of the anterior eye, based on the learning image of the anterior eye and information on the state in the learning image. In this way, the learning unit further calculates, by the machine learning, a feature amount for a normal state for determining the state of the determination image of the anterior eye, based on the learning image of the anterior eye and information on the state in the learning image. [Brief description of the drawings]

[0031] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image diagnosis support system according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram for explaining the main functions of a terminal device constituting the image diagnosis support system of FIG. 1. FIG. [Diagram 3] 2 is a block diagram for explaining the main functions of an image diagnosis support device constituting the image diagnosis support system of FIG. 1. FIG. [Figure 4] 4 is a diagram for explaining an outline of the operation of the determination unit in FIG. 3. [Diagram 5]2 is a flowchart illustrating an example of a control operation of the image diagnosis support system of FIG. 1. [Figure 6] 11A and 11B are diagrams showing experimental results of the image diagnosis support device of the present invention. [Figure 7] 13A and 13B are diagrams showing the results of another experiment on the image diagnosis support device of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of the configuration of an image diagnosis support device in another embodiment of the present invention, and corresponds to FIG. 1. [Figure 9] 9 is a block diagram for explaining the main part of the functions of the image diagnosis support device constituting the image diagnosis support device of FIG. 8, and corresponds to FIGS. 2 and 3. FIG. [Figure 10] 9 is a flowchart illustrating an example of a control operation of the image diagnosis support apparatus of FIG. 8, and corresponds to FIG. 5. [Figure 11] 10 is a flowchart illustrating an example of the operation of a learning unit and a determination unit in another embodiment of the present invention. [Figure 12] 11A and 11B are diagrams illustrating examples of rectangular regions (judgment regions) and their corresponding judgment values. [Figure 13] FIG. 13 is a diagram illustrating an example of a calculated determination value. [Figure 14] FIG. 13 is a diagram illustrating an example of an output. [Figure 15] FIG. 13 is a diagram illustrating an example of an output. [Figure 16] FIG. 13 is a diagram illustrating an example of an output. [Figure 17] FIG. 13 is a diagram illustrating an example of an output. [Figure 18] FIG. 13 is a diagram illustrating an example of an output. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. EXAMPLES

[0033] FIG. 1 is a diagram illustrating the configuration of an image diagnosis support system 10 (hereinafter, simply referred to as the support system 10) according to an embodiment of the present invention. As shown in FIG. 1, the support system 10 of this embodiment is configured to include an image processing determination device 12 and a terminal device 14. The image processing determination device 12 is configured to include a CPU 20, which is a central processing unit, a ROM 22, which is a read-only memory, a RAM 24, which is a readable and writable memory, a storage device 26, and a communication device 28. The storage device 26 is preferably a known storage device (storage medium) such as a hard disk drive capable of storing information. It may also be a removable storage medium such as a small memory card or an optical disk. The image processing determination device 12 is a so-called computer that processes and controls electronic information using the ROM 22, RAM 24, etc., based on a predetermined program previously stored in the storage device 26, etc. Note that the image processing determination device 12 may be provided with an input device such as a keyboard or mouse that accepts input from an operator, and an output device such as a display or a printer, as necessary. These input / output devices are connected to the image processing determination device 12 via an input / output interface.

[0034] The communication device 28 is a so-called network interface, and is connected to other devices by various connection methods, whether wired or wireless, so that information can be communicated therewith. In this embodiment, the image processing determination device 12 and the terminal device 14 are connected to each other by wireless communication so that information can be communicated therewith.

[0035] The storage device 26 is provided with an image database 30. This image database 30 corresponds to the image storage unit of the present invention. In this image database 30, a plurality of images of the anterior eye are stored as learning images, and information on the state of the anterior eye is stored as a label for each of the plurality of images. In this embodiment, the state of the anterior eye is either an infectious disease, a non-infectious disease, or normal. The anterior eye refers to the part of the eyeball in front of the crystalline lens, and in this embodiment, it corresponds to the eyeball or cornea of ​​the subject, which can be observed from the outside of the subject. The plurality of images are images of the anterior eye taken in advance as cases, for example, images obtained by a visible camera under visible light, and are equivalent to images seen when a doctor observes the anterior eye of a patient from the outside when examining the anterior eye of the patient. The information on the state of the anterior eye is obtained, for example, as a result of a specialist actually examining and diagnosing the subject corresponding to the image of the anterior eye.

[0036] As shown in FIG. 1, the terminal device 14 includes a CPU 40, a ROM 42, a RAM 44, and a communication device 50, and their functions are the same as those of the CPU 20, the ROM 22, the RAM 24, and the communication device 28 in the image processing determination device 12. The terminal device 14 also includes a storage device 46. The storage device 46 may be a known storage medium such as a semiconductor memory capable of storing information, or may be a hard disk drive or the like similar to the storage device 26 of the image processing determination device 12. The terminal device 14 is a so-called computer that processes and controls electronic information using the ROM 42, the RAM 44, etc. based on a predetermined program previously stored in the ROM 42, the storage device 46, etc. Note that if the capacity of the information to be stored in the terminal device 14 can be covered by the RAM 44, there is no need to provide the storage device 46.

[0037] The terminal device 14 is also provided with a camera 48 as an imaging device, a display device 52 such as a liquid crystal panel, and an input device 54 for accepting operations from an operator. The input device 54 may be, for example, a transparent touch panel that is placed on top of the display device 52. The image processing determination device 12 may also be provided with input devices such as a keyboard and a mouse for accepting inputs from an operator, and output devices such as a display and a printer, as necessary. These input / output devices are connected to the image processing determination device 12 via an input / output interface.

[0038] The memory device 46 stores programs for operating the terminal device 14, as well as saving images captured by the camera 48 and storing information transmitted from other devices via the communication device 50.

[0039] The camera 48 is a visible light camera that captures images under visible light, and saves the captured images as electronic data in a predetermined format, which are then stored in the aforementioned storage device 46 or the like. The display device 52 displays information about the operation of the terminal device 14, specifically, for example, information about the operation and settings of the camera 48, and a preview of the image to be captured. The input device 54 receives operations from the terminal device 14, and is used, for example, to function as a shutter switch for the camera 48, or to perform an operation to transmit the captured image to the image processing and determination device 12.

[0040] 2 is a functional block diagram for explaining the main parts of the control function provided in the terminal device 14. As shown in FIG. 2, the terminal device is configured to functionally include an imaging unit 62 and a communication control unit 64. The imaging unit 62 controls the camera 48 in response to an operation by an operator via the input device 54 described above to capture an image including the anterior eye of the subject (hereinafter referred to as a subject image P). The captured subject image P is also stored in the storage device 46.

[0041] The communication control unit 64 transmits the subject image P stored in the storage device 46 to the image processing determination device 12 via the communication device 50.

[0042] Fig. 3 is a functional block diagram explaining the main control functions of the image processing judgment device 12. As shown in Fig. 3, the image processing judgment device 12 functionally comprises an image reading unit 32, an image clipping unit 34, a learning unit 35, and a judgment unit 36.

[0043] Of these, the image reading unit 32 reads out a subject image P to be determined by the determining unit 36, which will be described later. This subject image P is, for example, received by the communication control unit 30 and stored in the storage device 26.

[0044] The image cutout unit 34 cuts out the anterior eye part, more specifically the cornea, from the subject image P read out by the image reading unit 32 into a predetermined shape, for example, a rectangular shape, so that the anterior eye part, more specifically the cornea, occupies most of the image, and stores it as a cutout image P'. For example, the cutout may be performed so that the cutout image P' includes the entire cornea and the number of pixels from the edge of the cornea to the edge of the image in the cutout image P' is equal to or greater than a predetermined number of pixels, or so that at least a part of the edge of the cornea is included in the cutout image P'. The image cutout unit 34 may cut out the image based on a predetermined algorithm, for example, or may cut out an area of ​​a shape specified by the operator in the subject image P by the operation of the operator. In this way, when photographing the subject image P, it is not necessary to position the cornea at the center of the image, or to position the cornea to a desired size in the image. This cutout image P' corresponds to the image for determination, and the area including the cornea corresponds to the main part for determination. Note that, if subject image P is suitable for judgment by judging section 36 described below, image cropping section 34 does not need to generate cropped image P'. In other words, cropped image P' may be subject image P itself. In other words, in such a case, the captured subject image P corresponds to the judgment image.

[0045] The learning unit 35 calculates the feature amount used for determining the cut-out image P' in the determining unit 36 ​​described later by machine learning. Specifically, the learning unit 35 calculates the feature amount as follows. First, the learning unit 35 calculates a feature amount representing the feature from each of the learning images PL in the image database 30 in advance. In this embodiment, the feature amount is calculated from the corresponding learning image PL for each state according to information indicating the state of each image among the learning images PL stored in the image database 30, that is, whether each image is infectious, non-infectious, or normal. As a result, the feature amount for each of infectious, non-infectious, and normal is calculated. The calculated feature amount is stored in a storage device in the device 12, such as the storage unit 26. Specifically, the calculation of the feature amount is performed by an algorithm such as YOLOv3 proposed by Joseph Redmon et al. The operation of the learning unit 35, that is, the calculation of the feature amount, corresponds to the learning process.

[0046] The determination unit 36 ​​determines whether the image of the anterior eye in the cut-out image P' cut out by the image cut-out unit 34 is in an infectious disease state, a non-infectious disease state, or a normal state. The determination unit 36 ​​makes the determination by taking into consideration the similarity between the cut-out image P' and the feature amount for each state calculated by the learning unit 35. More specifically, for example, the determination unit 36 ​​makes the determination by an algorithm such as YOLOv3 proposed by Joseph Redmon et al. That is, the determination unit 36 ​​randomly sets a plurality of rectangular regions in the cut-out image P' as regions for determination (determination regions) having different sizes and positions, compares each of the plurality of rectangular regions with the feature amount for each state stored in the storage unit 26 or the like, and sets a determination value (likelihood). (Hereinafter, the determination region will be described as a rectangular region, but the determination region may be any shape other than a rectangle.) The higher the determination value, the higher the similarity between the teacher image for each state and the cut-out image P'. That is, if the extracted image P' is definitely estimated to be infectious, the calculated infectivity judgment value will be 1, and if it is definitely estimated not to be infectious, the judgment value will be 0. Similarly, judgment values ​​are also set for non-infectiousness and normality.

[0047] As described above, the feature amounts are calculated and stored according to whether the state of the anterior eye part in the learning image PL is in any of the states of an infectious disease, a non-infectious disease, or normal. Therefore, the determination unit 36 determines, for the feature amount that was the target when the highest determination value was given, whether the state of the corresponding learning image is any of the states of an infectious disease, a non-infectious disease, or normal, as the state of the cut-out image P'. The operation of this determination unit 36, that is, the calculation of each determination value and their comparison determination, corresponds to the determination step.

[0048] FIG. 4 is a diagram for explaining an outline of the determination operation by the determination unit 36. Rectangles R1 and R2 in FIG. 4 respectively show two of the plurality of rectangular regions set by the determination unit 36 in the cut-out image P'. Here, rectangle R1 is similar to the feature amount indicating a non-infectious disease among the feature amounts calculated by the learning unit 35, and the determination unit 36 determines that its determination value is Pr1. On the other hand, rectangle R2 is similar to the feature amount indicating an infectious disease among the feature amounts calculated by the learning unit 35, and the determination unit 36 determines that its determination value is Pr2. In such a case, the determination unit 36 compares the respective determination values Pr1 and Pr2, and adopts the rectangle with the highest determination value as the determination result. That is, when Pr1 < Pr2, rectangle R2 is adopted as the determination result, so it is determined that the state of the cut-out image P' is an infectious disease, which is the state corresponding to the feature amount to which rectangle R2 is considered similar.

[0049] FIG. 5 is a flowchart for explaining an example of the control operation of the image diagnosis support system 10 of the present embodiment, which describes a comparison between the operation of the terminal device 14 and the operation of the image processing determination device 12. Note that the calculation of the feature amount by the learning unit 35 in the image processing determination device 12 is executed, for example, prior to the execution of this flowchart, so the illustration is omitted. In other words, this flowchart is executed from the state where the feature amount has been obtained.

[0050] First, in step S1 (hereinafter, the word "step" will be omitted) corresponding to the photographing unit 62 of the terminal device 14, an image of the anterior eye of the subject is photographed as a subject image P based on the operation of the operator. At this time, the operator may be the same as the subject, or may be a different person. Note that the subject image P does not have to be an image including only the anterior eye as described above, and may be an image including, for example, the entire face or a part of the face, so long as it includes the anterior eye.

[0051] In S2 corresponding to the communication control unit 64 of the terminal device 14, the subject image P captured in S1 is transmitted from the terminal device 14 to the image processing and determination device 12. This transmission is performed in a manner that allows wireless information communication, for example, via a so-called wireless LAN or a mobile phone line.

[0052] In S3 corresponding to the communication control unit 64 of the image processing determination device 12, the subject image P transmitted in S2 is received and stored in the storage device 26, the RAM 24, or the like.

[0053] Next, in S4 corresponding to the image cropping unit 34, a cropped image P' is cropped as an image specialized for the anterior eye segment from the subject image P received in S3. This cropped image P' is obtained by cropping the subject image P to have a predetermined shape, such as a rectangle, so as to include the anterior eye segment, i.e., the cornea, and to minimize other parts as much as possible.

[0054] In S5 corresponding to the judgment unit 36, the state of the anterior eye in the cut-out image P' cut out in S4 is judged by machine learning. That is, a plurality of rectangular regions R are set in the cut-out image P', and each of the plurality of rectangular regions R is compared with a feature amount corresponding to each state, and a similarity is calculated as a judgment value. Then, the state corresponding to the feature amount with the highest judgment value is judged as the state of the cut-out image P'.

[0055] In S6 corresponding to the communication control unit 64, the result of the determination in S5 is transmitted to the terminal device 14.

[0056] In S7 corresponding to the communication control unit 64 of the terminal device 14, the judgment result transmitted in S6 is received. In addition, in S8, the judgment result received in S7 is displayed on the display device 52 of the terminal device 14. In this manner, since the judgment result is displayed on the display device 52 of the terminal device 14 that captured the subject image P, the subject can know the judgment result even if the terminal device 14, which is in the same place as the subject, is away from the location of the image processing judgment device 12.

[0057] (Experimental Example 1) In order to verify the effectiveness of the image diagnosis support device of the present invention, the inventors prepared 320 anterior eye images and performed an evaluation experiment by 5-fold cross-validation. That is, the 320 anterior eye images were divided into 5 groups. Then, one of the groups was set as a subject image P, and the remaining four groups were set as learning images PL, and the subject images P were judged in sequence as described in the above-mentioned embodiment so that each of the above 5 groups became a subject image P. Of the 320 anterior eye images, 100 points indicated infectious diseases, 96 points indicated non-infectious diseases, and 124 points indicated normal conditions.

[0058] FIG. 6 is a diagram showing the judgment result. As shown in FIG. 6, for 100 images showing infectious diseases, the image diagnosis support device of the present invention, i.e., the judgment unit 36 ​​of this embodiment, judged 88 of the images to be infectious diseases, with an accuracy rate of 88%. For 96 images showing non-infectious diseases, the image diagnosis support device of the present invention judged 81 of the images to be non-infectious diseases, with an accuracy rate of 84.4%. For 124 images showing a normal state, the image diagnosis support device of the present invention judged 118 of the images to be normal, with an accuracy rate of 95.2%. In addition, the judgments were correct for 287 of 320 images overall, with an accuracy rate of 89.7%. In this way, the image diagnosis support device of the present invention was able to output judgment results with high accuracy.

[0059] (Experimental Example 2) Next, 195 anterior eye images were prepared, and an evaluation experiment was performed using 5-fold cross-validation in the same manner as in the above-mentioned Experimental Example 1. Of the 195 anterior eye images in this experimental example, 99 points indicated infectious diseases, 96 points indicated non-infectious diseases, and no points indicated a normal state, so the determination means 36 made a determination between two classifications (2 classifications).

[0060] Fig. 7 is a diagram showing the judgment results. As shown in Fig. 7, for 99 images showing infectious diseases, the image diagnosis support device of the present invention judged 89 to be infectious diseases, with an accuracy rate of 89.9%. For 96 images showing non-infectious diseases, the image diagnosis support device of the present invention judged 84 to be non-infectious diseases, with an accuracy rate of 87.5%. In total, 173 out of 196 images were correctly judged, with an accuracy rate of 88.3%.

[0061] On the other hand, the inventors of the present application conducted a comparative example with the cooperation of doctors. In the comparative example, 27 ophthalmologists (hereinafter referred to as "doctors") were shown 196 anterior eye images, and the images were classified into two categories: infectious disease and non-infectious disease. Of the 196 anterior eye images in this comparative example, 100 of them showed infectious disease and 96 showed non-infectious disease. The 27 doctors included both corneal specialists and non-specialists. The accuracy rate of the doctors in this comparative example was 86.7% for the anterior eye images of infectious disease and 74.9% for the anterior eye images of non-infectious disease. In addition, the accuracy rate of the doctors in the entire 196 anterior eye images was 80.9%. It was found that the judgment results of the image diagnosis support device of the present invention were comparable to those of the doctors, at least in terms of the accuracy rate.

[0062] According to the image diagnosis support system 10 of this embodiment, the learning unit 35 calculates, through machine learning, features for determining the state of the captured image of the anterior eye based on the learning image PL of the anterior eye and information about the state in the learning image PL.

[0063] Furthermore, according to the image diagnosis support system 10 of this embodiment, the state of the determination image of the anterior eye segment is determined by the determination unit 36 ​​using the feature amount calculated by machine learning.

[0064] Furthermore, according to the image diagnosis support system 10 of this embodiment, an image processing and judgment device 12 is included which has an image database 30 which stores learning images PL of the anterior eye and information on the state in the learning images PL, and a judgment unit 36 ​​which judges the state of a subject image P by machine learning based on the image database 30. Therefore, the state of the subject image P, which is a photographed image of the anterior eye of the subject, can be judged by machine learning.

[0065] Furthermore, according to the image diagnosis support system 10 of this embodiment, the determining unit 36 ​​determines whether the captured subject image P indicates an infectious disease, a non-infectious disease, or a normal state.

[0066] Furthermore, according to the image diagnosis support system 10 of this embodiment, the system has an image cutting unit 34 which detects the position of the cornea from the subject image P and cuts out the image so as to include the position of the cornea, and the judgment unit 36 ​​judges the state of the anterior segment from the cut-out image P' cut out by the image cutting unit 34. Therefore, when capturing an image, it is sufficient to capture the image so as to include the anterior segment, and the degree of freedom in the images that can be handled is increased.

[0067] According to the image diagnosis support system 10 of the present embodiment, the image processing determination device 12 is configured to include the image processing determination device 12 and the terminal device 14 capable of information communication with the image processing determination device 12, the terminal device 14 has a camera 48 that captures an image including the anterior eye of the subject, and a communication device 50 that transmits the subject image P captured by the camera 48 to the image diagnosis support device 14, and the image processing determination device 12 determines the state of the anterior eye of the subject using the subject image P transmitted from the terminal device 14, so that the terminal device 14 having the camera 48 and the image processing determination device 12 can be located physically apart as long as they can communicate with each other. In other words, diagnostic support of images captured at separate locations is possible.

[0068] Furthermore, according to the image diagnosis support system 10 of this embodiment, the camera 48 captures visible images under visible light, so it is possible to use a general-purpose camera device that is equipped on various devices, such as smartphones, mobile phones with cameras, tablet PCs, and digital cameras with communication functions.

[0069] Next, another embodiment of the present invention will be described. In the following description, parts common to the embodiments are given the same reference numerals and the description thereof will be omitted. EXAMPLES

[0070] 8 is a diagram for explaining the configuration of an image diagnosis support device 110 according to another embodiment of the present invention. In the above-mentioned embodiment, the image diagnosis support system 10 is composed of an image processing determination device 12 and a terminal device 14, but in this embodiment, a function equivalent to that of the image diagnosis support system 10 is realized by a single image diagnosis support device 110.

[0071] 8, the image diagnosis support device 110 includes a CPU 120, a ROM 122, a RAM 124, a storage device 126, a camera 148, a display device 152 as an output device, and a touch panel 154 as an input device. Preferably, the image diagnosis support device 110 is hardware such as a tablet PC, a smartphone, or a PDA that includes the camera 148 and the display 152 with a built-in touch panel 154, and in such a case, it has an advantage in terms of portability.

[0072] The CPU 120, ROM 122, RAM 124, memory device 126, camera 148, display device 152, and touch panel 154 have the same functions as the CPU 20, 40, ROM 22, 42, RAM 24, 44, memory device 26, 46, camera 48, display device 52, and input device 54 in the previously described embodiments, and so their explanations will be omitted.

[0073] Fig. 9 is a functional block diagram for explaining the main parts of the control function provided in the image diagnosis support device 110 of Fig. 1. As shown in Fig. 9, the image diagnosis support device 110 functionally includes an imaging unit 162, an image reading unit 132, an image clipping unit 134, a learning unit 135, a determination unit 136, and a display control unit 138.

[0074] The photographing unit 162 corresponds to the photographing unit 62 in the above-mentioned embodiment, and controls the camera 148 to photograph an image including the anterior eye of the subject (hereinafter referred to as a subject image P) by the operation of the operator via the above-mentioned input device 154. The photographed subject image P is stored in the storage device 126, the RAM 124, or the like. At this time, for example, the photographing is performed while a preview screen of the camera 148 is displayed on the display device 152, thereby improving the convenience of the photographer.

[0075] Image reading unit 132 corresponds to image reading unit 32 in the above-described embodiment, and reads out subject image P stored in RAM 124, storage device 126, or the like. This subject image P is, for example, photographed by imaging unit 162 and stored in storage device 126, or the like.

[0076] The image cutout unit 134 corresponds to the image cutout unit 34 in the above-described embodiment, and performs the same operation. That is, the image cutout unit 134 cuts out a predetermined shape, for example a rectangle, from the subject image P read by the image reading unit 132 so that the anterior eye segment, more specifically the cornea, occupies most of the image, and stores the cut-out image P'.

[0077] The learning unit 135 corresponds to the learning unit 35 in the above-described embodiment and performs the same operation. That is, the learning unit 135 calculates feature amounts for each state of the learning images PL stored in the learning database 130, that is, for each of infectious disease, non-infectious disease, and normal.

[0078] The determination unit 136 corresponds to the determination unit 36 ​​in the above-described embodiment and performs the same operation. That is, the determination unit 136 determines whether the image of the anterior eye segment in the cut-out image P′ cut out by the image cut-out unit 134 is in an infectious disease state, a non-infectious disease state, or a normal state, based on the feature amount for each state calculated by the learning unit 135.

[0079] The display control unit 138 displays the result of the determination made by the determination unit 136 on the display device 152 of the image diagnosis support device 110. For example, this display can display that the subject image P is highly likely to be of an infectious disease, that the subject image P is highly likely to be of a non-infectious disease, or that the subject image P is highly likely to be in a normal state, depending on the result of the determination made by the determination unit 136. Also, when the determination unit 136 determines that the subject image P is of an infectious disease or a non-infectious disease, a message urging the subject to visit an ophthalmologist can be displayed.

[0080] FIG. 10 is a flow chart for explaining an example of the control operation of the image diagnosis support apparatus 110 of this embodiment, and corresponds to FIG. 5 of the above-mentioned embodiment.

[0081] First, in S11 corresponding to the photographing unit 162, an image of the anterior eye of the subject is photographed as a subject image P based on the operation of the operator, and in S12, the image is stored in the RAM 124 or the storage device 126. At this time, the operator may be the same as the subject, or may be a different person. Note that the subject image P does not need to be an image including only the anterior eye as described above, and may be an image including, for example, the entire face or a part of the face, as long as it includes the anterior eye.

[0082] Next, in S13 corresponding to the image cropping unit 134, a cropped image P' is cropped as an image specialized for the anterior eye segment from the subject image P stored in S12. This cropped image P' is obtained by cropping the subject image P to have a predetermined shape, such as a rectangle, so as to include the anterior eye segment, i.e., the cornea, and to minimize other parts as much as possible.

[0083] In S14 corresponding to the determination unit 136, the state of the anterior eye segment in the cut-out image P' cut out in S13 is determined by machine learning. That is, a plurality of rectangular regions R are set in the cut-out image P', and each of the plurality of rectangular regions R is compared with each of the plurality of learning images PL stored in the image database 130, and the similarity is calculated as a determination value. Then, the state of the learning image PL with the highest determination value is determined as the state of the cut-out image P'.

[0084] In S15, the display device 152 displays information about the determination result of the subject image P in S14.

[0085] According to the image diagnosis support device 110 of this embodiment, the same effects as those of the above-mentioned embodiment can be obtained, and the image diagnosis support device 110 has a camera 148 that captures a subject image P including the anterior eye of the subject, and the subject image P is captured by the camera 148, so that a network or the like is not required, and both image capture and judgment on the image can be performed by the same image diagnosis support device 110. In particular, when the image diagnosis support device 110 is implemented by a tablet PC, a smartphone, or the like, an image diagnosis support device with excellent portability can be provided. EXAMPLES

[0086] This embodiment relates to a different operation of the learning unit 35 and the determination unit 36. In the above-mentioned embodiments 1 and 2, the determination unit 36 ​​sets a plurality of rectangular regions Ri for the cut-out image P', calculates a determination value for each state for the plurality of rectangular regions Ri, and determines the state corresponding to the highest determination value as the state of the cut-out image P'. In the learning unit 35 and the determination unit 36 ​​of this embodiment, similarly to the above-mentioned embodiments, a plurality of rectangular regions Ri are set, a determination value for each state for the plurality of rectangular regions Ri is calculated, and an evaluation value or a statement based on the evaluation value is output based on the calculated determination value. Hereinafter, the operation of the learning unit 35 and the determination unit 36 ​​in this embodiment will be described based on a flowchart.

[0087] Fig. 11 is a flowchart for explaining an example of the operation of the learning unit 35 and the determination unit 36 ​​in this embodiment. This flowchart is executed, for example, in place of the operation of the determination unit in the above-mentioned embodiment, specifically, S5 in Fig. 5 or S14 in Fig. 10.

[0088] In S21, one rectangular region Ri is set from the cut-out image P'. In S22, for the rectangular region Ri set in S21, a judgment value (likelihood) which is the degree of similarity with the teacher image is calculated for each state, i.e., the three states of infectious, non-infectious, and normal. The operations in S21 and S22 are the same as those in the above-mentioned embodiment, so a description thereof will be omitted.

[0089] FIG. 12 is a diagram for explaining an example of setting a rectangular region Ri for a cut-out image P'. In FIG. 12, three rectangular regions R1, R2, and R3 are set. The number of set rectangular regions is not limited to three. The learning unit 35 calculates a judgment value for each of the rectangular regions R1, R2, and R3 in FIG. 12. The judgment value is calculated for each of the three states in this embodiment, that is, infectious, non-infectious, and normal. In this way, the judgment value (likelihood) and the corresponding judgment region for each of infectious and non-infectious are output or displayed, and information for diagnosing whether the pathology of the anterior eye is infectious or non-infectious (or normal), which has a significantly different treatment plan depending on whether it is infectious or non-infectious, is provided to an ophthalmologist.

[0090] 13 is a diagram illustrating an example of the judgment values ​​calculated for each of the three states, i.e., infectious, non-infectious, and normal, for the rectangular regions R1, R2, and R3. The calculated judgment values ​​are stored in a storage device (not shown), for example, in a table format as shown in FIG.

[0091] Returning to Fig. 11, in S23, it is determined whether or not a sufficient number of rectangular regions Ri have been set. Here, the sufficient number is a predetermined number that is set in advance to ensure appropriate accuracy of the determination unit, and this determination is made based on whether or not the number of set rectangular regions Ri has reached the predetermined number. If the determination in this step is positive, it is determined that calculation of the judgment values ​​for a sufficient number of rectangular regions Ri has been completed, and S24 is subsequently executed. If the determination in this step is negative, S21 and S22 are repeated to continue setting rectangular regions Ri and calculating judgment values ​​for the set rectangular regions Ri.

[0092] In S24, a judgment value corresponding to each state is determined. As a result of repeating S21 to S23, a judgment value for each state is calculated for each of the rectangular regions Ri. For each state, the highest value among the calculated judgment values ​​is determined as the judgment value for that state. In this manner, each judgment value of infectious, non-infectious, and normal is determined for the cut-out image P'.

[0093] An overview of the operation in S24 will be described with reference to Fig. 13. As a result of repeating S21 to S23, a judgment value for each of the infectious, non-infectious, and normal states is calculated for each of the rectangular regions Ri (i = 1, 2, ...). Then, for each state, that is, for each of the infectious, non-infectious, and normal columns in the table, the maximum value is found and the maximum value is determined as the judgment value.

[0094] Returning to FIG. 11, in S25, an evaluation value is calculated. The evaluation value is calculated using the judgment value of each state determined in S24. Specifically, for example, the evaluation value is calculated using a Softmax function. Therefore, the evaluation value is obtained with a minimum of 0 and a maximum of 1. The Softmax function is generally expressed by the following formula (1).

number

number

[0095] In this way, the evaluation of each of the infectious, non-infectious and normal states for all of the infectious, non-infectious and normal states can be calculated as each evaluation value.

[0096] Note that multiple evaluation values ​​are appropriately used depending on the evaluation method. In the above formulas (1) and (2), the evaluation values ​​are for evaluating each of the infectious, non-infectious, and normal states for all of the infectious, non-infectious, and normal states, but by using the following formula (3) instead of formula (1) and the following formula (4) instead of formula (2), it is possible to calculate evaluation values ​​for evaluating each of the infectious and non-infectious states for the two states of infectious and non-infectious.

number

number

number

number

[0097] Returning to FIG. 11, in S26, the display contents are generated based on the judgment value determined in S24 and the evaluation value calculated in S25. For example, when the evaluation values ​​are calculated for each state, the display contents can display the state corresponding to the highest evaluation value among them. FIG. 14(a) is an example of this display content displayed on a display device. The highest evaluation value and the state corresponding to it may be displayed as the display contents. FIG. 14(b) is an example of this display content displayed. Alternatively, the above-mentioned evaluation value may be displayed for each state. FIG. 15 is an example of the display content displayed when the evaluation value is calculated using the above-mentioned formula (1), and FIG. 16 is an example of the display content displayed when the evaluation value is calculated using the above-mentioned formula (2). When the evaluation value is calculated using the above-mentioned formula (5) or formula (6), the definition of the evaluation value and the value of the evaluation value may be displayed as the display contents. FIG. 17 is an example of this display content displayed.

[0098] As shown in Figures 12 to 16, the judgment values ​​(likelihood) and evaluation values ​​for each of infectious and non-infectious are output and displayed, providing ophthalmologists with information to diagnose whether anterior eye pathologies are infectious or non-infectious (or normal), which has a significantly different treatment plan depending on whether they are infectious or non-infectious.

[0099] Furthermore, in Figs. 12 to 16, the maximum value for each of the infectious, non-infectious, and normal states is determined and displayed as the judgment value for each state from among a plurality of different rectangular regions. That is, the judgment values ​​for each of the infectious, non-infectious, and normal states may be calculated from different judgment regions (rectangular regions). As a result, information on the likelihood or evaluation value of each of the infectious, non-infectious, and normal states as the state of the anterior eye is output as unique information of the judgment image that does not depend on the selection of the rectangular region, and is provided to the ophthalmologist. For example, in the examples of Figs. 12, 13, and 15, not only is it known that the judgment value for infectious is high at 0.8, but also that the judgment values ​​for non-infectious and normal are low at 0.3 and 0.2, respectively, so that the ophthalmologist can use this output information to make a judgment not only to select a drug that is effective against infectious diseases, but also to avoid drugs that are undesirable in terms of side effects when applied to non-infectious cases.

[0100] In addition, in S26, one or more sentences selected from a plurality of sentences prepared in advance may be displayed. The plurality of sentences may be, for example, created in advance and stored in a storage means (not shown). In addition, a condition using the evaluation value may be set in advance, and for example, when the evaluation value for evaluating the two states of infectious and non-infectious for all the states of infectious, non-infectious, and normal exceeds a predetermined threshold, a sentence such as "There is a high possibility of abnormality, we recommend receiving the notification" may be selected and displayed. This threshold corresponds to the condition. In addition, the threshold is set, for example, by evaluating the relationship between the evaluation value by the image diagnosis support system 10 of the present invention and cases diagnosed as abnormal by an actual doctor, and more specifically, a value smaller than the evaluation value corresponding to those cases is set so that cases diagnosed as abnormal by a doctor are included. FIG. 18 is an example of the display content. In addition, when the evaluation value for evaluating the two states of infectious and non-infectious for all the states of infectious, non-infectious, and normal falls below a predetermined threshold that can be determined as normal, a sentence such as "There is no problem" may be selected. In addition, appropriate sentences are selected according to the evaluation value. Furthermore, the selection of sentences may not only be performed based on a threshold value set for one evaluation value, but also may be performed by selecting corresponding sentences for each area set in a multi-dimensional space or plane defined by two or more evaluation values, or by selecting corresponding sentences for each area set in a multi-dimensional space or plane defined by an evaluation value and a judgment value.

[0101] According to the image diagnosis support devices 10 and 110 of the above-mentioned embodiment, the judgment unit 36 ​​(S24, S25) calculates an evaluation value for evaluating the judgment image P' based on the feature amounts of the judgment image P' calculated for each of the infectious, non-infectious, and normal states, so that it is possible to calculate an evaluation value for evaluating the judgment image P' that is different from the feature amounts. In particular, since one evaluation value is calculated based on three feature amounts, which is a plurality of values, the evaluation can be easily performed.

[0102] According to the image diagnosis support devices 10, 110 of the above-mentioned embodiments, the judgment unit 36 ​​(S24, S25) calculates an evaluation value by applying the feature amounts of the judgment image calculated for each of the states to a Softmax function, so that the evaluation value is calculated as a value between 0 and 1, making it easy to perform evaluation using the evaluation value.

[0103] According to the image diagnosis support device 10, 110 of the above-mentioned embodiment, the judgment unit 36 ​​(S26) selects and outputs at least one of a plurality of sentences stored in advance according to the evaluation value, and since a sentence according to the evaluation value is output, the evaluation can be obtained intuitively.

[0104] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the present invention can be applied to other embodiments.

[0105] For example, in the above-described embodiments, the judgment units 36 and 136 used YOLOv3 as a machine learning method, but this is not limited to this, and other methods may be used as long as they can calculate the likelihood between the cut-out image P' and the learning image.

[0106] In the image database 30, 130 in the above embodiment, information on the state of each learning image PL, i.e., information on whether the image is infectious, non-infectious, or normal, is stored as a label for each learning image PL, but the present invention is not limited to this. For example, when the learning image PL is infectious or non-infectious, in addition to information on whether the learning image PL is infectious or non-infectious, a more specific disease name can be stored. In this case, the determination unit 36, 136 may determine a disease name with a high probability instead of or in addition to the determination in the above embodiment, i.e., whether the cut-out image P' used for the determination is infectious, non-infectious, or normal. Specifically, for example, the determination unit 36, 136 may read information on the disease name stored in the image database 30, 130 for the learning image PL that is determined to have the highest similarity to the cut-out image P', and determine that the cut-out image P' is highly likely to be the disease. In this way, the judgment units 36, 136 can judge the name of the disease with the highest probability based on the extracted image P', instead of or in addition to judging whether the image is infectious, non-infectious, or normal.

[0107] Here, examples of infectious diseases include bacterial keratitis, fungal keratitis, herpes virus keratitis, and amebic keratitis, while examples of non-infectious diseases include peripheral corneal ulcer, catarrhal corneal ulcer, and phlyctenular keratitis.

[0108] Bacterial keratitis can be caused by many types of bacteria, such as Staphylococcus aureus and Pseudomonas aeruginosa. Fungal keratitis can also be caused by a number of types of bacteria, such as Candida and Aspergillus. If the learning image PL is bacterial keratitis or fungal keratitis, information on the causative bacteria causing it can also be stored in the image database 30, 130. If the determination unit 36, 136 determines that the learning image PL having the highest similarity to the cut-out image P' is bacterial keratitis or fungal keratitis, the determination unit 36, 136 may additionally read information on the causative bacteria of the learning image PL having the highest similarity and determine that the cut-out image P' is highly likely to be caused by the causative bacteria. This determination is not limited to bacterial keratitis or fungal keratitis and the causative bacteria that cause them, but is effective when a correlation is found between the causative bacteria and image features of the case.

[0109] In the above embodiment, the judgment unit 36, 136 judges whether the cut-out image P' is infectious, non-infectious, or normal, but may also have a function of proposing a treatment method. Specifically, for example, when the judgment unit 36, 136 judges that the cut-out image P' is infectious, the affected area corresponding to the cut-out image P' is in a state where there is a high possibility of corneal inflammation caused by microorganisms, and representative examples of the microorganisms are bacteria (so-called bacteria), fungi (so-called mold), viruses, and amoebas. Therefore, when the judgment unit 36, 136 judges that the cut-out image P' is infectious, the judgment unit 36, 136 can output a proposal (suggestion) of a treatment method such as administering an antibiotic, an antifungal drug, or the like to eliminate the microorganism. On the other hand, when the cut-out image P' is judged to be non-infectious, the affected area corresponding to the cut-out image P' is in a state where there is a high possibility that the patient's own immune system has gone wild and caused corneal inflammation (for example, an allergic reaction such as hay fever), and the immune system is going wild by targeting a harmless substance. For example, rheumatism is one type of this, and is known to cause "peripheral corneal ulcers" in the cornea. Therefore, when the cut-out image P' is judged to be non-infectious, it can be output to suggest (suggest) a treatment method of administering a drug that suppresses the immune system (immunosuppressant).

[0110] In the above embodiment, the judgment unit 36, 136 judges whether the corneal infiltration, which is inflammation in the anterior eye image, is infectious, non-infectious, or normal, but the judgment is not limited to such diseases. That is, if the disease is similar in appearance but the appearance differs depending on the cause or condition, the image diagnosis support device of the present invention can judge it in the same way. Specifically, it is also possible to make a judgment when the learning image PL and the subject image P are cases of corneal scars, which are lesions of inflammation, or corneal degeneration in which abnormal substances are deposited, instead of or in addition to corneal infiltration.

[0111] In the above-mentioned embodiment, the determination unit 36, 136 determines whether the cut-out image P' is infectious, non-infectious, or normal, but is not limited thereto. For example, it is also possible to determine whether the cut-out image P' is infectious corneal infiltration, non-infectious corneal infiltration, a case other than corneal infiltration, or normal. In such a case, the image database 30, 130 stores in advance the learning images PL of infectious corneal infiltration, non-infectious corneal infiltration, a case other than corneal infiltration, and normal, together with information (labels) indicating the state, in the number necessary for machine learning, and machine learning is performed.

[0112] In the above-mentioned embodiment, the image cropping unit 34, 134 generates the cropped image P' such that the cornea occupies the majority of the image, but the present invention is not limited to this embodiment. Specifically, the image may include the entire cornea and a part of the conjunctiva and eyelids. The subject image P is preferably a color image captured under visible light. In this way, the cropped image P' cropped from the captured image can be used for judgment without requiring special imaging equipment or imaging environment.

[0113] In the above embodiment, the judgment unit 36, 136 determines the state corresponding to the highest judgment value for the cut-out image P' as the judgment result for the cut-out image P', but is not limited to such an embodiment. For example, the highest judgment value and the corresponding state may be combined to obtain the judgment result. In this way, the certainty of the judgment result can be obtained. Alternatively, the judgment result may include the highest corresponding judgment value for at least one of the infectious, non-infectious, and normal states. In this way, it is possible to grasp the possibility that judgments may differ between multiple states.

[0114] In the above embodiment, the calculation of the feature amount in the judgment unit 36, 136 is performed for the learning image PL stored in advance in the image database 30, 130, but the present invention is not limited to this embodiment. For example, when the cut-out image P' is once judged by the judgment unit 36, 136 and its condition is confirmed by a medical examination or test, the cut-out image P' that was the subject of the judgment and the confirmed information on the condition of the cut-out image P' may be newly added to the image database 30, 130, and the calculation of the feature amount may be performed again. In this way, the image database 30, 130 can be updated as appropriate. Alternatively, information on the appropriateness of the judgment made by the judgment unit 36, 136 for the cut-out image P' that was the subject of the judgment may be given to the image database 30, 130, and the calculation of the feature amount may be performed again. In this way, learning by so-called reinforcement learning can be performed.

[0115] In the above-mentioned embodiment, the image processing determination device 12 and the image diagnosis support device 110 have the learning unit 35, 135 corresponding to the learning step of calculating a feature amount representing the feature from each of the learning images PL in the image database 30, 130 in advance, and the determination unit 36, 136 corresponding to the determination step including calculating the similarity between the cut-out image P' and the learned feature amount as a determination value (likelihood), but are not limited to such an embodiment. That is, when the learned feature amount is calculated and given in advance, the given learned feature amount is stored in the storage device 26, 126, etc., and only the operation corresponding to the determination step is performed using the stored learned feature amount. Therefore, the image processing determination device 12 and the image diagnosis support device 110 may have the determination unit 36, 136 without the learning unit 35, 135. In such a case, the calculation of the learned feature amount may be performed in a learning device that is a device separate from the image processing determination device 12 and the image diagnosis support device 110. In this case, the learning device is connected to the image processing determination device 12 equipped with the determination units 36, 136, and the image diagnosis support device 110 so as to be able to transmit and receive information including the learned features. Such a connection may be made via an information communication line such as a LAN or the Internet, or may be a connection form such as a bus connection. In addition, it does not matter whether the connection is wired or wireless. In this way, the function corresponding to the learning process and the function corresponding to the determination process can be processed in a distributed manner by different hardware (computers).

[0116] In the above embodiment, the image database 30, 130 storing the learning image PL and information on the state of the image is provided in the image processing determination device 12 and the image diagnosis support device 110, but the present invention is not limited to this embodiment. That is, the storage device 26, 126 including the image database 30, 130 may be provided as a storage device separate from the image processing determination device 12 and the image diagnosis support device 110. In this case, the separate storage device is connected to the image processing determination device 12 and the image diagnosis support device 110 including the learning unit 35, 135, or the learning device as the separate device, so that information stored in the image database 30, 130 can be transmitted and received. Such a connection may be made by an information communication line such as a LAN or the Internet, or may be a connection form such as a bus connection. In addition, it does not matter whether the connection is wired or wireless.

[0117] In other words, when the separate storage devices are provided, the image processing determination device 12 and image diagnosis support device 110 do not need to have the image databases 30 and 130, and the configurations can be simplified. Also, when a learning device is provided as a separate device, there is no need to have the learning units 35 and 135, so it is sufficient to have a calculation device (such as a CPU) that is appropriate for the load, and it is not necessarily necessary to have a highly functional one.

[0118] In the above embodiment, the determination unit 36, 136 performs a determination each time one subject image P is captured, but the present invention is not limited to this. That is, after multiple subject images P are read out, the determination may be performed on the multiple subject images P collectively.

[0119] In the first embodiment described above, the image processing determination device 12 and the terminal device 14 are connected by wireless LAN, but the present invention is not limited to this method and may be connected by other wireless methods, or may be connected by wired connection using a wired LAN cable, a serial cable, etc. In addition, the subject image P is directly transmitted and received from the terminal device 14 to the image processing determination device 12, but the present invention is not limited to this method and may be transmitted and received indirectly, for example, the terminal device 14 uploads the subject image P to a third device in the same network, and the image processing determination device 12 downloads it from the third device.

[0120] In the above embodiment, the Softmax function is used to calculate the evaluation value, but the present invention is not limited to this. That is, a function may be used that calculates the evaluation value between 0 and 1, in other words, that normalizes the evaluation value, or a function may be used that calculates an evaluation value that takes a value in a predetermined range, not limited to 0 to 1. Also, a value calculated between 0 and 1 can be converted to a value in any range, for example, a range of 0 to 100, and used as the evaluation value. [Explanation of symbols]

[0121] 10: Image diagnosis support system 12: Image processing judgment device 14: Terminal device 30: Image database (image storage section) 34,134: Image extraction section 36,136: Judgment section 48,148: Camera 50:Communication equipment 110: Image diagnosis support device P: Subject image PL: Learning image

Claims

1. A learning unit that calculates, by machine learning, feature amounts for each of infectious and non-infectious states for determining the state of a determination image of the anterior eye segment based on a learning image of the anterior eye segment and information on the state in the learning image; An image diagnosis support device characterized by:

2. a determination unit that determines a state of a determination image of the anterior eye using a feature amount calculated by machine learning based on a learning image of the anterior eye and information on the state of the learning image, the determination unit calculates a likelihood that the state of the determination image is an infectious state and a likelihood that the state is a non-infectious state, An image diagnosis support device characterized by:

3. an image storage unit that stores learning images of the anterior eye and information on the state of the anterior eye in the learning images; a learning unit that calculates a feature amount for determining a state of the determination image of the anterior eye segment by machine learning based on the image storage unit; a determination unit that determines a state of the determination image of the anterior eye segment by using the feature amount, the determination unit calculates a likelihood that the state of the determination image is an infectious state and a likelihood that the state is a non-infectious state, An image diagnosis support device characterized by:

4. the determination unit further calculates a likelihood that the state of the determination image is a normal state; 4. The image diagnosis support device according to claim 2 or 3.

5. the determination unit determines whether the state of the determination image is infectious, non-infectious, or normal; 5. The image diagnosis support device according to claim 2, wherein:

6. the determination unit calculates an evaluation value for evaluating the image for determination based on the feature amounts of the image for determination calculated for each of the infectious, non-infectious, and normal states; 6. The image diagnosis support device according to claim 2, wherein

7. the determination unit calculates and outputs a likelihood that the state of the determination image is infectious, non-infectious, or normal, or an evaluation value for evaluating the determination image; 6. The image diagnosis support device according to claim 2, wherein

8. the determination unit calculates the evaluation value by applying the feature amounts of the determination image calculated for each of the infectious, non-infectious, and normal states to a Softmax function; 7. The image diagnosis support device according to claim 6,

9. the determination unit selects and outputs at least one of a plurality of sentences stored in advance in accordance with the evaluation value; 9. The image diagnosis support device according to claim 6 or 8,

10. The determination unit estimates a disease name; 10. The image diagnosis support device according to claim 2, wherein

11. The learning unit re-executes the machine learning based on a determined state regarding a determination made by the determination unit in the past; 11. The image diagnosis support device according to claim 3,

12. an image cutout unit that detects a position of a part to be determined from the image for determination and cuts out an image so as to include the position of the part to be determined; the determination unit determines a state of the anterior eye segment from the cut-out image cut out by the image cut-out unit; 12. The image diagnosis support device according to claim 2, wherein

13. Having a camera for capturing an image including the anterior segment of the subject's eye; 13. The image diagnosis support device according to claim 2, wherein

14. An image diagnosis support device according to any one of claims 2 to 12, An image diagnosis support system including the image diagnosis support device and a terminal device capable of information communication, The terminal device a camera for capturing an image including an anterior segment of the subject's eye; a transmission unit that transmits the image captured by the camera to the image diagnosis support device, the image diagnosis support device is configured to determine a state of the anterior eye of the subject using an image transmitted from the terminal device; An image diagnosis support system characterized by the above.

15. A learning process for performing machine learning based on a learning image of the anterior eye and information on the state in the learning image; A determination step of determining a state of the determination image using a learning result from the learning step, The determination step includes calculating a likelihood that the state of the determination image is an infectious state and a likelihood that the state is a non-infectious state, An image diagnosis support method comprising:

16. A learning process for performing machine learning based on a learning image of the anterior eye and information on the state in the learning image; a likelihood calculation step of calculating the likelihood that the state of the determination image is an infectious state and the likelihood that the state of the determination image is a non-infectious state using a learning result from the learning step; and an output step of outputting the evaluation value of the determination image calculated based on the likelihood or the likelihood for each of the infectious and non-infectious states. An image diagnosis support method comprising:

17. The image diagnosis support method according to claim 16, the likelihood calculation step further includes a step of calculating a likelihood that the state of the determination image is a normal state, the output step includes a step of further outputting the evaluation value of the determination image calculated based on the likelihood or the likelihood for a normal state; An image diagnosis support method comprising:

18. The image diagnosis support method according to claim 16, the likelihood calculation step includes a step of setting a plurality of judgment regions in the judgment image, and calculating a judgment region likelihood of an infectious state and a judgment region likelihood of a non-infectious state for each of the judgment regions, the output step includes a step of outputting an evaluation value of the determination image calculated based on the determination region likelihoods of the determination regions that differ between an infectious state and a non-infectious state, or the determination region likelihoods of the determination regions that differ; An image diagnosis support method comprising:

19. the learning unit further calculates, by the machine learning, a feature amount regarding a normal state for determining a state of the determination image of the anterior eye segment, based on the learning image of the anterior eye segment and information regarding a state in the learning image; 2. The image diagnosis support device according to claim 1 .

Citation Information

Patent Citations

  • Image processing device and image processing program

    JP2018121886A

  • Medical image processing apparatus, medical image processing method, and program

    JP2019216848A

  • Ophthalmic measurement device

    WO2018003906A1

  • Image processing device and program

    WO2019073962A1