Ophthalmic system

Abstract

To provide an ophthalmic system capable of easily determining the possibility of an eye disease when inputting inspection image data.SOLUTION: The ophthalmic system S1 includes a reference image data-classifying unit 20, a reference image transformation data-generating unit 30, an examination image transformation data-generating unit 40, and a data-comparing unit 50. The reference image transformation dataset generator 30 generates a reference image transformation dataset from each reference image dataset using the first encoder 31, which is a trained AI model trained to extract a feature from an image dataset of an eye and which converts the extracted feature into an embedding dataset through a plurality of computation layers. The inspection image conversion data generation unit 40 generates inspection image conversion data from one piece of newly acquired inspection image data using the second encoder 41 which is the same as the first encoder 31. The data comparison unit 50 compares the inspection image conversion data with the reference image conversion data, and calculates the similarity between the two image conversion data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to ophthalmic systems. [Background technology]

[0002] Non-Patent Document 1 describes a knowledge-based universal model of the retina developed from a collection of 37 public datasets. This public dataset contains 284,660 fundus images and 96 different categories (disease types). The basic model consists of an image encoder and a text encoder, which are trained contrastively on paired image and text descriptors.

[0003] Patent Document 1 describes a medical information processing system that uses AI (artificial intelligence) to detect eye diseases (such as glaucoma and age-related macular degeneration) from images obtained during ophthalmic examinations. The system accepts a patient's fundus retinal tomographic image (hereinafter referred to as "fundus OCT image") and eye information (diagnosis, visual field test data, and intraocular pressure value), and uses an artificial intelligence engine to classify the fundus OCT image into two or more categories, including a glaucoma category and a non-glaucoma category. A category is selected based on the eye information, and a determination is made as to whether the classification result based on the fundus OCT image matches the selection result based on the eye information. If there is no match, the fundus OCT image is classified into a unique category, and machine learning is performed based on the fundus OCT images classified into the unique category. Note that OCT is an abbreviation for "Optical Coherence Tomography." [Prior art documents] [Patent documents]

[0004] [Non-Patent Document 1] “A Foundation LAnguage-Image model of the Retina (FLAIR): Encoding expert knowledge in text supervision” (arXiv:2308.07898v1[cs.CV] 15 Aug 2023)

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-116134 Summary of the Invention [Problem to be solved by the invention]

[0006] Currently, ophthalmologists primarily visually assess the type and severity of eye disease based on fundus images and fundus OCT images displayed or captured by ophthalmic equipment. However, the accuracy of determining the type and severity of eye disease varies depending on the experience of the ophthalmologist making the assessment. Meanwhile, there has been significant technological development in discriminative AI, such as image recognition, and generative AI, which generates text, images, and data. Against this technological background, there is a demand for an ophthalmological system that can easily determine the possibility of eye disease simply by inputting examination image data.

[0007] In contrast, the technology described in Non-Patent Document 1 is a technology that configures a basic model using an image encoder and a text encoder. The technology described in Patent Document 1 is a technology for classifying fundus OCT images using an artificial intelligence engine. Therefore, these prior art technologies cannot meet the demand for easily determining the possibility of eye disease by simply inputting test image data.

[0008] The present disclosure has been made in light of the above-mentioned problems, and aims to provide an ophthalmologic system that can easily determine the possibility of eye disease when test image data is input. [Means for solving the problem]

[0009] To solve the above-mentioned problems, an ophthalmologic system disclosed herein determines the possibility of eye disease based on eye image data. The ophthalmologic system includes a reference image data classification unit, a reference image transformation data generation unit, a test image transformation data generation unit, and a data comparison unit. The reference image data classification unit groups a large number of image data stored in a database by type of eye disease to generate reference image data for each group. The reference image transformation data generation unit is a learned AI model trained to extract features from eye image data, and generates reference image transformation data from each of the reference image data using a first encoder that converts the extracted features through multiple computational layers into embedded data. The test image transformation data generation unit uses a second encoder identical to the first encoder to generate test image transformation data from a newly acquired piece of test image data. The data comparison unit compares the test image transformation data with the reference image transformation data and calculates the similarity between the two image transformation data. [Effects of the Invention]

[0010] The ophthalmology system according to the present disclosure can easily determine the possibility of eye disease when examination image data is input. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a system configuration diagram showing an ophthalmologic system according to a first embodiment. [Figure 2] FIG. 10 is an explanatory diagram showing generation of embedded data by an encoder. [Figure 3] FIG. 10 is an explanatory diagram showing a data comparison between a reference image vector value and an inspection image vector value. [Figure 4] FIG. 10 is an explanatory diagram showing an example of a display screen that displays information on the possibility of eye disease. [Figure 5] FIG. 10 is an explanatory diagram showing another example of a display screen that displays the progression of the degree of an eye disease. [Figure 6] 10A and 10B are explanatory diagrams showing experimental results of determination sensitivity using the ophthalmologic system of the first embodiment. [Figure 7] FIG. 10 is a system configuration diagram showing an ophthalmologic system according to a second embodiment. [Figure 8] FIG. 10 is a detailed explanatory diagram showing details of the optimization process of the reference image transformation data. [Figure 9] 10 is an explanatory diagram showing experimental results of determination sensitivity using the ophthalmologic system of the second embodiment. FIG. [Figure 10] FIG. 10 is an explanatory diagram showing the state of stage movement due to image data optimization. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment for carrying out the ophthalmologic system of the present disclosure will be described below based on Embodiment 1 and Embodiment 2 shown in the drawings.

[0013] The ophthalmologic systems of the first and second embodiments are applied to predict and determine eye diseases and the stages of eye diseases that can be detected by inputting detected image data such as fundus images and fundus OCT images acquired from an ophthalmologic device.

[0014] [System configuration (Figure 1)] The ophthalmic system S1 of the first embodiment determines the possibility of an eye disease based on image data acquired from an ophthalmic device. The ophthalmic device referred to here is not limited to the ophthalmic device 10 shown in FIG. 1 that acquires examination image data, but also includes various ophthalmic devices in various locations, such as a fundus camera capable of acquiring fundus images, an OCT device capable of acquiring fundus OCT images, and a slit lamp microscope capable of acquiring anterior segment images. Furthermore, eye diseases that can be determined include diabetic retinopathy (DR), branch retinal vein occlusion (BRE), age-related macular degeneration (AMD), drusen (precursor to age-related macular degeneration), tessellated fundus, and optic disc cupping (see FIG. 4). Other eye diseases include glaucoma, retinal detachment, and branch retinal vein occlusion, and these may also be included.

[0015] As shown in Figure 1, the ophthalmologic system S1 has a reference image data classification unit 20, a reference image conversion data generation unit 30, an examination image conversion data generation unit 40, a data comparison unit 50, a judgment information processing unit 60, and a screen display unit 70.

[0016] The reference image data classification unit 20 groups a large number of image data stored in the database by type of eye disease to generate reference image data for each group, such as RI(A), RI(B), etc. For example, the reference image data for each group may be such that reference image data RI(A) in group A is assigned to diabetic retinopathy, and reference image data RI(B) in group B is assigned to glaucoma. Furthermore, the reference image data for each group may be such that reference image data RI(A) in group A is assigned to diabetic retinopathy, and reference image data RI(B) in group B is assigned to disease-free (healthy). In this way, the groups of reference image data may include a disease-free, healthy group as a disease type.

[0017] The reference image data classification unit 20 subdivides the reference image data RI(A), RI(B), etc. by group into a plurality of stages for each type of disease, ranging from mild to severe eye disease, including a no-eye disease stage. For example, if the group of reference image data RI(A) is a DR group due to diabetic retinopathy, the group is subdivided into five stages ranging from mild to severe, including healthy, with no eye disease, stage 1, stage 2, stage 3, and stage 4 (see FIG. 6).

[0018] The reference image transformation data generation unit 30 is a trained AI model that has been trained to extract features from eye image data, and uses a first encoder 31 that converts the features into embedded data through multiple nonlinear calculation layers. The reference image transformation data generation unit 30 inputs each of the reference image data RI(A), RI(B), etc. to the first encoder 31, which then generates multiple reference image vector values ​​R1, R2, R3, ..., Rn as multiple reference image transformation data RN(A), RN(B), etc., using the first encoder 31. The reference image vector values ​​R1, R2, R3, ..., Rn are vector values ​​representing feature vectors generated by compressing each of the reference image data RI(A), RI(B), etc. input to the first encoder 31. The first encoder 31 then outputs the multiple reference image vector values ​​R1, R2, R3, ..., Rn to the data comparison unit 50. Note that the vector values ​​refer to the component values ​​of a vector.

[0019] The test image conversion data generation unit 40 is a trained AI model that has been trained to extract features from eye image data. It uses a second encoder 41, which is the same as the first encoder 31 and converts image data into embedded data through multiple nonlinear computation layers. The test image conversion data generation unit 40 inputs test image data PI, for example, newly acquired from the ophthalmologic apparatus 10, to the second encoder 41, and generates multiple test image vector values ​​P1, P2, P3, ..., Pn as test image conversion data PN using the second encoder 41. The second encoder 41 then outputs the multiple test image vector values ​​P1, P2, P3, ..., Pn to the data comparison unit 50. Here, the test image data PI may be a fundus image or a fundus OCT image obtained by an examination. The test image vector values ​​P1, P2, P3, ..., Pn are vector values ​​representing feature vectors generated by compressing the test image data PI input to the second encoder 41. The multiple reference image vector values ​​R1, R2, R3, ..., Rn and the multiple inspection image vector values ​​P1, P2, P3, ..., Pn are all embedded data, and the details of the embedded data will be described later with reference to Figure 2.

[0020] The data comparison unit 50 compares one piece of inspection image transformation data PN from the inspection image transformation data generation unit 40 with multiple pieces of reference image transformation data RN(A), RN(B), etc. from the reference image transformation data generation unit 30, and calculates the similarity between the two pieces of image transformation data. Here, one piece of inspection image transformation data PN is data based on multiple inspection image vector values ​​P1, P2, P3, ..., Pn. Each of the multiple reference image transformation data RN(A), RN(B), etc. is data based on multiple reference image vector values ​​R1, R2, R3, ..., Rn. The similarity between the two pieces of image transformation data is calculated by a scalar product of vector multiplication between paired values ​​of the reference image vector values ​​R1, R2, R3, ..., Rn and the inspection image vector values ​​P1, P2, P3, ..., Pn, and a similarity calculation value SC is obtained as the calculation result. Details of the data comparison will be described later with reference to FIG. 3.

[0021] The determination information processing unit 60 receives the calculated similarity values ​​SC from the data comparison unit 50 and processes the determination information required to determine the possibility of an eye disease. The determination information processing by the determination information processing unit 60 includes a calculation process for calculating an average similarity value AS by averaging multiple calculated similarity values ​​SC within the same group, and a selection process for selecting the maximum similarity value MS from the multiple calculated similarity values ​​SC within the same group. In addition to the calculation process and selection process, the determination information processing by the determination information processing unit 60 also includes a process for acquiring test image data PI, a process for acquiring reference image data RI(A), RI(B), etc. that has a high calculated similarity value SC, and a process for acquiring past similarity data from the same patient.

[0022] The screen display unit 70 has a display screen 71 that displays the determination information output from the determination information processing unit 60. The screen display unit 70 uses, for example, a display unit of a personal computer connected to the ophthalmologic apparatus 10. Details of the content displayed on the display screen 71 will be described later with reference to FIGS. 4 and 5.

[0023] The ophthalmologic system S1 may be configured as a system that can acquire information for determining the possibility of eye disease by inputting test image data indicating whether or not an eye disease is present, through operation of a terminal device capable of exchanging information with the ophthalmologic apparatus 10. Specifically, the ophthalmologic system S1 may be configured using edge computing or cloud computing. Furthermore, the ophthalmologic system S1 may be configured using a combination of edge computing and cloud computing. In the case of a system using both edge computing and cloud computing, the combined system reduces the storage capacity and processing load of the terminal devices, such as a server or a personal computer. In a system using both, for example, the first encoder 31 and the second encoder 41, which handle massive amounts of big data and require training, such as re-learning, over time, can be shared by cloud computing.

[0024] [Embedded data generation details (Figure 2)] Figure 2 shows the process of generating multiple reference image transformation data RN(A), RN(B), etc. by the first encoder 31, and the process of generating inspection image transformation data PN by the second encoder 41, and details of embedded data generation will be explained based on Figure 2.

[0025] The reference image transformation data generation unit 30 generates a plurality of reference image transformation data RN(A), RN(B), etc. Each of the reference image transformation data RN(A), RN(B), etc. is a plurality of reference image vector values ​​R1, R2, R3, ..., Rn representing feature vectors obtained by compressing the reference image data RI(A), RI(B), etc. input to the first encoder 31 (see FIG. 3).

[0026] The inspection image conversion data generator 40 generates one inspection image conversion data PN. The inspection image conversion data PN is a plurality of inspection image vector values ​​P1, P2, P3, ..., Pn representing feature vectors obtained by compressing the inspection image data PI input to the second encoder 41 (see FIG. 3).

[0027] First, one reference image data RI or one inspection image data PI is image data made up of, for example, 512 (number of vertical pixels) x 512 (number of horizontal pixels) x 3 (three primary colors of RGB) pixels.

[0028] The first encoder 31 and the second encoder 41 are one type of neural network mechanism, called an autoencoder, and are also known as data generation models. The first encoder 31 and the second encoder 41 are configured with an input layer, an intermediate layer, and an output layer, and extract features from the input image data through the multi-layered intermediate layer. In other words, the first encoder 31 and the second encoder 41 extract features, which are the most useful information from the image data, and perform a compression process that discards the remaining image data.

[0029] The first encoder 31 and the second encoder 41 have multiple nonlinear computation layers in the middle layer, extract features from input image data, and convert the extracted features into embedded data through the multiple nonlinear computation layers. The embedded data output from the first encoder 31 and the second encoder 41 are multiple image vector values ​​obtained by converting feature vectors obtained by compressing image data into vector values, as shown on the right side of Figure 2. The multiple image vector values ​​are reference image vector values ​​R1, R2, R3, ..., Rn and test image vector values ​​P1, P2, P3, ..., Pn. These vector values ​​are expressed as, for example, 512 vector values ​​through compression. Here, "embedded" refers to a numerical representation of real-world objects (e.g., images, text) that machine learning systems (ML systems) and artificial intelligence systems (AI systems) use to understand complex knowledge domains like humans.

[0030] [Details of data comparison (Figure 3)] FIG. 3 shows a data comparison between reference image vector values ​​R1, R2, R3, . . . , Rn and inspection image vector values ​​P1, P2, P3, .

[0031] The reference image transformation data generation unit 30 and the inspection image transformation data generation unit 40 generate the same number (e.g., n=512) of reference image vector values ​​R1, R2, R3, ..., Rn and inspection image vector values ​​P1, P2, P3, ..., Pn. The data comparison unit 50 then calculates the similarity by calculating the scalar product of vector multiplication between paired output values ​​of the reference image vector values ​​R1, R2, R3, ..., Rn and the inspection image vector values ​​P1, P2, P3, ..., Pn. The paired output values ​​refer to pairs such as vector value R1 and vector value P1, vector value R2 and vector value P2, etc.

[0032] Therefore, the calculation formula for the scalar product (inner product) of vector multiplication in the data comparison unit 50 is expressed by the following formula. ΣRi·Pi=R1·P1+R2·P2+…+Rn·Pn=|Ri||Pi|cosθ Here, i=1 to n. The reason for the above formula is that the scalar product (inner product) of vectors a and b is expressed as a = (a1, a2) and b = (b1, b2) when the plane vectors a and b are written in components. (a, b)=a1·b1+a2·b2=|a||b|cosθ This is because it is expressed as

[0033] The magnitude of the scalar product of vector multiplication calculated by the data comparator 50 represents the degree of similarity between the inspection image data PI and the reference image data RI(A), RI(B), etc. That is, for example, when the paired vector values ​​P1 and R1 match and θ=0, cosθ=1 when θ=π / 2, cosθ=0, and cosθ=−1 when θ=π. Therefore, if the degree of similarity between the inspection image data PI and the reference image data RI(A), RI(B), etc. is high, cosθ will be close to 1, resulting in a large value for the scalar product. If the degree of similarity is low, the value of the scalar product will be a small positive number including zero, or a negative number.

[0034] The calculation of the scalar product of vector multiplication is repeated between the inspection image vector values ​​P1, P2, P3, ..., Pn and all of the multiple reference image vector values ​​R1, R2, R3, ..., Rn. In this way, the similarity between the inspection image data PI and the reference image data RI(A), RI(B), etc. is measured between all of the reference image data RI(A), RI(B), etc. in the multiple groups.

[0035] Here, the similarity calculation value calculated by the data comparison unit 50 is converted into a similarity calculation value (-1.0 to +1.0) using, for example, an activation function, which makes it easy to calculate the judgment information and understand the similarity. Note that the similarity calculation value calculated by the data comparison unit 50 may be output as is to the judgment information processing unit 60, and the value may be converted using an activation function at the input unit of the judgment information processing unit 60.

[0036] [Detailed display of eye disease diagnosis information (Figure 4, Figure 5)] Figure 4 is an example of a display screen that displays information for determining the possibility of eye disease, and Figure 5 is an example of a display screen that displays the progression of the severity of eye disease. Detailed display of eye disease determination information will be explained based on Figures 4 and 5.

[0037] The screen display unit 70 receives necessary information from the determination information processing unit 60 and displays information useful for determining eye diseases on the display screen 71. As shown in Fig. 4, an example of the screen display unit 70 has a test image display unit 71a, a similar reference image display unit 71b, and a similarity calculation value display unit 71c on the display screen 71.

[0038] The examination image display unit 71a displays an eye image based on examination image data PI acquired by the ophthalmologic apparatus 10 and which is to be used to determine the possibility of an eye disease.

[0039] The similar reference image display unit 71b displays an eye image based on reference image data RI having a high similarity to the test image data PI, among the reference image data RI(A), RI(B), etc. Note that the similar reference image display unit 71b allows the user to select, by tapping the switching operation unit 71d, the reference image with the highest similarity, the reference image with the second highest similarity, or the reference image with the third highest similarity as the reference image data RI having a high similarity.

[0040] The similarity calculation value display section 71c displays the average similarity value AS and the maximum similarity value MS for each group, such as the DR group, in the form of a comparison table. For example, in the case of the DR group, the average similarity value for similar groups is 0.662, and the maximum similarity value is 0.973.

[0041] 5, another example of the screen display unit 70 has a similarity transition graph display section 71e that displays a similarity transition graph in which past and current similarities are plotted along a time axis on the display screen 71 when past similarity data for the same patient exists. Here, the past and current similarities may be represented by an average similarity value AS calculated as the average of multiple similarity calculation values ​​SC within the same group, or may be represented by a maximum similarity value MS selected from multiple similarity calculation values ​​within the same group.

[0042] In the similarity transition graph in the similarity transition graph display unit 71e, in a case where the eye disease is progressing, the similarity to the reference image data RI related to the eye disease increases over time, as shown by the similarity transition characteristic U indicated by the thick solid line in Fig. 5. On the other hand, in a case where the eye disease is improved by the application of treatment, the similarity to the reference image data RI related to the eye disease decreases over time, as shown by the similarity transition characteristic D indicated by the thin solid line in Fig. 5.

[0043] [Eye disease assessment using the Ophthalmology System S1] In the ophthalmologic system S1, the reference image data classification unit 20 classifies groups of reference image data RI(A), RI(B), etc. into groups by type of ophthalmological disease. The grouped reference image data RI(A), RI(B), etc. are further subdivided into a plurality of stages for each type of disease, ranging from a mild ophthalmological disease stage to a severe ophthalmological disease stage, including a no-ophthalmological stage. Next, the reference image transformation data generation unit 30 uses a first encoder 31 to generate a plurality of reference image vector values ​​R1, R2, R3, ..., Rn, which are reference image transformation data, from each of the subdivided reference image data RI(A), RI(B), etc. into a plurality of stages.

[0044] Meanwhile, in the test image transformation data generation unit 40, by using the second encoder 41 that is the same as the first encoder 31, a plurality of test image vector values ​​P1, P2, P3, ..., Pn, which are test image transformation data, are generated from one test image data item PI newly acquired from the ophthalmologic apparatus 10. The number of the test image vector values ​​P1, P2, P3, ..., Pn is the same as the number of reference image vector values ​​R1, R2, R3, ..., Rn (for example, 512).

[0045] Next, the data comparison unit 50 calculates similarity using the input multiple inspection image vector values ​​P1, P2, P3, ..., Pn and multiple reference image vector values ​​R1, R2, R3, ..., Rn. To calculate similarity, the data comparison unit 50 calculates a scalar product of vector multiplication between paired values ​​of the multiple reference image vector values ​​R1, R2, R3, ..., Rn and the multiple inspection image vector values ​​P1, P2, P3, ..., Pn. The scalar product of vector multiplication is used as a similarity calculation value SC, which numerically represents the degree of similarity. The similarity calculation value SC is output to the judgment information processing unit 60.

[0046] The determination information processing unit 60 calculates an average similarity value AS for each group and selects a maximum similarity value MS for each group as determination information for determining the possibility of an eye disease based on the calculated similarity values ​​SC input from the data comparison unit 50. Various determination information including the average similarity value AS and the maximum similarity value MS is output to a screen display unit 70 having a display screen 71.

[0047] When the display screen 71 of the screen display unit 70 selects information display for determining the possibility of eye disease, the display screen 71 displays a test image display area 71a, a similar reference image display area 71b, and a similarity calculation value display area 71c, as shown in FIG. 4. By viewing the display screen 71 shown in FIG. 4, an eye disease assessor, such as an ophthalmologist, can determine which eye disease group has the highest similarity among the groups and thereby determine whether a certain eye disease is present or suspected. For example, in the case of the DR group on the display screen 71 shown in FIG. 4, the average similarity value AS of the related group is 0.662, which is higher than that of the other groups, and the maximum similarity value MS is 0.973, which is higher than that of the other groups. Therefore, the eye disease assessor can determine that the eye of the patient from whom the test image data PI was acquired is highly suspected of having diabetic retinopathy. Furthermore, the eye disease assessor can visually compare the test image displayed in the test image display area 71a with the reference image displayed in the similar reference image display area 71b to confirm whether their determination that diabetic retinopathy is highly suspected is correct.

[0048] Furthermore, when past similarity data for the same patient exists and information display for determining the progression of the severity of an eye disease is selected, a similarity progression graph display section 71e is displayed on the display screen 71 of the screen display unit 70, as shown on the display screen 71 in Fig. 5. Therefore, when the similarity progression graph display section 71e displays the similarity progression characteristic U shown by the thick solid line in Fig. 5, an eye disease evaluator such as an ophthalmologist can determine that the eye disease is progressing because the similarity for a certain eye disease is increasing over time. On the other hand, when the similarity progression graph display section 71e displays the similarity progression characteristic D shown by the thin solid line, an eye disease evaluator can determine that the eye disease is improving due to the application of treatment because the similarity for a certain eye disease is decreasing over time.

[0049] [Confirmation of sensitivity for determining eye diseases (Figure 6)] FIG. 6 shows the experimental results of the determination sensitivity of the ophthalmologic system S1 of the first embodiment, and the confirmation action of the determination sensitivity for eye diseases will be described with reference to FIG.

[0050] As described above in the eye disease determination function, when determining that there is a high suspicion of a certain eye disease, when determining that the eye disease is progressing, or when determining that the eye disease is improving, high sensitivity of the eye disease prediction stage is a condition for accurate determination. Therefore, the inventor conducted an experiment to confirm the sensitivity of the eye disease prediction stage of the ophthalmologic system S1 of embodiment 1.

[0051] In the experiment, 366 pieces of image data were prepared as image data of eyes belonging to the DR group due to diabetic retinopathy. The ophthalmologic system S1 of embodiment 1 determined the predicted eye disease stage from the 366 pieces of image data. Meanwhile, a group of experts, including experienced ophthalmologists, determined the true eye disease stage for the 366 pieces of image data. Then, the results of the predicted eye disease stage determination were plotted on the horizontal axis, and the results of the true eye disease stage determination were plotted on the vertical axis, and the 366 pieces of image data were sorted into regions to which they belonged, as shown in the lower right of Figure 6.

[0052] The experimental results showed that the predicted eye disease stage and the true eye disease stage matched for 312 of the 366 image data. In other words, the experimental results showed that the sensitivity was 1.0 for the 312 image data. However, the experimental results showed that the predicted eye disease stage and the true eye disease stage did not match for the remaining 54 image data, and the sensitivity was less than 1.0 for the 54 image data.

[0053] Furthermore, the experimental results showed that of the 197 image data pieces with an eye disease prediction stage of stage 0 (healthy), the predicted eye disease stage matched the true eye disease stage for 196 images. However, when the predicted eye disease stages were stage 1, stage 2, stage 3, and stage 4, the degree of agreement between the predicted eye disease stage and the true eye disease stage decreased compared to stage 0.

[0054] As a result, the experimental results showed that the total sensitivity of the agreement between the predicted stage of eye disease and the true stage of eye disease was 0.85, confirming that this sensitivity can determine the stage of eye disease. The experimental results also showed a high specificity of 0.98.

[0055] Other experimental results using the ophthalmology system S1 are shown in the upper part of Figure 6. The average similarity values ​​for the same stage and different stages are listed for the DR group. The average similarity values ​​for the same stage are calculated by performing vector multiplication between two data sets within the same stage and averaging the results. The average similarity values ​​for different stages are calculated by performing vector multiplication between each two data sets in different stages and averaging the results. The average similarity values ​​for the same stage are 0.74 for the no-eye disease stage, 0.69 for stage 1, 0.75 for stage 2, 0.74 for stage 3, and 0.69 for stage 4. The average similarity values ​​between different stages are 0.06 between the no-eye disease stage and stage 1, 0.65 between stages 1 and 2, 0.67 between stages 2 and 3, and 0.65 between stages 3 and 4.

[0056] [Effects of the S1 Ophthalmology System] The ophthalmologic system S1 of the first embodiment has the following advantages.

[0057] (1) The ophthalmologic system S1 determines the possibility of eye disease based on eye image data and includes a reference image data classification unit 20, a reference image conversion data generation unit 30, a test image conversion data generation unit 40, and a data comparison unit 50. The reference image data classification unit 20 groups a large number of image data stored in a database by type of eye disease to generate reference image data RI(A), RI(B), etc. for each group. The reference image conversion data generation unit 30 is a trained AI model trained to extract features from eye image data. It uses a first encoder 31 that converts the extracted features into embedded data through multiple computational layers to generate reference image conversion data RN(A), RN(B), etc. from each of the reference image data RI(A), RI(B), etc. The test image conversion data generation unit 40 uses a second encoder 41 identical to the first encoder 31 to generate test image conversion data PN from a newly acquired set of test image data PI. The data comparison unit 50 compares the test image conversion data PN with the reference image conversion data RN(A), RN(B), etc., and calculates the similarity between the two image conversion data. This ophthalmology system S1 compares two pieces of image conversion data generated using encoders 31 and 41 based on a trained AI model and calculates the similarity, making it possible to easily determine the possibility of eye disease when test image data PI is input.

[0058] (2) The reference image data classification unit 20 has reference image data for each group, such as RI(A), RI(B), etc., which are subdivided into multiple stages for each type of disease, ranging from mild eye disease stages to severe eye disease stages, including a stage without eye disease. When the examination image data PI is input, this ophthalmology system S1 can not only determine the possibility of eye disease, but also the stage of eye disease, which indicates the degree of progression of the eye disease. Furthermore, since the eye disease stages determined include mild eye disease stages, it can contribute to the early detection and treatment of eye diseases and the improvement of medical efficiency.

[0059] (3) The reference image transformation data generation unit 30 sets the reference image transformation data RN(A), RN(B), etc. as a plurality of reference image vector values ​​R1, R2, R3, ..., Rn representing feature vectors generated by compressing the reference image data input to the first encoder 31. The inspection image transformation data generation unit 40 sets the inspection image transformation data PN as a plurality of inspection image vector values ​​P1, P2, P3, ..., Pn representing feature vectors generated by compressing the inspection image data PI input to the second encoder 41. This ophthalmologic system S1 can convert the reference image conversion data RN(A), RN(B), etc. and the test image conversion data PN into vector values ​​that are easy to handle when calculating data comparison.

[0060] (4) The reference image transformation data generation unit 30 and the inspection image transformation data generation unit 40 generate the same number of reference image vector values ​​R1, R2, R3, ..., Rn and inspection image vector values ​​P1, P2, P3, ..., Pn, respectively. The data comparison unit 50 obtains a similarity calculation value SC by taking the scalar product of the reference image vector values ​​R1, R2, R3, ..., Rn and the inspection image vector values ​​P1, P2, P3, ..., Pn. In this ophthalmologic system S1, the data comparison unit 50 can obtain a similarity calculation value SC, which serves as information for determining image similarity when determining eye diseases, by a simple calculation process of the scalar product of vector multiplication.

[0061] (5) A determination information processing unit 60 is provided which receives the calculated similarity values ​​SC from the data comparison unit 50 and processes the determination information required to determine the possibility of an eye disease. The determination information processing unit 60 calculates an average similarity value AS, which is an average of the calculated similarity values ​​SC within the same group, as one piece of determination information. When determining eye diseases, the ophthalmologic system S1 uses the average similarity AS as a determination criterion, and can determine whether or not there is a suspicion of eye disease based on the magnitude of the average similarity AS.

[0062] (6) The judgment information processing unit 60 calculates the maximum similarity value MS, which is the largest value among the multiple similarity calculation values ​​SC within the same group, as one piece of judgment information. When determining eye disease, this ophthalmologic system S1 uses the highest similarity value MS as a determination criterion, and can determine whether eye disease is suspected based on the group that produced the highest similarity value MS.

[0063] (7) A screen display unit 70 is provided with a display screen 71 that displays the judgment information output from the judgment information processing unit 60. The screen display unit 70 has, on the display screen 71, a test image display unit 71a that displays an eye image based on the test image data PI, a similar reference image display unit 71b that displays an eye image based on data with high similarity among the reference image data RI(A), RI(B), etc., and a similarity calculation value display unit 71c that displays the average similarity value AS and the highest similarity value MS for each group. When diagnosing an eye disease, this ophthalmologic system S1 can display on one display screen 71 image information and numerical information that are useful for diagnosing an eye disease.

[0064] (8) A screen display unit 70 is provided having a display screen 71 that displays the determination information output from the determination information processing unit 60. When past similarity data for the same patient exists, the screen display unit 70 has a similarity transition graph display unit 71e that displays on the display screen 71 a similarity transition graph that graphs past similarities and current similarities on a time axis. When determining whether an eye disease has occurred, the ophthalmologic system S1 can determine whether the eye disease is progressing or whether the eye disease has improved as a result of treatment by viewing the displayed similarity transition graph.

[0065] The second embodiment is a configuration in which an optimizer that optimizes image conversion data output from the encoder is added to the system configuration of the first embodiment.

[0066] [System configuration (Fig. 7)] As shown in Fig. 7, the ophthalmologic system S2 of the second embodiment includes a reference image data classification unit 20, a reference image conversion data generation unit 30, a reference image data optimization processing unit 80, a test image conversion data generation unit 40, a test image data optimization processing unit 90, and a data comparison unit 50'. The ophthalmologic system S2 of the second embodiment also includes the determination information processing unit 60 and the screen display unit 70 shown in Fig. 1, but these are not shown in Fig. 7.

[0067] The reference image data optimization processing unit 80 is located between the reference image transformation data generation unit 30 and the data comparison unit 50'. This reference image data optimization processing unit 80 uses a first optimizer 81 that assigns different weights to multiple pieces of embedding data through training to increase the similarity between the same data groups and decrease the similarity between different data groups. Here, the optimizer has the function of converting the embedding data from the reference image transformation data generation unit 30 into different embedding data.

[0068] The training objective of the first optimizer 81 is to increase the similarity between the same data groups and decrease the similarity between different data groups. In other words, when the same data groups are grouped, the training objective is to increase the similarity within the group and decrease the similarity between the groups. Furthermore, when the same data groups are grouped into the same stage, the training objective is to increase the similarity within each stage in each group and decrease the similarity between each stage. Aiming for this objective, the first optimizer 81 performs training in machine learning, thereby assigning different weighting coefficients to each of the multiple reference image vector values ​​R1, R2, R3, ..., Rn, which are embedded data. In other words, the reference image data optimization processing unit 80 uses the first optimizer 81 to optimize the reference image transformation data RN(A), RN(B), etc. into optimized reference image transformation data RNO(A), RNO(B), etc.

[0069] The inspection image data optimization processor 90 is located between the inspection image conversion data generator 40 and the data comparator 50'. This inspection image data optimization processor 90 uses a second optimizer 91, which is the same as the first optimizer 81, for each of the multiple pieces of embedded data. Therefore, the second optimizer 91 performs processing to apply different weighting coefficients obtained by the first optimizer 81 to each of the multiple inspection image vector values ​​P1, P2, P3, ..., Pn, which are the embedded data. In other words, the inspection image data optimization processor 90 uses the second optimizer 91 to optimize the inspection image conversion data PN into optimized inspection image conversion data PNO.

[0070] The data comparison unit 50' is a comparison unit that compares the optimized inspection image conversion data PNO with the optimized reference image conversion data RNO(A), RNO(B), etc., and calculates the similarity between the two image conversion data. Note that the reference image data classification unit 20, reference image conversion data generation unit 30, and inspection image conversion data generation unit 40 are the same as the reference image data classification unit 20, reference image conversion data generation unit 30, and inspection image conversion data generation unit 40 of the first embodiment, and therefore their explanations will be omitted.

[0071] [Optimization process for reference image transformation data (Figure 8)] FIG. 8 shows details of the optimization process for reference image transformation data that increases the similarity within a group and decreases the similarity between groups, and the optimization process for reference image transformation data will be described with reference to FIG.

[0072] The first optimizer 81 is trained by machine learning with the objective of increasing the similarity within a group and decreasing the similarity between groups. For this reason, when the first optimizer 81 receives a plurality of reference image vector values ​​R1, R2, R3, ..., Rn as embedded data, it assigns different weights to each of the plurality of reference image vector values ​​R1, R2, R3, ..., Rn, and assigns different weighting coefficients to each of the plurality of reference image vector values ​​R1, R2, R3, ..., Rn.

[0073] 8, the multiple reference image vector values ​​R1, R2, R3, ..., Rn in group A and group B have small variations in similarity values ​​within each group. On the other hand, group A and group B have large differences in values, making them clearly distinguishable.

[0074] Furthermore, the first optimizer 81 performs training by machine learning with the objective of increasing the similarity within each stage in each group and decreasing the similarity between each stage. For this reason, when the first optimizer 81 receives a plurality of reference image vector values ​​R1, R2, R3, ..., Rn as embedded data, it assigns different weights to each of the plurality of reference image vector values ​​R1, R2, R3, ..., Rn, and assigns different weighting coefficients to each of the plurality of reference image vector values ​​R1, R2, R3, ..., Rn.

[0075] Therefore, for example, the multiple reference image vector values ​​R1, R2, R3, ..., Rn in group A's stages 0 to 4 have small variations in similarity values ​​within each stage. On the other hand, the multiple reference image vector values ​​R1, R2, R3, ..., Rn in group A's stages 0 to 4 have large differences in values ​​between each of stages 0 to 4, so they can be clearly distinguished.

[0076] [Confirmation of the sensitivity of eye disease detection (Figures 9 and 10)] FIG. 8 shows the experimental results of the determination sensitivity using the ophthalmologic system S2 of embodiment 2, and FIG. 9 shows the state of stage movement due to image data optimization. The confirmation action of the determination sensitivity for eye diseases will be explained based on FIGS. 9 and 10.

[0077] The experimental results using the ophthalmology system S2 are shown in the upper part of Figure 9, which lists the average similarity values ​​within the same stage and across different stages in the DR group. The average similarity values ​​within the same stage increased from 0.74 to 0.98 for the stage without ocular disease, from 0.69 to 0.94 for stage 1, from 0.75 to 0.79 for stage 2, from 0.74 to 0.93 for stage 3, and from 0.69 to 0.91 for stage 4. In other words, it was confirmed that the ophthalmology system S2 increased the similarity within each stage in the DR group through data optimization. Furthermore, the average similarity between different stages decreased from 0.06 to -0.16 between the stage without ocular disease and stage 1, from 0.65 to -0.28 between stage 1 and stage 2, from 0.67 to -0.28 between stage 2 and stage 3, and from 0.65 to -0.17 between stage 3 and stage 4. In other words, it was confirmed that ophthalmology system S2 reduces the similarity between each stage in the DR group due to data optimization compared to ophthalmology system S1.

[0078] The experiment shown in Fig. 6 of the above-mentioned embodiment 1 was also conducted in embodiment 2. Then, with the judgment results of the predicted eye disease stage on the horizontal axis and the judgment results of the true eye disease stage on the vertical axis, the 366 image data were divided into areas to which they belonged, as shown in the lower right of Fig. 9.

[0079] The experimental results showed that the predicted and true eye disease stages matched for 318 of the 366 image data sets. In other words, the experimental results showed a sensitivity of 1.0 for the 318 image data sets, six more than for the ophthalmic system S1. The experimental results showed that the predicted and true eye disease stages did not match for 54 image data sets in the ophthalmic system S1, whereas the predicted and true eye disease stages did not match for 48 image data sets, six fewer than the ophthalmic system S2. The breakdown of the six missing image data sets is shown in Figure 10. The ophthalmic system S2 performed an optimization process on the comparison data, which shifted six image data sets to positions where the predicted and true eye disease stages matched, as indicated by the arrows on the right side of Figure 10. The reason why the predicted and true eye disease stages for the 48 image data sets still did not match could be, for example, due to dirt adhering to the lens of the ophthalmic device capturing the images, resulting in the dirt being captured in the image data.

[0080] As a result, the experimental results showed that the total sensitivity of the predicted eye disease stage and the true eye disease stage was 0.88 (>0.85), confirming that the eye disease stage can be determined with 88% sensitivity.The experimental results also showed a high specificity of 0.98 (=0.98).

[0081] [Effects of the S2 Ophthalmology System] The ophthalmologic system S2 of the second embodiment has the following advantages in addition to the advantages (1) to (8) of the first embodiment.

[0082] (9) Between the reference image conversion data generation unit 30 and the data comparison unit 50', there is a reference image data optimization processing unit 80 that optimizes the reference image conversion data RN(A), RN(B), etc. into optimized reference image conversion data RNO(A), RNO(B), etc. The reference image data optimization processing unit 80 uses a first optimizer 81 that assigns different weights to each of multiple embedded data through training to increase the similarity between the same data groups and decrease the similarity between different data groups. Between the inspection image conversion data generation unit 40 and the data comparison unit 50', there is an inspection image data optimization processing unit 90 that optimizes the inspection image conversion data PN into optimized inspection image conversion data PNO. The inspection image data optimization processing unit 90 uses a second optimizer 91 that is the same as the first optimizer 81. The data comparison unit 50' is a comparison unit that compares the optimized inspection image conversion data PNO with the optimized reference image conversion data RNO(A), RNO(B), etc., and calculates the similarity between the two image conversion data. By adding the first optimizer 81 and the second optimizer 91, the ophthalmic system S2 has higher sensitivity in determining what eye disease a patient has, compared to the ophthalmic system S1 which does not have an optimizer, and can improve the accuracy of predicting eye diseases.

[0083] The ophthalmic system S1 of Embodiment 1 and the ophthalmic system S2 of Embodiment 1 have been described above with reference to the drawings. However, the specific configuration of the ophthalmic system of the present disclosure is not limited to Embodiments 1 and 2, and design changes and additions are permitted as long as they do not deviate from the gist of the invention according to each claim of the claims.

[0084] In the first and second embodiments, the reference image data classifying unit 20 has been described as subdividing the reference image data for each group into a plurality of stages, ranging from a mild eye disease stage to a severe eye disease stage, including a stage without eye disease. However, the reference image data classifying unit is not limited to subdividing the reference image data into a plurality of stages in addition to the disease type. For example, the reference image data classifying unit may simply classify the reference image data into groups by disease type and determine whether there is a possible eye disease.

[0085] In the first and second embodiments, the reference image transformation data generation unit 30 and the test image transformation data generation unit 40 are configured to use multiple image vector values ​​representing feature vectors generated by compressing image data as image transformation data. However, the reference image transformation data generation unit and the test image transformation data generation unit are not limited to the example in which multiple image vector values ​​are used as image transformation data. In other words, the image transformation data may be embedded data obtained by converting feature quantities extracted through multiple calculation layers.

[0086] In the first and second embodiments, the determination information processing unit 60 acquires the average similarity value AS and the maximum similarity value MS as the determination information. However, the determination information processing unit is not limited to the example in which the average similarity value and the maximum similarity value are used as the determination information. In other words, the determination information may be any information useful for determining the possibility of an eye disease based on the calculated similarity value.

[0087] In the first and second embodiments, the screen display unit 70 includes a test image display unit 71a, a similar reference image display unit 71b, a similarity calculation value display unit 71c, and a similarity transition graph display unit 71e on the display screen 71. However, the screen display unit is not limited to having the above-described display units on the display screen. For example, the screen display unit may display other graphs, such as bar graphs, on the display screen. [Explanation of symbols]

[0088] S1, S2 Ophthalmology System 10 Ophthalmological equipment 20 Reference image data classification unit 30 Reference image conversion data generation unit 31 First Encoder 40 Inspection image conversion data generation unit 41 Second Encoder 50 Data comparison section 60 Judgment information processing unit 70 Screen display section 71 Display screen 80 Reference image data optimization processing unit 81 First Optimizer 90 Inspection image data optimization processing section 91 Second Optimizer 50' Data comparison section

Claims

1. An ophthalmology system for determining a possibility of an eye disease based on eye image data, comprising: a reference image data classification unit that groups a large number of image data stored in a database by type of eye disease to generate reference image data for each group; a reference image transformation data generation unit that generates reference image transformation data from each of the reference image data using a first encoder that is a trained AI model that has been trained to extract features from eye image data and that converts the extracted features into embedded data through multiple calculation layers; and an inspection image conversion data generating unit that generates inspection image conversion data from one newly acquired inspection image data by using a second encoder that is the same as the first encoder; a data comparison unit that compares the inspection image transformation data with the reference image transformation data and calculates the similarity between the two image transformation data. An ophthalmology system comprising:

2. 10. The ophthalmology system according to claim 1, The reference image data classification unit has, as the reference image data for each group, data for each type of disease, which is subdivided into a plurality of stages ranging from a mild eye disease stage to a severe eye disease stage, including a stage without eye disease. An ophthalmology system comprising:

3. 3. The ophthalmology system according to claim 2, the reference image transformation data generation unit sets the reference image transformation data as a plurality of reference image vector values ​​representing feature vectors generated by compressing the reference image data input to the first encoder, The inspection image transformation data generation unit generates the inspection image transformation data as a plurality of inspection image vector values ​​representing feature vectors generated by compressing the inspection image data input to the second encoder. An ophthalmology system comprising:

4. 4. The ophthalmology system according to claim 3, the reference image transformation data generation unit and the inspection image transformation data generation unit generate the same number of reference image vector values ​​and the same number of inspection image vector values, respectively; The data comparison unit obtains a similarity calculation value by a scalar product of the reference image vector value and the inspection image vector value. An ophthalmology system comprising:

5. 5. The ophthalmology system according to claim 4, a determination information processing unit that receives the calculated similarity value from the data comparison unit and processes determination information necessary for determining the possibility of an eye disease; The determination information processing unit calculates an average similarity value by averaging a plurality of the similarity calculation values ​​within the same group as one of the determination information. An ophthalmology system comprising:

6. 6. The ophthalmology system according to claim 5, The determination information processing unit calculates a maximum similarity value among a plurality of the similarity calculation values ​​in the same group as one of the determination information. An ophthalmology system comprising:

7. 7. The ophthalmology system according to claim 6, a screen display unit having a display screen for displaying the determination information output from the determination information processing unit, The screen display unit has a test image display unit that displays an eye image based on the test image data on the display screen, a similar reference image display unit that displays an eye image based on data with a high similarity among the reference image data, and a similarity calculation value display unit that displays the average similarity value and the highest similarity value for each group. An ophthalmology system comprising:

8. 7. The ophthalmology system according to claim 6, a screen display unit having a display screen for displaying the determination information output from the determination information processing unit, The screen display unit has a similarity transition graph display unit that displays, on the display screen when past similarity data of the same patient exists, a similarity transition graph that graphs past similarities and current similarities on a time axis. An ophthalmology system comprising:

9. 9. The ophthalmology system according to claim 1, a reference image data optimization processing unit between the reference image conversion data generation unit and the data comparison unit, which uses a first optimizer that assigns different weights to each of a plurality of embedded data by training to increase the similarity between the same data group and decrease the similarity between different data groups, and which optimizes the reference image conversion data to generate optimized reference image conversion data; an inspection image data optimization processing unit between the inspection image conversion data generation unit and the data comparison unit, which uses a second optimizer identical to the first optimizer to optimize the inspection image conversion data into optimized inspection image conversion data; The data comparison unit is a comparison unit that compares the optimized inspection image transformation data with the optimized reference image transformation data and calculates the similarity between the two image transformation data. An ophthalmology system comprising:

Images