Ophthalmic device, method for controlling an ophthalmic device, program, and recording medium

Abstract

To automate evaluation of an inflammatory condition based on an eye image.SOLUTION: An ophthalmologic apparatus 1000 according to an exemplary embodiment composes an image acquisition unit 1010 and a data processing unit 1020. The image acquisition unit 1010 is configured to acquire a shine proof image of a subject eye. The data processing unit 1020 is configured to execute processing for generating inflammatory condition information indicating an inflammatory condition of the subject eye from the shine proof image acquired by the image acquisition unit 1010.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an ophthalmic device and a method for controlling the ophthalmic device. , pu This document relates to a program and a recording medium. [Background technology]

[0002] Diagnostic imaging plays an important role in the field of ophthalmology. Various ophthalmic devices are used for diagnostic imaging. These devices include slit lamp microscopes, fundus cameras, scanning laser ophthalmoscopes (SLO), and optical coherence tomography (OCT). In addition, various examination and measurement devices, such as refractometers, keratometers, tonometers, specular microscopes, wavefront analyzers, and microperimeters, are also equipped with the function to photograph the anterior segment and fundus.

[0003] Among these various ophthalmic devices, one of the most widely and frequently used is the slit lamp microscope, also known as the stethoscope for ophthalmologists. A slit lamp microscope is an ophthalmic device that illuminates the subject's eye with a slit of light and observes and photographs the illuminated cross-section from the side using a microscope (see, for example, Patent Documents 1 and 2). Also known is a slit lamp microscope that can scan a three-dimensional area of ​​the subject's eye at high speed by using an optical system configured to satisfy the Scheimpflug condition (see, for example, Patent Document 3). In addition to slit lamp microscopes, there are also imaging methods such as rolling shutter cameras that scan an object with a slit of light.

[0004] An ophthalmic device that performs measurements using slit light is a flare cell meter for evaluating the inflammatory state of the subject's eye (see, for example, Patent Documents 4 and 5). The flare cell meter is an ophthalmic device that measures the number of inflammatory cells floating in the anterior chamber and the protein concentration (flare concentration) in the anterior chamber, and generates slit light, for example, by one-dimensionally scanning a laser beam or by restricting LED light with a slit aperture. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-159073 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-179004 [Patent Document 3] Japanese Patent Application Publication No. 2019-213733 [Patent Document 4] Japanese Patent Application Laid-Open No. 2005-102938 [Patent Document 5] International Publication No. 2018 / 003906 Summary of the Invention [Problem to be solved by the invention]

[0006] One object of the present invention is to automate the assessment of inflammatory conditions based on ocular images. [Means for solving the problem]

[0007] An ophthalmologic apparatus according to an exemplary embodiment includes an image acquisition unit and a data processing unit. The image acquisition unit acquires a Scheimpflug image of a subject's eye. The data processing unit executes processing to generate inflammation state information indicating an inflammation state of the subject's eye from the Scheimpflug image acquired by the image acquisition unit. [Effects of the Invention]

[0008] According to exemplary embodiments, the assessment of inflammatory conditions based on eye images can be automated. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 2A] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 2B] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 2C] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 3] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 4] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 5] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 6] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 7] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 8] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 9] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 10] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 11] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 12] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 13] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 14] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 15] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 16] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 17] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 18] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 19] FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. [Figure 20] FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. [Figure 21] FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. [Figure 22] FIG. 10 is a flowchart illustrating a process performed by an ophthalmic apparatus according to an exemplary embodiment. [Figure 23] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 24] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. [Figure 25A] 1 is a schematic diagram illustrating the operation of an ophthalmic apparatus according to an exemplary embodiment. [Figure 25B] 1 is a schematic diagram illustrating the operation of an ophthalmic apparatus according to an exemplary embodiment. [Figure 26] 1 is a schematic diagram illustrating the operation of an ophthalmic apparatus according to an exemplary embodiment. [Figure 27] 1 is a schematic diagram illustrating a configuration of an ophthalmic apparatus according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Some exemplary aspects of the embodiments will be described in detail with reference to the drawings. Any known technology can be combined with the embodiments. For example, any known technology in the technical field, such as any matter disclosed in the documents cited in this specification, can be combined with any of the embodiments. In particular, all contents disclosed in Patent Document 3 (JP 2019-213733 A) are incorporated by reference into the present disclosure. Similarly, any technical matter disclosed by the applicant of the present application regarding technology related to the present disclosure (matters disclosed in patent applications, papers, etc.) can be combined with any of the embodiments. Furthermore, any two or more of the various aspects of the present disclosure can be at least partially combined.

[0011] At least a portion of the functionality of the elements described in this disclosure is implemented using circuitry or processing circuitry. The circuitry or processing circuitry may be a general-purpose processor, a special-purpose processor, an integrated circuit, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA)), or a combination of these devices configured and / or programmed to perform at least a portion of the disclosed functionality. The term "circuitry," "unit," "means," or the like refers to hardware that performs at least a portion of the disclosed functions or that is programmed to perform at least a portion of the disclosed functions. The hardware may be the hardware disclosed herein or may be known hardware that is programmed and / or configured to perform at least a portion of the described functions. In the case of a processor, where the hardware can be considered a type of circuitry, the term "circuitry," "unit," "means," or the like refers to a combination of hardware and software, where the software is used to configure the hardware and / or the processor.

[0012] <Outline of the embodiment> The embodiment relates to a technology for generating information indicating the inflammatory state of a test eye (referred to as inflammatory state information) from a digital image (referred to as a Scheimpflug image) generated by photographing the test eye using an optical system that satisfies the Scheimpflug condition.

[0013] The number of Scheimpflug images processed by the embodiments may be any number, with some embodiments processing one Scheimpflug image and some embodiments processing multiple images. The multiple images may be collected, for example, by scanning with a slit of light (referred to as slit scanning). Slit scanning is an ophthalmic imaging technique developed by the applicant of the present application that collects a series of images by scanning a three-dimensional region of a subject's eye with a slit of light, and is described in Patent Document 3 (JP 2019-213733 A) and the like.

[0014] The Scheimpflug image processed by some embodiments may be an image created by processing a Scheimpflug image, such as an image created by applying any digital image processing such as correction, editing, or enhancement to a Scheimpflug image, a three-dimensional image (volume image) constructed from multiple Scheimpflug images, or an image created by applying any rendering to a three-dimensional image.

[0015] The inflammation status information generated by the embodiments may be any information related to the inflammation status of the subject's eye. The inflammation status information of the embodiments may include one or more of information on inflammatory cells (anterior chamber cells) present in the anterior chamber, information on proteins (anterior chamber flare) present in the anterior chamber, information on lens opacity, information on the onset and progression of a disease, and information on disease activity. Some embodiments may generate comprehensive information (e.g., comprehensive evaluation information) based on two or more pieces of information.

[0016] Information about inflammatory cells includes information on arbitrary parameters such as the density (concentration), number, location, and distribution of inflammatory cells, as well as evaluation information based on information on specified parameters. Evaluation information about inflammatory cells is called cell evaluation information. Information about anterior chamber flare includes information on arbitrary parameters such as the density, number, location, and distribution of flare, as well as evaluation information based on information on specified parameters. Information about lens opacity includes information on arbitrary parameters such as the density, number, location, and distribution of opacity, as well as evaluation information based on information on specified parameters. Information about the onset and progression of disease includes information on arbitrary parameters such as the presence or absence of onset, the state of onset, the duration of disease, and the state of progression, as well as evaluation information based on information on specified parameters. Information about disease activity includes information on arbitrary parameters such as the state of disease activity, as well as evaluation information based on information on specified parameters.

[0017] An exemplary embodiment of generating evaluation information regarding an inflammatory condition may be configured to generate the evaluation information based not only on information generated by an ophthalmic device that performs the process of generating the evaluation information, but also on information input to the ophthalmic device from outside (e.g., information obtained by another ophthalmic device, information input by a doctor).

[0018] The information referenced to generate the inflammatory status information exemplified above may be any information and may include, for example, the classification criteria for uveitis diseases proposed by the Standardization of Uveitis Nomenclature (SUN) Working Group ("Standardization of uveitis nomenclature for reporting clinical data. Results of the First International Workshop," American Journal of Ophthalmology, Volume 140, Issue 3, September 2005, Pages 509-516).

[0019] It should be noted that the inflammation state information of the embodiment is not limited to the above example, and the reference information for generating the inflammation state information of the embodiment is not limited to the above example either.

[0020] In the embodiments, exemplary aspects in which slit scanning can be applied to the anterior segment of the subject's eye are also described. The portion of the subject's eye to which slit scanning is applied includes at least a portion or the entire portion to which slit scanning is applied to acquire data used to generate inflammation state information. For example, when generating inflammation state information including information on inflammatory cells and / or information on anterior chamber flare, slit scanning is applied to a region including at least a portion of the anterior chamber. Furthermore, when generating inflammation state information including information on lens opacity, slit scanning is applied to a region including at least a portion of the lens. The same applies when acquiring two or more Scheimpflug images using a method other than slit scanning, or when acquiring only one Scheimpflug image.

[0021] Generally, the region (site) of the subject's eye to which the slit scan is applied includes at least a portion of the anterior segment (e.g., tissues such as the cornea, iris, anterior chamber, angle, ciliary body, zonules, lens, nerves, and blood vessels; lesions; treatment scars; and artificial structures such as intraocular lenses and minimally invasive glaucoma surgery (MIGS) devices) and / or at least a portion of the posterior segment (e.g., tissues such as the vitreous body, retina, choroid, sclera, optic disc, lamina cribrosa, macula, nerves, and blood vessels; lesions; treatment scars; and artificial structures such as retinal prostheses). In some exemplary embodiments, the slit scan may be applied to at least a portion of tissues near the eye, such as the eyelids and meibomian glands. In some exemplary embodiments, the slit scan may be applied to a three-dimensional region including any two or all of at least a portion of the anterior segment, at least a portion of the posterior segment, and at least a portion of tissues near the eye.

[0022] An embodiment capable of generating inflammation state information including the above-described cell evaluation information may be configured to perform at least one of the following three processes: (1) segmentation (referred to as first segmentation or anterior chamber segmentation) for identifying an image region corresponding to the anterior chamber of the subject's eye (referred to as an anterior chamber region); (2) segmentation (referred to as second segmentation or cell segmentation) for identifying an image region corresponding to inflammatory cells (referred to as a cell region); (3) processing for generating cell evaluation information (referred to as cell evaluation information generation processing). Some exemplary aspects of such an embodiment are described below.

[0023] The first exemplary embodiment is configured to perform at least one of a first segmentation for identifying an anterior chamber region from a Scheimpflug image, a second segmentation for identifying a cellular region from the anterior chamber region identified by the first segmentation, and a cell evaluation information generation process for generating cell evaluation information from the cellular region identified by the second segmentation. The first segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The second segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

[0024] A second exemplary embodiment is configured to perform at least one of a second segmentation for identifying a cellular region from a Scheimpflug image without performing a first segmentation for identifying an anterior chamber region, and a cell evaluation information generation process for generating cell evaluation information from the cellular region identified by the second segmentation. The second segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

[0025] A third exemplary embodiment is configured to perform a cell evaluation information generation process for generating cell evaluation information from a Scheimpflug image without performing a first segmentation for identifying an anterior chamber region and a second segmentation for identifying a cellular region. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

[0026] A fourth exemplary embodiment is configured to perform at least one of a first segmentation for identifying an anterior chamber region from a Scheimpflug image and a cell evaluation information generation process for generating cell evaluation information from the anterior chamber region identified by the first segmentation, without performing a second segmentation for identifying a cellular region. The first segmentation of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. The cell evaluation information generation process of this embodiment may be performed using a neural network trained by machine learning, but is not limited to this. Details of this embodiment will be described later.

[0027] A fifth exemplary embodiment is implemented without using a neural network trained by machine learning. This embodiment is configured to perform a first segmentation process to identify an anterior chamber region by analyzing a Scheimpflug image, a second segmentation process to identify a cellular region by analyzing the anterior chamber region identified by the first segmentation, and a cell evaluation information generation process to generate cell evaluation information based on the cellular region identified by the second segmentation. Details of this embodiment will be described later.

[0028] The sixth exemplary aspect is configured to execute at least one or more of at least a part of the first segmentation, at least a part of the second segmentation, and at least a part of the cell evaluation information generation process using a machine learning-based configuration, and to execute processes other than those executed using the machine learning-based configuration using a non-machine learning-based configuration. This aspect is realized, for example, by partially combining any of the above-described first to fourth exemplary aspects with the fifth exemplary aspect, and therefore a detailed description thereof will be omitted.

[0029] <Ophthalmological equipment> An exemplary embodiment of the ophthalmic apparatus according to the present invention will be described below.

[0030] 1 shows the configuration of an ophthalmic apparatus according to this embodiment. The ophthalmic apparatus 1000 includes an image acquisition unit 1010 and a data processing unit 1020.

[0031] The image acquisition unit 1010 is configured to acquire a Scheimpflug image of the subject's eye. In some exemplary embodiments, the image acquisition unit 1010 is configured to photograph the subject's eye and acquire the Scheimpflug image. An example configuration of such an image acquisition unit 1010 is shown in FIG. 2A.

[0032] 2A includes an illumination system 1011 and an imaging system 1012. The illumination system 1011 is configured to project a slit of light onto the subject's eye. The imaging system 1012 is configured to capture an image of the subject's eye, and includes an image sensor 1013 and an optical system (not shown) that guides light from the subject's eye to the image sensor 1013.

[0033] The illumination system 1011 and the imaging system 1012 are configured to satisfy the Scheimpflug condition and function as a Scheimpflug camera. More specifically, the illumination system 1011 and the imaging system 1012 are configured so that a plane passing through the optical axis of the illumination system 1011 (a plane including the object plane), the principal plane of the imaging system 1012, and the imaging plane of the image sensor 1013 intersect on the same straight line. This allows imaging to be performed with the imaging system 1012 in focus at all positions within the object plane (all positions in the direction along the optical axis of the illumination system 1011).

[0034] 2A is configured to collect a series of Scheimpflug images by scanning a three-dimensional region of the subject's eye with a slit of light. The image acquisition unit 1010A in this example is configured to collect the series of Scheimpflug images by repeatedly capturing images while moving the projection position of the slit of light relative to the three-dimensional region of the subject's eye.

[0035] In some exemplary embodiments, the image acquisition unit 1010A may be configured to scan a three-dimensional region of the subject's eye by translating the slit light in a direction perpendicular to the longitudinal direction of the slit light, which differs from conventional anterior segment imaging devices that scan the anterior segment by rotating the slit light.

[0036] Here, the longitudinal direction of the slit light is the longitudinal direction of the beam cross section of the slit light at the projection position on the subject's eye, in other words, the longitudinal direction of the slit light image formed on the subject's eye, and may be approximately coincident with the direction along the subject's body axis (body axis direction). Furthermore, the dimension of the slit light in the longitudinal direction may be equal to or greater than the corneal diameter in the body axis direction of the subject, and the distance of translation of the slit light may be equal to or greater than the corneal diameter in the direction perpendicular to the body axis direction of the subject.

[0037] The series of Scheimpflug images collected by the image acquisition unit 1010A in this example is a group of images (group of frames) collected continuously in time, but since they are a group of images collected sequentially from multiple different positions in the three-dimensional area of ​​the test eye, they are a group of images that are spatially distributed, unlike general moving images.

[0038] In the image acquisition unit 1010A of this example, an illumination system 1011 projects slit light onto a three-dimensional area of ​​the subject's eye, and an imaging system 1012 images the three-dimensional area of ​​the subject's eye onto which the slit light from the illumination system 1011 is projected. The image acquisition unit 1010A of this example further includes a mechanism for moving the illumination system 1011 and the imaging system 1012.

[0039] The data processing unit 1020 of the ophthalmologic apparatus 1000 to which the image acquisition unit 1010A of the present example is applied may be configured to generate inflammation state information from a Scheimpflug image included in the series of Scheimpflug images collected by the image acquisition unit 1010A. The data processing unit 1020 of the present example generates the inflammation state information by processing one or more Scheimpflug images included in the series of Scheimpflug images collected by the image acquisition unit 1010A. Here, the number of Scheimpflug images used to generate the inflammation state information may be arbitrary.

[0040] In addition, the data processing unit 1020 of the ophthalmic device 1000 to which the image acquisition unit 1010A of this example is applied may be configured to execute a process of processing a series of Scheimpflug images collected by the image acquisition unit 1010A to generate processed image data, and a process of generating inflammation status information based on the generated processed image data.

[0041] For example, the data processing unit 1020 of this example may be configured to execute a process of constructing a three-dimensional image (an example of processed image data) from a plurality of Scheimpflug images included in the series of Scheimpflug images, and a process of generating inflammation state information based on this three-dimensional image. Alternatively, the data processing unit 1020 of this example may be configured to execute a process of constructing a three-dimensional image from a plurality of Scheimpflug images included in the series of Scheimpflug images, a process of generating a rendering image (an example of processed image data) from this three-dimensional image, and a process of generating inflammation state information based on this rendering image.

[0042] In some exemplary embodiments, the imaging system 1012 of the image acquisition unit 1010A may include two or more imaging systems. For example, the imaging system 1012A of the image acquisition unit 1010B shown in FIG. 2B includes a first imaging system 1014 and a second imaging system 1015 that capture images from different directions.

[0043] The image acquiring unit 1010B of this example may be configured to capture images of a three-dimensional region of the subject's eye from different directions using the first imaging system 1014 and the second imaging system 1015 in a slit scan for collecting a series of Scheimpflug images. For example, in a case where the image acquiring unit 1010B is configured so that the longitudinal direction of the beam cross section of the slit light at the incident position on the subject's eye is the up-down direction (Y direction) and the moving direction of the slit light is the horizontal direction (left-right direction, X direction), the first imaging system 1014 and the second imaging system 1015 may be arranged so that one captures an image of the subject's eye from an oblique left direction and the other captures an image of the subject's eye from an oblique right direction.

[0044] A series of Scheimpflug images collected by the first imaging system 1014 is called a first Scheimpflug image group, and a series of Scheimpflug images collected by the second imaging system 1015 is called a second Scheimpflug image group. The series of Scheimpflug images collected by the image acquisition unit 1010B includes a first Scheimpflug image group and a second Scheimpflug image group.

[0045] Note that even when one Scheimpflug image (first Scheimpflug image) is acquired by the first imaging system 1014 and one Scheimpflug image (second Scheimpflug image) is acquired by the second imaging system 1015 without performing slit scanning, for convenience of terminology, the one Scheimpflug image acquired by the first imaging system 1014 may be referred to as a first Scheimpflug image group, and the one Scheimpflug image acquired by the second imaging system 1015 may be referred to as a second Scheimpflug image group. Thus, in the present disclosure, the term "group" may be used not only when multiple elements are included, but also when only one element is included.

[0046] When the image acquiring unit 1010B of this example performs slit scanning, the imaging of the subject's eye by the first imaging system 1014 and the imaging of the subject's eye by the second imaging system 1015 are performed in parallel. That is, the image acquiring unit 1010B performs imaging by the first imaging system 1014 and the imaging by the second imaging system 1015 in parallel while moving the projection position of the slit light with respect to the three-dimensional region of the subject's eye.

[0047] Furthermore, the image acquisition unit 1010B of this example may be configured to synchronize the image capture by the first imaging system 1014 (collection of Scheimpflug images) with the image capture by the second imaging system 1015 (collection of Scheimpflug images). By referring to this synchronization relationship, the first Scheimpflug image group and the second Scheimpflug image group can be easily associated with each other without using image processing or the like. This association is performed, for example, so as to associate Scheimpflug images that are acquired at a small difference in time with each other.

[0048] In such an embodiment, the ophthalmic device 1000 can reconstruct a series of Scheimpflug images corresponding to the slit scan from the first Scheimpflug image group and the second Scheimpflug image group by referring to the mutual synchronization relationship between the image capture by the first image capture system 1014 and the image capture by the second image capture system 1015.

[0049] Fig. 2C shows an example of the configuration of an ophthalmologic apparatus 1000 to which the image acquisition unit 1010B shown in Fig. 2B is applied. In the image acquisition unit 1010B of this example, the optical axis of the first imaging system 1014 and the optical axis of the second imaging system 1015 are arranged to be inclined in opposite directions relative to the optical axis of the illumination system 1011. Furthermore, the data processing unit 1020A of this example includes an image selection unit 1021 and an inflammation state information generation unit 1022.

[0050] The image selection unit 1021 is configured to select an image from among the first Scheimpflug image group acquired by the first imaging system 1014 and the second Scheimpflug image group acquired by the second imaging system 1015. For example, the image selection unit 1021 may be configured to select one of the first Scheimpflug image acquired by the first imaging system 1014 and the second Scheimpflug image acquired by the second imaging system 1015.

[0051] The image selection unit 1021 is configured to select, from the first Scheimpflug image group and the second Scheimpflug image group, a new series of Scheimpflug images corresponding to the slit scans from which the first Scheimpflug image group and the second Scheimpflug image group were acquired, based on a correspondence relationship between the first Scheimpflug image group and the second Scheimpflug image group, respectively, acquired based on synchronization between image acquisition by the first imaging system 1014 and image acquisition by the second imaging system 1015. In short, the image selection unit 1021 is configured to reconstruct a series of Scheimpflug images from the first Scheimpflug image group and the second Scheimpflug image group acquired by the first imaging system 1014 and the second imaging system 1015, respectively.

[0052] Any method may be used for the image selection process executed by the image selection unit 1021. For example, the method for the image selection process may be determined and / or selected based on predetermined conditions or predetermined parameters, such as the configuration and / or arrangement of the first imaging system 1014 and the second imaging system 1015, or the purpose and / or use of image selection.

[0053] The image acquisition unit 1010B performs imaging by the first imaging system 1014 and imaging by the second imaging system 1015 in synchronization with each other. As described above, the optical axis of the first imaging system 1014 and the optical axis of the second imaging system 1015 are arranged to be inclined in opposite directions with respect to the optical axis of the illumination system 1011. For example, the optical axis of the first imaging system 1014 is arranged to be inclined leftward with respect to the optical axis of the illumination system 1011, and the optical axis of the second imaging system 1015 is arranged to be inclined rightward with respect to the optical axis of the illumination system 1011. The first imaging system 1014 and the second imaging system 1015 arranged in this manner may be referred to as the left imaging system and the right imaging system, respectively.

[0054] The tilt angle of the optical axis of the first imaging system 1014 relative to the optical axis of the illumination system 1011 and the tilt angle of the optical axis of the second imaging system 1015 relative to the optical axis of the illumination system 1011 may be equal to or different from each other. Furthermore, these tilt angles may be fixed or variable.

[0055] The illumination system 1011 of this example is constructed and arranged to project slit light, the longitudinal direction of which is oriented in the Y direction, onto the subject's eye from the front direction. The image acquisition unit 1010B of this example also applies slit scanning to a three-dimensional region of the anterior segment of the subject's eye by integrally moving the illumination system 1011, the first imaging system 1014, and the second imaging system 1015 in the X direction.

[0056] In this example, the image selection unit 1021 selects a new set of Scheimpflug images corresponding to the slit scan from which the first and second Scheimpflug image groups were acquired by selecting a plurality of Scheimpflug images that do not contain artifacts from the first and second Scheimpflug image groups based on the correspondence between the first and second Scheimpflug image groups acquired by the first imaging system 1014 and the second Scheimpflug image group acquired by the second imaging system 1015. This artifact may be any type of artifact. In the case of an anterior segment scan as in this example, this artifact may be an artifact caused by corneal reflection (referred to as a corneal reflection artifact). Below, several examples of the processing performed by the image selection unit 1021 will be described.

[0057] The projection position (Scheimpflug image, frame) of the slit light at which the corneal reflection artifact occurs differs between the left and right imaging systems. For example, as in this example, when slit scanning is performed by projecting slit light whose cross-section longitudinal direction is oriented in the Y direction onto the subject's eye from the front direction while integrally moving the illumination system 1011, the first imaging system 1014, and the second imaging system 1015 in the X direction, the corneal reflection light of the slit light is likely to enter the left imaging system when the slit light is projected to a position to the left of the corneal apex, and is likely to enter the right imaging system when the slit light is projected to a position to the right of the corneal apex.

[0058] Taking these circumstances into consideration, the image selection unit 1021 in some exemplary embodiments is configured to first identify a Scheimpflug image (first corneal apex image) corresponding to the corneal apex from among the first Scheimpflug image group collected by the first imaging system 1014 as the left imaging system, and to identify a Scheimpflug image (second corneal apex image) corresponding to the corneal apex from among the second Scheimpflug image group collected by the second imaging system 1015 as the right imaging system.

[0059] In some exemplary embodiments, the process of identifying the corneal apex image may include a process of detecting an image corresponding to the corneal surface from each Scheimpflug image included in the first Scheimpflug image group, a process of identifying a pixel closest to the ophthalmic apparatus 1000 of this example based on the Z coordinates of pixels in these detected images, and a process of setting the Scheimpflug image including the identified pixel as the first corneal apex image. Setting the second corneal apex image may be performed in the same manner.

[0060] Next, the image selection unit 1021 selects a Scheimpflug image group located to the right of the first corneal vertex image from the first Scheimpflug image group, and selects a Scheimpflug image group located to the left of the second corneal vertex image from the second Scheimpflug image group, thereby forming a series of Scheimpflug images consisting of the two selected Scheimpflug image groups (and the first corneal vertex image and / or the second corneal vertex image). This results in a series of Scheimpflug images that cover the three-dimensional region of the anterior segment to which the slit scan has been applied and that are (highly likely to) not contain corneal reflection artifacts.

[0061] Another example of the process for identifying a corneal apex image will be described. In some exemplary embodiments, the image selection unit 1021 is configured to determine whether a corneal reflection artifact is included in either of two images acquired substantially simultaneously by the first imaging system 1014 (e.g., the left imaging system) and the second imaging system 1015 (e.g., the right imaging system). This corneal reflection artifact determination process includes a predetermined image analysis, such as threshold processing on brightness information assigned to pixels. Note that the process for determining whether the two images were acquired substantially simultaneously can be performed based on a synchronization relationship between the imaging by the first imaging system 1014 and the imaging by the second imaging system 1015.

[0062] The threshold processing used in the artifact determination process is performed, for example, to identify pixels assigned a brightness value exceeding a preset threshold. Typically, the threshold may be set higher than the brightness value of the slit light image (the slit light projection area) in the image. As a result, the image selection unit 1021 is configured to determine an image brighter than the slit light image as an artifact without determining the slit light image as an artifact. Considering that an image brighter than the slit light image in a Scheimpflug image is likely to be an image caused by specular reflection from the cornea, it can be considered that an artifact detected by the image selection unit 1021 configured in this manner is likely to be a corneal reflection artifact.

[0063] For artifact determination, the image selection unit 1021 may perform any image analysis other than threshold processing, such as pattern recognition, segmentation, edge detection, etc. In general, any information processing technology, such as image analysis, image processing, machine learning, artificial intelligence, cognitive computing, etc., can be applied to artifact determination.

[0064] When it is determined as a result of the artifact determination that one of the two images acquired substantially simultaneously by the first imaging system 1014 and the second imaging system 1015 contains an artifact, the image selection unit 1021 selects the other image. In other words, the image selection unit 1021 selects the other image, which is not the image determined to contain the artifact, from the two images acquired substantially simultaneously by the first imaging system 1014 and the second imaging system 1015.

[0065] Assuming that both images are determined to contain artifacts, the image selection unit 1021 may be configured to, for example, perform a process of evaluating the magnitude of the adverse effect that the artifacts have on observation or diagnosis, and a process of selecting the image with the smaller adverse effect. This evaluation process may be performed based on, for example, one or more conditions of the size, intensity, shape, and position of the artifact. Typically, artifacts with large size, high intensity, and artifacts located in or near a region of interest such as a slit light image are evaluated as having a large adverse effect.

[0066] In cases where both images contain artifacts, the artifact removal disclosed in Patent Document 3 (JP 2019-213733 A) may be applied.

[0067] By providing the image selection unit 1021 as described above, it is possible to provide an image of a three-dimensional region of the subject's eye that does not contain artifacts that interfere with observation, analysis, or diagnosis. Furthermore, it is possible to provide an image of a three-dimensional region of the subject's eye that does not contain artifacts to subsequent processing. For example, it is possible to construct a three-dimensional image or a rendering image of the subject's eye based on a group of images that do not contain artifacts.

[0068] Even when images are obtained by photographing substantially the same position, the dimensions of the depicted predetermined region may differ between the image obtained by the left photographing system and the image obtained by the right photographing system. For example, the depicted corneal thickness, dimensions of inflammatory cells, and dimensions of anterior chamber flare may differ between the left and right images obtained by photographing substantially the same position with the left and right photographing systems, respectively. Even in such cases, the dimensions of the depicted predetermined region can be matched by using the image selection unit 1021.

[0069] The inflammation state information generating unit 1022 is configured to generate inflammation state information based on the Scheimpflug images selected by the selecting unit 1021. The number of Scheimpflug images used to generate the inflammation state information may be set arbitrarily. Furthermore, the inflammation state information generating unit 1022 may be configured to execute a process of generating processed image data by processing one or more Scheimpflug images included in the group of Scheimpflug images selected by the selecting unit 1021, and a process of generating inflammation state information based on the generated processed image data.

[0070] As described above, the inflammation state information generated by the data processing unit 1020 may include cell evaluation information, which is evaluation information regarding inflammatory cells in the anterior chamber of the subject's eye. In this case, the data processing unit 1020 may be configured to execute at least one of a first segmentation, a second segmentation, and a cell evaluation information generation process. Here, the first segmentation is a process for identifying an anterior chamber region corresponding to the anterior chamber, the second segmentation is a process for identifying a cellular region corresponding to inflammatory cells, and the cell evaluation information generation process is a process for generating cell evaluation information.

[0071] The first segmentation may be a machine learning-based process or a non-machine learning-based process, or a combination of machine learning and non-machine learning-based processes. The second segmentation may be a machine learning-based process or a non-machine learning-based process, or a combination of machine learning and non-machine learning-based processes. The cell evaluation information generation process may be a machine learning-based process or a non-machine learning-based process, or a combination of machine learning and non-machine learning-based processes.

[0072] The types of data input into the first segmentation, the second segmentation, and the cell evaluation information generation process may all be arbitrary. Below, several examples of possible combinations of multiple processes including the first segmentation, the second segmentation, and the cell evaluation information generation process will be described.

[0073] The data processing unit 1030 shown in Fig. 3 is an example of the configuration of the data processing unit 1020 in Fig. 1. The data processing unit 1030 in this example includes a first segmentation unit 1031, a second segmentation unit 1032, and a cell evaluation information generation processing unit 1033.

[0074] The first segmentation unit 1031 includes a processor that performs first segmentation to identify the anterior chamber region, and is configured to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

[0075] 4 shows an example configuration of the first segmentation unit 1031 when the first segmentation of this example is performed using machine learning. The first segmentation unit 1031A of this example is configured to perform the first segmentation using a pre-constructed inference model 1034 (referred to as the first inference model). The first inference model 1034 includes a neural network 1035 (referred to as the first neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by other modalities, or Scheimpflug images of the eye and eye images acquired by other modalities).

[0076] The data input to the first neural network 1035 is a Scheimpflug image, and the data output from the first neural network 1035 is an anterior chamber region. That is, the first segmentation unit 1031A is configured to receive a Scheimpflug image (e.g., one or more Scheimpflug images, one or more processed image data, or one or more Scheimpflug images and one or more processed image data) acquired by the image acquisition unit 1010, input the Scheimpflug image to the first neural network 1035 of the first inference model 1034, and acquire output data from the first neural network 1035 (the anterior chamber region in the input Scheimpflug image).

[0077] The device (inference model construction device) that constructs the first inference model 1034 may be provided in the ophthalmic device 1000, or may be provided in a peripheral device (such as a computer) of the ophthalmic device 1000, or may be another computer.

[0078] The model construction unit 2000 shown in FIG. 5 is an example of an inference model construction device, and includes a learning processing unit 2010 and a neural network 2020.

[0079] The neural network 2020 typically includes a convolutional neural network (CNN). Reference numeral 2030 in Figure 5 indicates an example of the structure of a convolutional neural network.

[0080] An image is input to the input layer. After the input layer, multiple pairs of convolutional layers and pooling layers are arranged. In the example shown in Figure 5, three pairs of convolutional layers and pooling layers are provided, but the number of pairs may be arbitrary.

[0081] The convolution layer performs convolution operations to extract features (such as contours) from an image. A convolution operation is a multiplication and accumulation operation of a filter function (weighting coefficients, filter kernel) of the same dimension as the input image on the input image. The convolution layer applies the convolution operation to multiple parts of the input image. More specifically, the convolution layer multiplies the value of each pixel in the partial image to which the filter function has been applied by the value (weight) of the filter function corresponding to that pixel to calculate the product, and then calculates the sum of the products across multiple pixels in this partial image. The resulting sum-of-products value is assigned to the corresponding pixel in the output image. By performing the multiplication and accumulation operation while shifting the location (partial image) to which the filter function is applied, the convolution operation result for the entire input image is obtained. This convolution operation generates multiple images in which various features have been extracted using multiple weighting coefficients. In other words, multiple filtered images, such as smoothed images and edge images, are obtained. The multiple images generated by the convolution layer are called feature maps.

[0082] The pooling layer compresses (e.g., thins out data) the feature map generated by the immediately preceding convolutional layer. More specifically, the pooling layer calculates statistical values ​​of predetermined neighboring pixels of a pixel of interest in the feature map at predetermined pixel intervals, and outputs an image with dimensions smaller than the input feature map. The statistical values ​​applied to the pooling operation are, for example, maximum values ​​(max pooling) or average values ​​(average pooling). The pixel interval applied to the pooling operation is called the stride.

[0083] A convolutional neural network can extract many features from an input image by processing it using multiple pairs of convolutional layers and pooling layers.

[0084] A fully connected layer is provided after the last pair of convolutional and pooling layers. In the example shown in Figure 5, two fully connected layers are provided, but any number of fully connected layers may be used. In the fully connected layer, features compressed by a combination of convolution and pooling are used to perform processes such as image classification, image segmentation, and regression. After the last fully connected layer, an output layer is provided to provide output results.

[0085] In some exemplary embodiments, the convolutional neural network may not include a fully connected layer (e.g., a fully convolutional network (FCN)), but may include a support vector machine, a recurrent neural network (RNN), or the like. Furthermore, the machine learning performed on the neural network 2020 may include transfer learning. That is, the neural network 2020 may include a neural network that has already been trained using other training data (training images) and whose parameters have been adjusted. Furthermore, the model construction unit 2000 (the learning processing unit 2010) may be configured to be able to apply fine tuning to the trained neural network (the neural network 2020). The neural network 2020 may be constructed using a known open-source neural network architecture.

[0086] The learning processing unit 2010 applies machine learning using training data to the neural network 2020. When the neural network 2020 includes a convolutional neural network, the parameters adjusted by the learning processing unit 2010 include, for example, filter coefficients of the convolutional layer and connection weights and offsets of the fully connected layer.

[0087] As described above, the training data may include one or more Scheimpflug images acquired for one or more eyes. Because the Scheimpflug images of the eyes are the same type of images as the images input to the first neural network 1035, the quality (accuracy, precision, etc.) of the output of the first neural network 1035 can be improved compared to when machine learning is performed using training data that includes only other types of images.

[0088] The types of images included in the training data are not limited to Scheimpflug images, and the training data may include, for example, images acquired by other ophthalmic modalities (fundus cameras, OCT devices, SLO, surgical microscopes, etc.), images acquired by imaging diagnostic modalities of any medical department (ultrasound diagnostic devices, X-ray diagnostic devices, X-ray CT devices, magnetic resonance imaging (MRI) devices, etc.), images generated by processing actual eye images (processed image data), pseudo images, etc. Furthermore, the number of images, etc. included in the training data may be increased using techniques such as data expansion and data augmentation.

[0089] The training method (machine learning method) for constructing the first neural network 1035 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of any two or more of these.

[0090] In some exemplary embodiments, supervised learning is performed using training data generated by annotation that labels input images. For example, this annotation involves identifying and labeling the anterior chamber region in each image included in the training data. The anterior chamber region is identified by, for example, at least one of a doctor, a computer, and another inference model. The learning processing unit 2010 can construct the first neural network 1035 by applying supervised learning using such training data to the neural network 2020.

[0091] The first inference model 1034 including the first neural network 1035 constructed in this manner is a trained model that receives a Scheimpflug image (e.g., a Scheimpflug image acquired by the image acquisition unit 1010, or its processed image data) as input and outputs the anterior chamber region in the input Scheimpflug image (e.g., information indicating the range or position of the anterior chamber region).

[0092] To avoid concentrating processing on specific units of the first neural network 1035, the learning processing unit 2010 may randomly select and disable some units of the neural network 2020 and perform learning using the remaining units (dropout).

[0093] The techniques used to build the inference model are not limited to the examples shown here. For example, any technique such as a support vector machine, a Bayesian classifier, boosting, k-means, kernel density estimation, principal component analysis, independent component analysis, self-organizing map, random forest, or generative adversarial network (GAN) can be used to build the inference model.

[0094] The first segmentation unit 1031A of this example uses such a first inference model 1034 (first neural network 1035) to perform processing to identify the anterior chamber region from the Scheimpflug image of the subject's eye.

[0095] The second segmentation unit 1032 includes a processor that performs second segmentation to identify a cellular region, and is configured to identify the cellular region from the anterior chamber region identified by the first segmentation unit 1031.

[0096] 6 shows an example of the configuration of the second segmentation unit 1032 when the second segmentation of this example is performed using machine learning. The second segmentation unit 1032A of this example is configured to perform the second segmentation using a pre-constructed inference model 1036 (referred to as the second inference model). The second inference model 1036 includes a neural network 1037 (referred to as the second neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by another modality, or Scheimpflug images of the eye and eye images acquired by another modality).

[0097] The eye images included in the training data in this example include images corresponding to at least a portion of the anterior chamber of the eye (referred to as an anterior chamber image). The eye images included in the training data in this example may include the results of manual or automatic segmentation of an anterior segment image (e.g., a Scheimpflug image or an image acquired by another modality), such as an anterior chamber image extracted from the anterior segment image, or information indicating the extent or location of the anterior chamber image within the anterior segment image.

[0098] The data input to the second neural network 1037 is the anterior chamber region identified from the Scheimpflug image by the first segmentation unit 1031 (or the identified and extracted anterior chamber region (the same applies below)), and the data output from the second neural network 1037 is a cellular region. That is, the second segmentation unit 1032A is configured to receive the anterior chamber region identified by the first segmentation unit 1031, input this anterior chamber region to the second neural network 1037 of the second inference model 1036, and obtain output data (cellular regions in the input anterior chamber region) from the second neural network 1037.

[0099] The construction of the second inference model 1036 (second neural network 1037) may be performed in the same manner as the construction of the first inference model 1034 (first neural network 1035). For example, the construction of the second inference model 1036 (second neural network 1037) is performed by the model construction unit 2000 shown in Figure 5. Unless otherwise specified, the model construction unit 2000 (learning processing unit 2010 and neural network 2020) in this example may be the same as that used in the construction of the first inference model 1034 (first neural network 1035).

[0100] The training data used to construct the second neural network 1037 may include one or more Scheimpflug images (e.g., anterior segment images with the anterior chamber region identified, anterior chamber images) acquired for one or more eyes. The types of images included in the training data are not limited to Scheimpflug images, and the training data may include, for example, images acquired by other ophthalmology modalities, images acquired by an imaging diagnostic modality of any medical department, images generated by processing actual eye images, pseudo images, etc.

[0101] The training method (machine learning method) for constructing the second neural network 1037 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of two or more of these.

[0102] In some exemplary embodiments, supervised learning is performed using training data generated by annotation that labels input images. For example, this annotation involves identifying and labeling cellular regions in each image included in the training data. The identification of cellular regions may be performed by, for example, at least one of a doctor, a computer, and / or another inference model. The learning processing unit 2010 can construct the second neural network 1037 by applying supervised learning using such training data to the neural network 2020.

[0103] The second inference model 1036 including the second neural network 1037 constructed in this manner is a trained model that receives the anterior chamber region identified by the first segmentation unit 1031 as input and outputs the cellular region (e.g., information indicating the range or position of the cellular region) within the input anterior chamber region.

[0104] The second segmentation unit 1032A of this example uses such a second inference model 1036 (second neural network 1037) to perform a process of identifying a cellular region from the anterior chamber region in the Scheimpflug image of the subject's eye.

[0105] The cell evaluation information generation processing unit 1033 includes a processor that executes cell evaluation information generation processing for generating cell evaluation information, and generates cell evaluation information from the cellular region identified by the second segmentation unit 1032.

[0106] 7 shows an example of the configuration of the cell evaluation information generation processing unit 1033 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1033A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1038 (referred to as the third inference model). The third inference model 1038 includes a neural network 1039 (referred to as the third neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by other modalities, or Scheimpflug images of the eye and eye images acquired by other modalities).

[0107] The ocular images included in the training data of this example include at least an anterior chamber image in which an image of inflammatory cells is depicted, and may further include an anterior chamber image in which an image of inflammatory cells is not depicted. The ocular images included in the training data of this example may include the results of manual or automatic segmentation of an anterior segment image (e.g., a Scheimpflug image or an image acquired by another modality), and may be, for example, a cell image extracted from the anterior chamber image in the anterior segment image, or information indicating the range or location of a cellular region in the anterior chamber image.

[0108] The data input to the third neural network 1039 is the output from the second segmentation unit 1032 or data based thereon (e.g., data indicating the range, position, distribution, etc. of the cellular region, or the anterior chamber region accompanied by the identification result of the cellular region), and the data output from the third neural network 1039 is cell evaluation information. That is, the cell evaluation information generation processing unit 1033A is configured to receive the identification result of the cellular region by the second segmentation unit 1032 or data based thereon, input the identification result of the cellular region or data based thereon to the third neural network 1039 of the third inference model 1038, and acquire output data (cell evaluation information) from the third neural network 1037. As described above, the cell evaluation information is evaluation information regarding predetermined parameters related to inflammatory cells (e.g., the density, number, position, distribution, etc. of the inflammatory cells).

[0109] The construction of the third inference model 1038 (third neural network 1039) may be performed in the same manner as the construction of the first inference model 1034 (first neural network 1035). For example, the construction of the third inference model 1038 (third neural network 1039) is performed by the model construction unit 2000 shown in Figure 5. Unless otherwise specified, the model construction unit 2000 (learning processing unit 2010 and neural network 2020) in this example may be the same as that used in the construction of the first inference model 1034 (first neural network 1035).

[0110] The training data used to construct the third neural network 1039 may include one or more Scheimpflug images acquired for one or more eyes (e.g., an anterior segment image including an anterior chamber region in which a cellular region is identified, an anterior chamber image in which a cellular region is identified). The types of images included in the training data are not limited to Scheimpflug images, and the training data may include, for example, images acquired by other ophthalmology modalities, images acquired by an imaging diagnostic modality of any medical department, images generated by processing actual eye images, pseudo images, etc.

[0111] The training method (machine learning method) for constructing the third neural network 1039 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of any two or more of these.

[0112] In some exemplary embodiments, supervised learning is performed using training data generated by annotation that labels input images. For example, in this annotation, each image (in which a cellular region is identified) included in the training data is labeled with cell evaluation information generated from the image. The generation of cell evaluation information from the image is performed by, for example, at least one of a doctor, a computer, and another inference model. The learning processing unit 2010 can construct the third neural network 1039 by applying supervised learning using such training data to the neural network 2020.

[0113] The third inference model 1038, including the third neural network 1039 constructed in this manner, is a trained model that takes the cell area identification results by the second segmentation unit 1032 or data based thereon as input, and outputs cell evaluation information based on the input cell area identification results or data based thereon.

[0114] The cell evaluation information generation processing unit 1033A in this example uses such a third inference model 1038 (third neural network 1039) to perform a process of generating cell evaluation information from a cell area in the anterior chamber region in a Scheimpflug image of the test eye.

[0115] The data processing unit 1040 shown in Fig. 8 is an example of the configuration of the data processing unit 1020 in Fig. 1. The data processing unit 1040 in this example includes a first segmentation unit 1041, a conversion processing unit 1042, a second segmentation unit 1043, and a cell evaluation information generation processing unit 1044.

[0116] The first segmentation unit 1041 has the same configuration and function as the first segmentation unit 1031 in FIG. 3 (for example, the first segmentation unit 1031A in FIG. 4), and is configured to perform first segmentation to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

[0117] The conversion processing unit 1042 converts the anterior chamber region identified by the first segmentation unit 1041 into data with a structure corresponding to the second segmentation performed by the second segmentation unit 1043. The second segmentation unit 1043 in this example is configured to perform the second segmentation using a neural network (second neural network) constructed by machine learning, like the second segmentation unit 1032A in FIG. 6. The conversion processing unit 1042 is configured to perform conversion processing for converting the anterior chamber region identified from the Scheimpflug image by the first segmentation unit 1041 into image data with a structure corresponding to the input layer of the second neural network of the second segmentation unit 1043.

[0118] For example, the input layer of the second neural network (convolutional neural network) of the second segmentation unit 1043 may be configured to accept data of a predetermined structure (shape, format). This predetermined data structure may be, for example, a predetermined image size (e.g., the number of vertical and horizontal pixels) or a predetermined image shape (e.g., square or rectangular). However, the image size and shape of the anterior chamber region identified by the first segmentation unit 1041 vary depending on the specifications of the ophthalmic apparatus, the conditions and settings at the time of imaging, and individual differences in the size and shape of the subject's eye. The conversion processing unit 1042 converts the structure of the anterior chamber region identified by the first segmentation unit 1041 (e.g., image size and / or image shape) into a structure that can be accepted by the input layer of the second neural network of the second segmentation unit 1043.

[0119] The image size conversion may be performed using any known image size conversion technique, and may include, for example, a process of dividing the anterior chamber region identified by the first segmentation unit 1041 into multiple partial images having image sizes according to the input layer, or a process of resizing the anterior chamber region identified by the first segmentation unit 1041 into a single image having an image size according to the input layer. The image shape conversion may be performed using any known image deformation technique. The same applies to conversion processes of other data structures.

[0120] In this disclosure, several examples are described in detail for applying conversion processing corresponding to the structure of a neural network to the anterior chamber region of a Scheimpflug image, but the manner of conversion processing and the configuration therefor are not limited to these.

[0121] For example, if the image input to the neural network is a Scheimpflug image, a configuration can be adopted in which a similar conversion process is applied to the input Scheimpflug image.Also, if the image input to the neural network is any processed image data of a Scheimpflug image, a configuration can be adopted in which a similar conversion process is applied to the input processed image data.

[0122] Furthermore, the arrangement of the element that performs the conversion process (the processor that performs the conversion process, referred to as the conversion processing unit) may also be arbitrary. For example, the conversion processing unit may be arranged in a stage before the target neural network in the flow of a series of processes performed based on the acquired Scheimpflug image (e.g., a stage before the inference model including this neural network, or a stage inside this inference model but before this neural network), or may be arranged inside the target neural network. When arranged inside the target neural network, the conversion processing unit is arranged in a stage before the input layer that receives input that directly corresponds to the output of this neural network.

[0123] The second segmentation unit 1043 has the same configuration and function as the second segmentation unit 1032A in Fig. 6, and is configured to perform second segmentation to identify a cellular region from the anterior chamber region whose data structure has been converted by the conversion processing unit 1042. The second neural network of the second segmentation unit 1043 is configured to receive input of the image data (the anterior chamber region whose data structure has been converted) generated by the conversion processing unit 1042 and output a cellular region. The machine learning for constructing the second neural network of this example may be performed in the same manner as the machine learning for constructing the second neural network 1037 in Fig. 6.

[0124] The cell evaluation information generation processing unit 1044 has the same configuration and function as the cell evaluation information generation processing unit 1033 of Figure 3 (for example, the cell evaluation information generation processing unit 1033A of Figure 7), and is configured to execute cell evaluation information generation processing for generating cell evaluation information from the cell region identified by the second segmentation unit 1043.

[0125] In this way, the data processing unit 1040 of this example may have a configuration in which the conversion processing unit 1042 is disposed between the first segmentation unit 1031 and the second segmentation unit 1032 of the data processing unit 1030 in Fig. 3. However, the configuration of the data processing unit 1040 of this example is not limited to this.

[0126] The data processing unit 1050 shown in Fig. 9 is an example of the configuration of the data processing unit 1020 in Fig. 1. The data processing unit 1050 of this example includes a second segmentation unit 1051 and a cell evaluation information generation processing unit 1052.

[0127] The second segmentation unit 1051 includes a processor that performs second segmentation to identify cellular regions, and is configured to identify cellular regions from the Scheimpflug image acquired by the image acquisition unit 1010.

[0128] 10 shows an example of the configuration of the second segmentation unit 1051 when the second segmentation of this example is performed using machine learning. The second segmentation unit 1051A of this example is configured to perform the second segmentation using a pre-constructed inference model 1053 (referred to as a fourth inference model). The fourth inference model 1053 includes a neural network 1054 (referred to as a fourth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by other modalities, or Scheimpflug images of the eye and eye images acquired by other modalities).

[0129] In some exemplary embodiments, the fourth neural network 1054 may include at least a portion of the first neural network 1035 of Figure 4 and at least a portion of the second neural network 1037 of Figure 6. For example, the fourth neural network 1054 may be a neural network in which the first neural network 1035 and the second neural network 1037 are arranged in series. The fourth neural network 1054 configured in this manner has a function of identifying an anterior chamber region from a Scheimpflug image and a function of identifying a cellular region from the anterior chamber region.

[0130] In some exemplary embodiments, the fourth neural network 1054 may be configured with machine learning to directly identify cellular regions from a Scheimpflug image without first identifying the anterior chamber region. The fourth neural network 1054 is not limited to these embodiments and may include any machine learning-based neural network for identifying cellular regions from a Scheimpflug image.

[0131] The data input to the fourth neural network 1054 is a Scheimpflug image, and the data output from the fourth neural network 1054 is a cell region. That is, the second segmentation unit 1051A is configured to receive a Scheimpflug image, input the Scheimpflug image to the fourth neural network 1054 of the fourth inference model 1053, and obtain output data from the fourth neural network 1054 (cell regions in the input Scheimpflug image).

[0132] The construction of the fourth inference model 1053 (fourth neural network 1054) may be performed in the same manner as the construction of the first inference model 1034 (first neural network 1035). For example, the construction of the fourth inference model 1053 (fourth neural network 1054) is performed by the model construction unit 2000 shown in Figure 5. Unless otherwise specified, the model construction unit 2000 (learning processing unit 2010 and neural network 2020) in this example may be the same as that used in the construction of the first inference model 1034 (first neural network 1035).

[0133] The training data used to construct the fourth neural network 1054 may include one or more Scheimpflug images acquired for one or more eyes. The types of images included in the training data are not limited to Scheimpflug images, and the training data may include, for example, images acquired by other ophthalmological modalities, images acquired by an imaging diagnostic modality of any medical department, images generated by processing actual eye images, pseudo-images, etc.

[0134] Any images included in the training data may be annotated with information to aid in the processing performed by the fourth neural network 1054. For example, the anterior chamber region in the image may be labeled by prior annotation.

[0135] The training method (machine learning method) for constructing the fourth neural network 1054 may be any method, such as supervised learning, unsupervised learning, or reinforcement learning, or a combination of two or more of these.

[0136] In some exemplary embodiments, supervised learning is performed using training data generated by annotation that labels input images. For example, this annotation involves identifying and labeling cellular regions in each image included in the training data. The identification of cellular regions may be performed by, for example, at least one of a doctor, a computer, and / or another inference model. The learning processing unit 2010 can construct the fourth neural network 1054 by applying supervised learning using such training data to the neural network 2020.

[0137] The fourth inference model 1053, which includes the fourth neural network 1054 constructed in this manner, is a trained model that takes as input the Scheimpflug image (or its processed image data, etc.) acquired by the image acquisition unit 1010 and outputs the cellular area (e.g., information indicating the range or position of the cellular area) in the input Scheimpflug image.

[0138] The second segmentation unit 1051A of the present example uses such a fourth inference model 1053 (fourth neural network 1054) to perform processing to identify a cellular region from a Scheimpflug image of the subject's eye.

[0139] The cell evaluation information generation processing unit 1052 includes a processor that executes cell evaluation information generation processing for generating cell evaluation information, and generates cell evaluation information from the cellular region identified by the second segmentation unit 1051.

[0140] 11 shows an example of the configuration of the cell evaluation information generation processing unit 1052 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1052A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1055 (referred to as the fifth inference model). The fifth inference model 1055 includes a neural network 1056 (referred to as the fifth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by other modalities, or Scheimpflug images of the eye and eye images acquired by other modalities).

[0141] The data input to the fifth neural network 1056 is the output from the second segmentation unit 1051 or data based thereon (for example, data indicating the range, position, distribution, etc. of the cellular region, or the anterior chamber region accompanied by the identification result of the cellular region), and the data output from the fifth neural network 1056 is cell evaluation information. That is, the cell evaluation information generation processing unit 1052A is configured to receive the identification result of the cellular region by the second segmentation unit 1051 or data based thereon, input the identification result of the cellular region or data based thereon to the fifth neural network 1056 of the fifth inference model 1055, and acquire output data (cell evaluation information) from the fifth neural network 1056.

[0142] The machine learning technique for constructing the fifth inference model 1055 (fifth neural network 1056) may be the same as the machine learning technique for constructing the third neural network 1039 of the cell evaluation information generation processing unit 1033 in Figure 7. Furthermore, the training data used in the machine learning for constructing the fifth inference model 1055 (fifth neural network 1056) may be the same as the training data used in the machine learning for constructing the third neural network 1039.

[0143] The fifth inference model 1055, which includes the fifth neural network 1056, is a trained model that receives as input the results of cell area identification by the second segmentation unit 1051 or data based thereon, and outputs cell evaluation information based on the input results of cell area identification or data based thereon.

[0144] The cell evaluation information generation processing unit 1052A of this example uses such a fifth inference model 1055 (fifth neural network 1056) to perform a process of generating cell evaluation information from a cellular region in a Scheimpflug image of the subject's eye.

[0145] The data processing unit 1060 shown in Fig. 12 is an example of the configuration of the data processing unit 1020 in Fig. 1. The data processing unit 1060 of this example includes a cell evaluation information generation processing unit 1061.

[0146] The cell evaluation information generation processing unit 1061 includes a processor that executes cell evaluation information generation processing for generating cell evaluation information, and generates cell evaluation information from the Scheimpflug image acquired by the image acquisition unit 1010.

[0147] 13 shows an example of the configuration of the cell evaluation information generation processing unit 1061 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1061A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1062 (referred to as the sixth inference model). The sixth inference model 1062 includes a neural network 1063 (referred to as the sixth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by other modalities, or Scheimpflug images of the eye and eye images acquired by other modalities).

[0148] In some exemplary embodiments, the sixth neural network 1063 may include at least a portion of the first neural network 1035 of Figure 4, at least a portion of the second neural network 1037 of Figure 6, and at least a portion of the third neural network 1039 of Figure 7. For example, the sixth neural network 1063 may be a neural network in which the first neural network 1035, the second neural network 1037, and the third neural network 1039 are arranged in series. The sixth neural network 1063 configured in this manner has a function of identifying an anterior chamber region from a Scheimpflug image, a function of identifying a cellular region from the anterior chamber region, and a function of generating cellular evaluation information from the cellular region.

[0149] In some exemplary embodiments, the sixth neural network 1063 may be configured to generate cell evaluation information directly from a Scheimpflug image without identifying an anterior chamber region and / or a cellular region. The sixth neural network 1063 is not limited to these embodiments and may include any machine learning-based neural network for identifying cell evaluation information from a Scheimpflug image.

[0150] The data input to the sixth neural network 1063 is the output from the image acquisition unit 1010 or data based thereon, and the data output from the sixth neural network 1063 is cell evaluation information. That is, the cell evaluation information generation processing unit 1061A is configured to receive the Scheimpflug image (and / or data based on this Scheimpflug image) acquired by the image acquisition unit 1010, input this Scheimpflug image or data based thereon to the sixth neural network 1063 of the sixth inference model 1062, and acquire output data (cell evaluation information) from the sixth neural network 1063.

[0151] The machine learning technique for constructing the sixth inference model 1062 (sixth neural network 1063) may be the same as the machine learning technique for constructing the third neural network 1039 of the cell evaluation information generation processing unit 1033 in Figure 7. Furthermore, the training data used in the machine learning for constructing the sixth inference model 1062 (sixth neural network 1063) may be the same as the training data used in the machine learning for constructing the third neural network 1039.

[0152] The sixth inference model 1062, which includes the sixth neural network 1063, is a trained model that receives as input a Scheimpflug image (and / or data based on this Scheimpflug image) acquired by the image acquisition unit 1010, and outputs cell evaluation information based on the input Scheimpflug image (and / or data based on this Scheimpflug image).

[0153] The cell evaluation information generation processing unit 1061A in this example uses such a sixth inference model 1062 (sixth neural network 1063) to perform a process of generating cell evaluation information from a Scheimpflug image of the test eye (and / or data based on this Scheimpflug image).

[0154] A data processing unit 1070 shown in Fig. 14 is an example of the configuration of the data processing unit 1020 in Fig. 1. The data processing unit 1070 of this example includes a first segmentation unit 1071 and a cell evaluation information generation processing unit 1072.

[0155] The first segmentation unit 1071 includes a processor that performs first segmentation to identify the anterior chamber region, and is configured to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

[0156] 15 shows an example of the configuration of the first segmentation unit 1071 when the first segmentation of this example is performed using machine learning. The first segmentation unit 1071A of this example is configured to perform the first segmentation using a pre-constructed inference model 1073 (referred to as a seventh inference model). The seventh inference model 1073 includes a neural network 1074 (referred to as a seventh neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by other modalities, or Scheimpflug images of the eye and eye images acquired by other modalities).

[0157] The machine learning technique for constructing the seventh inference model 1073 (seventh neural network 1074) may be similar to the machine learning technique for constructing the first neural network 1035 of the first segmentation unit 1031A in FIG. 4 . Furthermore, the training data used in the machine learning for constructing the seventh inference model 1073 (seventh neural network 1074) may be similar to the training data used in the machine learning for constructing the first neural network 1035. In some exemplary embodiments, the seventh neural network 1074 may be the same as or similar to the first neural network 1035, and the seventh inference model 1073 may be the same as or similar to the first inference model 1034.

[0158] The seventh inference model 1073 including the seventh neural network 1074 is a trained model that receives as input the Scheimpflug image (or its processed image data, etc.) acquired by the image acquisition unit 1010 and outputs the anterior chamber region in the input Scheimpflug image (e.g., information indicating the range or position of the anterior chamber region).

[0159] The first segmentation unit 1071A of the present example uses such a seventh inference model 1073 (seventh neural network 1074) to perform processing to identify the anterior chamber region from the Scheimpflug image of the subject's eye.

[0160] The cell evaluation information generation processing unit 1072 includes a processor that executes cell evaluation information generation processing for generating cell evaluation information, and generates cell evaluation information from the anterior chamber region identified by the first segmentation unit 1071.

[0161] 16 shows an example of the configuration of the cell evaluation information generation processing unit 1072 when the cell evaluation information generation processing of this example is performed using machine learning. The cell evaluation information generation processing unit 1072A of this example is configured to perform the cell evaluation information generation processing using a pre-constructed inference model 1075 (referred to as the eighth inference model). The eighth inference model 1075 includes a neural network 1076 (referred to as the eighth neural network) constructed by machine learning using training data including at least eye images (e.g., Scheimpflug images of the eye, eye images acquired by other modalities, or Scheimpflug images of the eye and eye images acquired by other modalities).

[0162] In some exemplary embodiments, the eighth neural network 1076 may include at least a portion of the second neural network 1037 of Figure 6 and at least a portion of the third neural network 1039 of Figure 7. For example, the eighth neural network 1076 may be a neural network in which the second neural network 1037 and the third neural network 1039 are arranged in series. The eighth neural network 1076 configured in this manner has a function of identifying a cellular region from the anterior chamber region and a function of generating cellular evaluation information from the cellular region.

[0163] In some exemplary embodiments, the eighth neural network 1076 may be configured with machine learning to generate cell-evaluation information directly from the anterior chamber region without identifying the cellular region. The eighth neural network 1076 may include any machine learning neural network for identifying cell-evaluation information from the anterior chamber region.

[0164] The data input to the eighth neural network 1076 is the output from the first segmentation unit 1071 or data based thereon, and the data output from the eighth neural network 1076 is cell evaluation information. That is, the cell evaluation information generation processing unit 1072A is configured to receive the anterior chamber region (and / or data based on this anterior chamber region) identified from the Scheimpflug image by the first segmentation unit 1071, input this anterior chamber region or data based thereon to the eighth neural network 1076 of the eighth inference model 1075, and acquire output data (cell evaluation information) from the eighth neural network 1076.

[0165] The machine learning technique for constructing the eighth inference model 1075 (eighth neural network 1076) may be the same as the machine learning technique for constructing the third neural network 1039 of the cell evaluation information generation processing unit 1033 in Figure 7. Furthermore, the training data used in the machine learning for constructing the eighth inference model 1075 (eighth neural network 1076) may be the same as the training data used in the machine learning for constructing the third neural network 1039.

[0166] The eighth inference model 1075 including the eighth neural network 1076 is a trained model that receives the anterior chamber region (and / or data based on this anterior chamber region) identified by the first segmentation unit 1071 as input and outputs cell evaluation information based on the input anterior chamber region (and / or data based on this anterior chamber region).

[0167] The cell evaluation information generation processing unit 1072A in this example uses such an eighth inference model 1075 (eighth neural network 1076) to perform a process of generating cell evaluation information from the anterior chamber region (and / or data based on this anterior chamber region) in the Scheimpflug image of the test eye.

[0168] A data processing unit 1080 shown in Fig. 17 is an example of the configuration of the data processing unit 1020 in Fig. 1. The data processing unit 1080 of this example includes a first segmentation unit 1081, a conversion processing unit 1082, and a cell evaluation information generation processing unit 1083.

[0169] The first segmentation unit 1081 has the same configuration and function as the first segmentation unit 1031 in FIG. 3 (for example, the first segmentation unit 1031A in FIG. 4), and is configured to perform first segmentation to identify the anterior chamber region from the Scheimpflug image acquired by the image acquisition unit 1010.

[0170] The conversion processing unit 1082 converts the anterior chamber region identified by the first segmentation unit 1081 into data with a structure corresponding to the cell evaluation information generation processing executed by the cell evaluation information generation processing unit 1083. The cell evaluation information generation processing unit 1083 of this example is configured to execute the cell evaluation information generation processing using a neural network (eighth neural network) constructed by machine learning, like the cell evaluation information generation processing unit 1072A in Fig. 16. The conversion processing unit 1082 is configured to execute conversion processing for converting the anterior chamber region identified from the Scheimpflug image by the first segmentation unit 1081 into image data with a structure corresponding to the input layer of the eighth neural network of the cell evaluation information generation processing unit 1083.

[0171] For example, the input layer of the eighth neural network (convolutional neural network) of the cell evaluation information generation processing unit 1083 may be configured to accept data of a predetermined structure (shape, format). This predetermined data structure may be, for example, a predetermined image size (e.g., the number of vertical and horizontal pixels) or a predetermined image shape (e.g., square or rectangular). However, the image size and shape of the anterior chamber region identified by the first segmentation unit 1081 vary depending on the specifications of the ophthalmic apparatus, the conditions and settings at the time of imaging, and individual differences in the size and shape of the subject's eye. The conversion processing unit 1082 converts the structure of the anterior chamber region identified by the first segmentation unit 1081 (e.g., image size and / or image shape) into a structure that can be accepted by the input layer of the eighth neural network of the cell evaluation information generation processing unit 1083.

[0172] The image size conversion may be performed using any known image size conversion technique, and may include, for example, a process of dividing the anterior chamber region identified by the first segmentation unit 1081 into multiple partial images having image sizes according to the input layer, or a process of resizing the anterior chamber region identified by the first segmentation unit 1081 into a single image having an image size according to the input layer. The image shape conversion may be performed using any known image deformation technique. The same applies to conversion processes of other data structures.

[0173] The cell evaluation information generation processing unit 1083 has the same configuration and function as the cell evaluation information generation processing unit 1072 in Figure 14 (for example, the cell evaluation information generation processing unit 1072A in Figure 16), and is configured to execute cell evaluation information generation processing for generating cell evaluation information from data obtained by processing the anterior chamber region identified by the first segmentation unit 1081 using the conversion processing unit 1082.

[0174] In this way, the data processing unit 1080 of this example may have a configuration in which the conversion processing unit 1082 is disposed between the first segmentation unit 1071 and the cell evaluation information generation processing unit 1072 of the data processing unit 1070 in Fig. 14. However, the configuration of the data processing unit 1080 of this example is not limited to this.

[0175] So far, several examples of the data processing unit 1020 including an inference model (neural network) constructed using machine learning have been mainly described. However, the data processing unit 1020 is not limited to such a machine learning-based configuration. The data processing unit 1020 according to the present disclosure may be implemented solely by a machine learning-based configuration, may be implemented by a combination of a machine learning-based configuration and a non-machine learning-based configuration, or may be implemented solely by a non-machine learning-based configuration.

[0176] Below, we will explain some examples of the data processing unit 1020 that has only a non-machine learning based configuration. A person skilled in the art would be able to understand aspects of the data processing unit 1020 that combines a machine learning based configuration and a non-machine learning based configuration based on some examples of the machine learning based configuration described above and some examples of the data processing unit 1020 that has only a non-machine learning based configuration described below.

[0177] 18 is an example of the configuration of the data processing unit 1020 in FIG. 1, and has a non-machine learning-based configuration. The data processing unit 1090 in this example includes a first analysis processing unit 1091, a second analysis processing unit 1092, and a third analysis processing unit 1093.

[0178] The first analysis processing unit 1091 includes a processor that performs first segmentation to identify the anterior chamber region, and is configured to apply a predetermined analysis process (referred to as the first analysis process) to the Scheimpflug image (and / or its processed image data) acquired by the image acquisition unit 1010, to identify the anterior chamber region in this Scheimpflug image.

[0179] The first analysis process may include any known segmentation for identifying the anterior chamber region in the Scheimpflug image. For example, the segmentation for identifying the anterior chamber region includes a segmentation for identifying an image region corresponding to the cornea (particularly, the posterior surface of the cornea) and a segmentation for identifying an image region corresponding to the lens (particularly, the anterior surface of the lens). The image region corresponding to the cornea is referred to as the corneal region, the image region corresponding to the posterior surface of the cornea is referred to as the posterior corneal region, the image region corresponding to the lens is referred to as the lens region, and the image region corresponding to the anterior surface of the lens is referred to as the anterior lens region.

[0180] The segmentation of the posterior corneal surface region may include any known segmentation method. In the segmentation of the posterior corneal surface region, artifacts in the Scheimpflug image and saturation of pixel values ​​can be problematic. To solve these problems, for example, the configuration shown in FIG. 2C can be employed. That is, by combining the imaging method using the first imaging system 1014 and the second imaging system 1015 with the Scheimpflug image selection method using the image selection unit 1021, a Scheimpflug image free of artifacts and saturation can be selected, and the posterior corneal surface region can be identified from this Scheimpflug image.

[0181] The segmentation of the anterior surface of the lens may include any known segmentation method. In the segmentation of the anterior surface of the lens, a problem arises, for example, in that the representation of the Scheimpflug image (the appearance of the Scheimpflug image) changes depending on the state of the pupil of the subject's eye (e.g., mydriatic state, non-mydriatic state, small pupil eye, etc.). For example, when the subject's eye is non-mydriatic or has a small pupil, the imaged area of ​​the lens is smaller than when the subject's eye is dilated. To solve this problem, a process for uniforming the representation of the Scheimpflug image can be applied, such as a process for estimating the position and shape of the unimaged portion of the anterior surface of the lens (the portion covered by the pupil) based on the anterior surface of the lens depicted in the Scheimpflug image. The process for uniforming the representation of the Scheimpflug image may be performed based on machine learning or non-machine learning. Furthermore, the process for estimating the position and shape of the anterior surface of the lens may include, for example, any known extrapolation process.

[0182] When the image acquisition unit 1010 collects a series of Scheimpflug images by slit scanning, some images may contain problems and some may not, or the degree of problems may vary among the images. For example, some images may contain artifacts or saturation and some may not, or some images may contain artifacts of various states (e.g., position, size, shape, etc.). These phenomena may adversely affect the quality (e.g., stability, robustness, reproducibility, accuracy, precision, etc.) of the processing performed by the data processing unit 1090. In some exemplary embodiments, measures may be taken to prevent these phenomena from occurring or to reduce the adverse effects caused by these phenomena. An example of the former measure is to combine an imaging method using the first imaging system 1014 and the second imaging system 1015 with a Scheimpflug image selection method using the image selection unit 1021. Examples of the latter measure include image correction, noise removal, noise reduction, and image parameter adjustment.

[0183] The second analysis processing unit 1092 includes a processor that performs second segmentation to identify a cellular region, and is configured to apply the second analysis processing to the anterior chamber region identified from the Scheimpflug image by the first analysis processing unit 1091 to identify the cellular region.

[0184] In some exemplary embodiments, the second analysis processing unit 1092 may be configured to identify a cellular region based on the value of each pixel in the anterior chamber region (e.g., at least one of a brightness value, an R value, a G value, and a B value). In some exemplary embodiments, the second analysis processing unit 1092 may be configured to apply segmentation to the anterior chamber region to identify a cellular region. This segmentation is performed, for example, according to a program created based on the standard morphology (e.g., size, shape, etc.) of inflammatory cells (cellular regions). In some exemplary embodiments, the second analysis processing unit 1092 may be configured to identify a cellular region by at least a partial combination of these two techniques.

[0185] Countermeasures that can be taken when the image acquisition unit 1010 collects a series of Scheimpflug images by slit scanning may be similar to those taken by the first analysis processing unit 1091. Furthermore, taking into consideration that a cellular region is generally a minute image region, countermeasures may be taken to distinguish between a cellular region and minute artifacts. For example, by performing processing to remove artifacts (such as ghosts), it is possible to prevent erroneous detection of artifacts in the detection of a cellular region.

[0186] The third analysis processing unit 1093 includes a processor that executes a cell evaluation information generation process for generating cell evaluation information, and is configured to apply the third analysis process to a cell region identified from the anterior chamber region of the Scheimpflug image by the second analysis processing unit 1092 to generate cell evaluation information.

[0187] As mentioned above, the cell evaluation information may be any evaluation information regarding inflammatory cells, and may include, for example, information representing the state of inflammatory cells (e.g., any parameters such as density, number, position, distribution, etc.), or may include evaluation information generated based on information on specified parameters regarding the state of inflammatory cells.

[0188] In some exemplary embodiments, the third analysis processing unit 1093 can determine the density, number, location, distribution, etc. of one or more cellular regions identified by the second analysis processing unit 1092.

[0189] The process of determining the density of inflammatory cells includes, for example, a process of setting an image region of a predetermined size (e.g., an image region of 1 mm square) and a process of counting the number of cellular regions detected by the second analysis processing unit 1092 in the set image region. Here, the size of the image region (e.g., a dimension in real space, such as "1 mm") is defined, for example, based on the specifications of the optical system of the ophthalmic apparatus 1000 (e.g., design data of the optical system and / or actual measurement data of the optical system), and is typically defined as a correspondence relationship between pixels and dimensions in real space (e.g., dot pitch). The cell evaluation information may include information on the density of inflammatory cells determined in this manner, or may include evaluation information obtained from this density information. This evaluation information may include, for example, an evaluation result using the classification criteria for uveitis diseases proposed by the SUN Working Group. This classification standard defines grades according to the number of inflammatory cells present in one field of view (a field of view measuring 1 mm square) (i.e., the density (concentration) of inflammatory cells), with grade "0" being defined as less than 1 cell, grade "0.5+" as 1 to 5 cells, grade "1+" as 6 to 15 cells, grade "2+" as 16 to 25 cells, grade "3+" as 26 to 50 cells, and grade "4+" as 50 or more cells. The grade divisions in this classification standard may be more detailed or coarse. Cell evaluation information may also be generated based on other classification standards.

[0190] In some exemplary embodiments, the data processing unit 1090 (third analysis processing unit 1093) may be configured to execute the following processes: identifying a partial region of the anterior chamber region (e.g., a 1-millimeter square image region) identified from the Scheimpflug image by the first analysis processing unit 1091; determining the number of cellular regions belonging to this partial region; and calculating the density of inflammatory cells based on the number and the dimensions of the partial region. Here, the data processing unit 1090 may be configured to select cellular regions located within this partial region from the cellular regions detected from the entire anterior chamber region by the second analysis processing unit 1092, and to calculate the density based on the selected cellular regions by the third analysis processing unit 1093. Alternatively, the data processing unit 1090 may be configured to analyze the partial region to identify cellular regions by the second analysis processing unit 1092, and to calculate the density based on the cellular regions identified from the partial region by the third analysis processing unit 1093.

[0191] The process of determining the number of inflammatory cells includes, for example, a process of counting the number of cellular regions detected by the second analysis processing unit 1092. The cell evaluation information may include information on the number of inflammatory cells determined in this manner, or may include evaluation information obtained from this number information. For example, the cell evaluation information can determine the average density of inflammatory cells in the entire anterior chamber region by dividing the number of cellular regions detected in the entire anterior chamber region by the dimensions of the anterior chamber region (e.g., area, volume, etc.). Furthermore, the cell evaluation information may include an evaluation result (e.g., grade) based on the number of cellular regions detected in the entire anterior chamber region, or may include the number of cellular regions in a partial region of the anterior chamber region and / or an evaluation result based thereon.

[0192] The process of determining the position of the inflammatory cells may include, for example, a process of identifying the position of the cellular region detected by the second analysis processing unit 1092. The position of the cellular region may be expressed, for example, as coordinates in a defined coordinate system of the Scheimpflug image, or as a relative position (e.g., distance, direction, etc.) to a predetermined image region (reference region) depicted in the Scheimpflug image. This reference region may be, for example, the corneal region, the posterior corneal region, the lens region, the anterior lens region, or an image region corresponding to the axis of the eye (e.g., a straight line connecting the vertex position of the cornea and the vertex position of the anterior lens). The cell evaluation information may include information on the position of the inflammatory cells determined in this manner, or may include evaluation information obtained from this position information. For example, the cell evaluation information may include information representing the distribution of inflammatory cells (distribution of multiple cell regions), or may include an evaluation result (e.g., grade) based on the positions of one or more inflammatory cells, or may include an evaluation result (e.g., grade) based on the positions (distribution) of multiple inflammatory cells.

[0193] The operation of the ophthalmologic apparatus 1000 will be described. Note that the operation described below is merely an example. For example, any matter related to the present disclosure, any matter related to the documents cited in the present disclosure, any matter related to the technical field to which the embodiments of the present disclosure belong, any matter related to the technical field related to the embodiments of the present disclosure, etc. can be combined with the following example operation.

[0194] 19 shows a first operation example of the ophthalmic apparatus 1000. It is assumed that various operations (preparatory operations) have been completed before imaging by the ophthalmic apparatus 1000. The preparatory operations include adjustment of the table on which the ophthalmic apparatus 1000 is placed, adjustment of the chair used by the subject, adjustment of the face rest (chin rest, forehead rest, etc.) of the ophthalmic apparatus 1000, alignment of the ophthalmic apparatus 1000 with respect to the subject's eye, adjustment of the slit light (for example, adjustment of light intensity, width, length, and direction), etc.

[0195] Upon receiving an instruction to start imaging, the ophthalmologic apparatus 1000 acquires a Scheimpflug image of the subject's eye by the image acquisition unit 1010 (S1).

[0196] Furthermore, the ophthalmologic apparatus 1000 generates inflammation state information indicating the inflammation state of the subject's eye based on the Scheimpflug image acquired in step S1 using the data processing unit 1020 (S2).

[0197] The number of Scheimpflug images acquired in step S1 of this operation example may be preset, and may be one, or may be two or more (for example, a series of Scheimpflug images collected by slit scanning). In step S2, one of the various data processing methods described above is performed depending on the number of Scheimpflug images acquired in step S1. This data processing may be machine learning-based processing, non-machine learning-based processing, or a combination of machine learning-based processing and non-machine learning-based processing.

[0198] When the ophthalmic apparatus 1000 includes two or more imaging systems, it is possible to process multiple Scheimpflug images acquired by these imaging systems. For example, when the ophthalmic apparatus 1000 includes a first imaging system 1014 and a second imaging system 1015 as shown in FIG. 2C , it is possible to process two or more Scheimpflug images acquired by the first imaging system 1014 and the second imaging system 1015 by a data processing unit 1020A (an image selection unit 1021, an inflammation state information generation unit 1022). The data processing performed by the data processing unit 1020A may be machine learning-based processing, non-machine learning-based processing, or a combination of machine learning-based processing and non-machine learning-based processing.

[0199] The ophthalmic apparatus 1000 can display the Scheimpflug image acquired in step S1 and / or the inflammation state information generated in step S2 on a display device. The display device may be a component of the ophthalmic apparatus 1000 or an external device connected to the ophthalmic apparatus 1000.

[0200] Below, several examples of information display that can be performed by the ophthalmologic apparatus 1000 will be described. The manner of information display is not limited to these examples. At least two of these examples can be at least partially combined.

[0201] In a first example of information display, the ophthalmic apparatus 1000 displays the Scheimpflug image acquired in step S1 and / or the inflammation state information generated in step S2 as is on the display device. The ophthalmic apparatus 1000 may display other information (referred to as additional information) together with the Scheimpflug image and / or the inflammation state information. The additional information may be any information that is useful for treating the subject's eye together with the Scheimpflug image and / or the inflammation state information.

[0202] In a second example of information display, the ophthalmologic apparatus 1000 generates an image simulating an image of the eye acquired by a conventional slit lamp microscope (referred to as a slit lamp image) from the Scheimpflug image and displays the generated simulated image on the display device. This makes it possible to provide an image simulating a slit lamp image that has traditionally been used to observe the inflammatory state of the subject's eye and is familiar to many physicians.

[0203] The process of generating a pseudo-image from a slit lamp image may be formed by a machine learning based process and / or a non-machine learning based process.

[0204] The machine learning-based processing is performed using a neural network constructed by machine learning using training data including a plurality of pairs of Scheimpflug and slit lamp images, for example, a convolutional neural network configured to receive a Scheimpflug image as an input and output a pseudo-image.

[0205] The non-machine learning based processing may include, for example, processing for transforming the appearance of an image, such as processing for generating artificial blur, color conversion, image quality conversion, etc. Exemplary aspects of the non-machine learning based processing include processing for constructing a three-dimensional image (e.g., an 8-bit grayscale volume) from a series of Scheimpflug images collected by slit scanning, processing for reducing the maximum value of the pixel value range (pixel value gradation) (e.g., processing for reducing 256 gradations to 10 gradations), processing for setting a region of interest (e.g., a rectangular parallelepiped region of predetermined dimensions) in the gradation-converted three-dimensional image, and processing for constructing a front image of the set region of interest (e.g., maximum intensity projection (MIP)).

[0206] The pseudo-image may be an image representing the same area as the Scheimpflug image with a wide focus, or it may be a partial area of ​​the area represented by the Scheimpflug image (for example, a field of view, i.e., a field of view measuring 1 mm square, which is the evaluation range in the classification criteria for uveitis diseases proposed by the SUN Working Group).

[0207] For example, the ophthalmologic apparatus 1000 can highlight a portion of interest in the pseudo image. Examples of the portion of interest include an image region corresponding to inflammatory cells, an image region corresponding to anterior chamber flare, an image region corresponding to opacity of the lens, etc.

[0208] In the third example of information display, the ophthalmologic apparatus 1000 creates a map showing the inflammatory state of the subject's eye (referred to as an inflammatory state map) and displays it on the display device.

[0209] Examples of inflammation state maps include an inflammatory cell map showing the location (distribution) of inflammatory cells in the anterior chamber, an inflammatory cell density map (inflammatory cell number map) showing the distribution of the density (or number) of inflammatory cells in the anterior chamber, etc. The process of creating these maps related to inflammatory cells includes, for example, a process of identifying image regions (cellular regions) corresponding to inflammatory cells from each of a series of Scheimpflug images collected by slit scanning (second segmentation), a process of determining the position of each identified cellular region (for example, two-dimensional coordinates in the definition coordinate system of the Scheimpflug images, or three-dimensional coordinates in the definition coordinate system of a series of three-dimensional images based on the Scheimpflug images), and a process of creating a map based on the determined position of each cellular region.

[0210] The ophthalmic apparatus 1000 can display an inflammation state map together with the Scheimpflug image and / or the inflammation state information. For example, the ophthalmic apparatus 1000 can display a front image based on a series of Scheimpflug images collected by slit scanning and an inflammation state map generated based on the same series of Scheimpflug images. As a specific example, the ophthalmic apparatus 1000 can display the inflammation state map superimposed on the front image, or display the front image and the inflammation state map side by side.

[0211] A second operation example of the ophthalmologic apparatus 1000 is shown in Fig. 20. This operation example is executed to acquire cell evaluation information. Unless otherwise specified, any of the matters described with respect to the operation example in Fig. 19 can be combined with this operation example.

[0212] First, the ophthalmologic apparatus 1000 acquires a Scheimpflug image of the subject's eye by the image acquisition unit 1010 (S11).

[0213] Next, the ophthalmic device 1000 applies at least one of the above-mentioned first segmentation, second segmentation, and cell evaluation information generation processes to the Scheimpflug image acquired in step S11 using the data processing unit 1020 (S12).

[0214] When the ophthalmic apparatus 1000 applies the first segmentation, the second segmentation, and the cell evaluation information generation processing to a Scheimpflug image, for example, the data processing unit 1030 in Fig. 3 or the data processing unit 1040 in Fig. 8 is adopted as the data processing unit 1020 of the ophthalmic apparatus 1000. Furthermore, the data processing unit 1020 in this example may include at least one of the first segmentation unit 1031A in Fig. 4, the second segmentation unit 1032A in Fig. 6, and the cell evaluation information generation processing unit 1033A in Fig. 7.

[0215] In another example where the ophthalmic apparatus 1000 applies the first segmentation, the second segmentation, and the cell evaluation information generation process to a Scheimpflug image, the data processing unit 1090 in FIG. 18 is adopted as the data processing unit 1020 of the ophthalmic apparatus 1000.

[0216] When the ophthalmic apparatus 1000 applies the second segmentation and the cell evaluation information generation processing to the Scheimpflug image (in other words, when the first segmentation is not performed), for example, the data processing unit 1050 in Fig. 9 is employed as the data processing unit 1020 of the ophthalmic apparatus 1000. Furthermore, the data processing unit 1020 in this example may include the second segmentation unit 1051A in Fig. 10 and / or the cell evaluation information generation processing unit 1052A in Fig. 11.

[0217] When the ophthalmic apparatus 1000 applies the cell evaluation information generation processing to the Scheimpflug image (in other words, when the first segmentation and the second segmentation are not performed), for example, the data processing unit 1060 in Fig. 12 is adopted as the data processing unit 1020 of the ophthalmic apparatus 1000. Furthermore, the data processing unit 1020 in this example may include the cell evaluation information generation processing unit 1061A in Fig. 13.

[0218] When the ophthalmic apparatus 1000 applies the first segmentation and the cell evaluation information generation processing to the Scheimpflug image (in other words, when the second segmentation is not performed), for example, the data processing unit 1070 in Fig. 14 or the data processing unit 1080 in Fig. 17 is adopted as the data processing unit 1020 of the ophthalmic apparatus 1000. Furthermore, the data processing unit 1020 in this example may include the first segmentation unit 1071A in Fig. 15 and / or the cell evaluation information generation processing unit 1072A in Fig. 16.

[0219] The configurations that can be adopted to perform step S12 are not limited to these. For example, the data processing unit 1020 of the ophthalmologic apparatus 1000 according to some exemplary aspects may be configured to be able to perform only the first segmentation, only the second segmentation, or only the first and second segmentation among the first segmentation, the second segmentation, and the cell evaluation information generation process.

[0220] Next, the ophthalmologic apparatus 1000 generates cell evaluation information by the data processing unit 1020 (S13). Note that if the cell evaluation information generation process is executed in step S12 and all of the cell evaluation information to be acquired in this examination has been acquired in step S12, it is not necessary to execute step S13 (in other words, step S13 is included in step S12).

[0221] Step S12 and / or step S13 may partially include operations by the user. For example, examples of operations by the user include an operation for specifying an anterior chamber region in a Scheimpflug image, an operation for specifying a cellular region in the Scheimpflug image, an operation for specifying a cellular region in the anterior chamber region, an operation for creating cell evaluation information from an anterior chamber region, an operation for creating cell evaluation information from a cellular region, an operation for editing (correcting) an anterior chamber region identified by the first segmentation, an operation for editing (correcting) a cellular region identified by the second segmentation, an operation for editing (correcting) cell evaluation information generated in the cell evaluation information generation process, an operation for creating other cell evaluation information from the cell evaluation information generated in the cell evaluation information generation process, etc.

[0222] These operations are performed using a user interface, which includes a display device and an operation device. The ophthalmologic apparatus 1000 may include at least a part of the user interface.

[0223] Next, the ophthalmologic device 1000 displays on the display device the Scheimpflug image acquired in step S11, the information acquired in step S12 (e.g., information based on the Scheimpflug image, the anterior chamber region, information based on the anterior chamber region, the cell region, information related to the cell region, and cell evaluation information), the cell evaluation information generated in step S13, etc. (S14).

[0224] A third operation example of the ophthalmologic apparatus 1000 is shown in Fig. 21. This operation example is executed to acquire cell evaluation information. Unless otherwise specified, any matter described with respect to the operation example in Fig. 19 and / or any matter described with respect to the operation example in Fig. 20 can be combined with this operation example.

[0225] The data processing unit 1020 of the ophthalmologic apparatus 1000 according to this example includes a second segmentation unit (e.g., second segmentation unit 1032A or second segmentation unit 1051A) including a convolutional neural network, and an element (e.g., conversion processing unit 1042 or conversion processing unit 1082) that performs data structure conversion to match this convolutional neural network.

[0226] First, the ophthalmologic apparatus 1000 acquires a Scheimpflug image of the subject's eye by the image acquisition unit 1010 (S21).

[0227] Next, the ophthalmic apparatus 1000 causes the data processing unit 1020 to apply a first segmentation (anterior chamber segmentation) for identifying the anterior chamber region from the Scheimpflug image to the Scheimpflug image acquired in step S21 (S22). The anterior chamber segmentation method in this step may be any method, and may include one or both of data processing by the data processing unit 1020 and user operation. Here, the anterior chamber segmentation by the data processing unit 1020 may be any of the various methods described above.

[0228] Next, the data processing unit 1020 removes ghosts in the anterior chamber region identified in the anterior chamber segmentation in step S22 (S23), thereby preventing the ghosts from being erroneously detected as a cell region in the cell segmentation in step S25, which will be described later.

[0229] Next, the data processing unit 1020 converts the anterior chamber region from which ghosts have been removed in step S23 into image data having a structure corresponding to the input layer of the convolutional neural network used in the next step S25 (S24).

[0230] In some exemplary embodiments, the order of removing ghosts from the anterior chamber region (step S23 in this example) and converting the data structure of the anterior chamber region (step S24 in this example) may be reversed.

[0231] Next, the data processing unit 1020 inputs the image data (transformed data of the anterior chamber region) acquired in step S24 into a convolutional neural network of a second segmentation unit configured to perform cell segmentation (S25), thereby identifying the cellular region within the anterior chamber region acquired in step S22.

[0232] Next, the data processing unit 1020 evaluates the density of inflammatory cells in the anterior chamber of the subject's eye based on the result of the cell region identification performed in step S25 (S26). The processing of this step is performed by, for example, any one of the various cell evaluation information generating processing units described above.

[0233] Next, the data processing unit 1020 generates cell evaluation information based on the results of the evaluation performed in step S26 (S27). The processing of this step is performed by, for example, any one of the various cell evaluation information generation processing units described above.

[0234] Next, the ophthalmologic apparatus 1000 displays the information acquired in steps S21 to S27 on the display device (S28).

[0235] Examples of information displayed in this step include the Scheimpflug image acquired in step S21, information acquired in step S22 (e.g., information based on the Scheimpflug image, anterior chamber region, information based on the anterior chamber region), information acquired in step S23 (e.g., anterior chamber region from which ghosts have been removed, information based on the anterior chamber region from which ghosts have been removed), information acquired in step S24 (e.g., a shaped anterior chamber region, information based on the shaped anterior chamber region), information acquired in step S25 (e.g., a cell region, information based on the cell region), information acquired in step S26 (e.g., inflammatory cell density, density evaluation information, information based on the density evaluation information), and information acquired in step S27 (e.g., cell evaluation information, information based on the cell evaluation information).

[0236] In some exemplary embodiments, in step S27, the inflammatory cell density value (the number of inflammatory cells present in an area 1 mm square) used in the evaluation proposed by the SUN Working Group and / or the grade corresponding to this density value are determined, and further, in step S28, at least the density value and / or grade determined in step S27 are displayed.

[0237] A fourth operation example of the ophthalmologic apparatus 1000 is shown in Fig. 22. This operation example is executed to acquire cell evaluation information. Unless otherwise specified, at least one of any of the matters described with respect to the operation example in Fig. 19, any of the matters described with respect to the operation example in Fig. 20, and any of the matters described with respect to the operation example in Fig. 21 can be combined with this operation example.

[0238] The data processing unit 1020 of the ophthalmic device 1000 in this example includes a cell evaluation information generation processing unit (e.g., cell evaluation information generation processing unit 1033A, cell evaluation information generation processing unit 1052A, cell evaluation information generation processing unit 1061A, or cell evaluation information generation processing unit 1072A) that includes a convolutional neural network, and an element (e.g., conversion processing unit 1042 or conversion processing unit 1082) that performs data structure conversion to match this convolutional neural network.

[0239] First, the ophthalmologic apparatus 1000 acquires a Scheimpflug image of the subject's eye by the image acquisition unit 1010 (S31).

[0240] Next, the ophthalmologic apparatus 1000 applies a first segmentation (anterior chamber segmentation) for identifying the anterior chamber region from the Scheimpflug image to the Scheimpflug image acquired in step S31 by the data processing unit 1020 (S32). As in step S22 of the third operation example, the anterior chamber segmentation method in this step may be any method.

[0241] Next, the data processing unit 1020 removes ghosts in the anterior chamber region identified by the anterior chamber segmentation in step S32 (S33), thereby preventing the ghosts from being reflected in the evaluation results in the cell evaluation information generation process in step S35, which will be described later.

[0242] Next, the data processing unit 1020 converts the anterior chamber region from which ghosts have been removed in step S33 into image data having a structure corresponding to the input layer of the convolutional neural network used in the next step S35 (S34).

[0243] In some exemplary embodiments, the order of removing ghosts from the anterior chamber region (step S33 in this example) and converting the data structure of the anterior chamber region (step S34 in this example) may be reversed.

[0244] Next, the data processing unit 1020 inputs the image data (converted data of the anterior chamber region) acquired in step S34 to a convolutional neural network of a cell evaluation information generation processing unit configured to execute cell evaluation information generation processing, thereby generating cell evaluation information based on the anterior chamber region acquired in step S32 (S35).

[0245] Next, the ophthalmologic apparatus 1000 displays the cell evaluation information generated in step S35 on the display device (S36).

[0246] In some exemplary embodiments, in step S35, the inflammatory cell density value (the number of inflammatory cells present in an area 1 mm square) used in the evaluation proposed by the SUN Working Group and / or the grade corresponding to this density value are determined, and further, in step S36, at least the density value and / or grade determined in step S35 are displayed.

[0247] In some exemplary embodiments, in step S35, the inflammatory cell density value (the number of inflammatory cells present in a 1 mm square area) used in the evaluation proposed by the SUN Working Group is determined. Data processing unit 1020 in this embodiment determines the grade corresponding to the density value determined in step S35 by referring to predetermined data that indicates the correspondence between density values ​​and grades. In step S36 in this embodiment, the grade determined by data processing unit 1020 (and the density value determined in step S35 in this embodiment) are displayed.

[0248] In addition to the cell evaluation information generated in step S35 and / or information based thereon, the ophthalmologic apparatus 1000 can display on the display device any information acquired in steps S31 to S34. Examples of information that can be displayed in addition to the cell evaluation information include the Scheimpflug image acquired in step S31, information acquired in step S32 (e.g., information based on the Scheimpflug image, the anterior chamber region, information based on the anterior chamber region), information acquired in step S33 (e.g., an anterior chamber region from which ghosts have been removed, information based on the anterior chamber region from which ghosts have been removed), information acquired in step S34 (e.g., a shaped anterior chamber region, information based on the shaped anterior chamber region), etc.

[0249] When generating inflammation state information other than cell evaluation information, the inflammation state information can be generated in the same manner as when generating cell evaluation information. As described above, examples of inflammation state information other than cell evaluation information include information on anterior chamber flare, information on crystalline lens opacity, information on the onset and progress of disease, information on disease activity, etc.

[0250] All of these exemplary inflammation status information, as well as the cellular evaluation information, may be generated and evaluated with reference to the classification criteria for uveitis diseases proposed by the SUN Working Group.

[0251] The processes for generating and evaluating these exemplary inflammation status information may be machine learning-based processes or non-machine learning-based processes, or may be a combination of machine learning-based processes and non-machine learning-based processes. The construction of a neural network for performing machine learning-based processes may be performed in the same manner as the construction of a neural network for generating cell evaluation information. The processor for performing non-machine learning-based processes may be configured to at least execute a process for determining an evaluation target (e.g., anterior chamber flare, crystalline lens opacity, disease onset / progression, disease activity, etc.), and may further be configured to execute a process for making an evaluation based on the determined evaluation target.

[0252] 23 shows an example of a specific configuration of an ophthalmic apparatus that can function as the above-described ophthalmic apparatus 1000. The ophthalmic apparatus of this example is a system (slit lamp microscope system 1) that combines a slit lamp microscope and a computer (information processing device).

[0253] The slit lamp microscope system 1 includes an illumination system 2, an imaging system 3, a video imaging system 4, an optical path coupling element 5, a moving mechanism 6, a control unit 7, a data processing unit 8, a communication unit 9, and a user interface 10. The cornea of ​​the subject's eye E is indicated by the symbol C, and the crystalline lens is indicated by the symbol CL. The anterior chamber corresponds to the region between the cornea C and the crystalline lens.

[0254] As a non-limiting example of the arrangement of elements of the slit lamp microscope system 1, the slit lamp microscope system 1 of some exemplary embodiments includes a microscope body, a computer, and a communication device that handles communication between the microscope body and the computer. The microscope body includes an illumination system 2, an imaging system 3, a video imaging system 4, an optical path coupling element 5, and a movement mechanism 6. The computer includes a control unit 7, a data processing unit 8, a communication unit 9, and a user interface 10. The computer may be installed, for example, near the microscope body or on a network.

[0255] The combination of the illumination system 2, the imaging system 3, and the moving mechanism 6 is an example of an image acquisition unit 1010 of the ophthalmic apparatus 1000. The illumination system 2 is an example of an illumination system 1011 of the ophthalmic apparatus 1000. The imaging system 3 is an example of an imaging system 1012 of the ophthalmic apparatus 1000.

[0256] The illumination system 2 projects slit light onto the anterior segment of the subject's eye E. Reference symbol 2a denotes the optical axis of the illumination system 2 (referred to as the illumination optical axis). The illumination system 2 may have a configuration similar to that of an illumination system of a conventional slit lamp microscope. For example, although not shown, the illumination system 2 includes, in order from the side farthest from the subject's eye E, an illumination light source, a positive lens, a slit forming unit, and an objective lens. The illumination light output from the illumination light source passes through the positive lens and is projected onto the slit forming unit. The slit forming unit generates slit light by transmitting a portion of the illumination light. The slit forming unit has a pair of slit blades. The width of the slit light can be changed by changing the distance between these slit blades (referred to as the slit width). Furthermore, the longitudinal direction of the slit light can be changed by rotating the pair of slit blades. Furthermore, the slit forming unit can change the longitudinal dimension of the slit light. The slit light generated by the slit forming unit is refracted by the objective lens and projected onto the anterior segment of the subject's eye E. The configuration for generating the slit light is not limited to this example, and any configuration usable for generating the slit light may be used. The illumination system 2 may include a focusing mechanism for changing the focus position of the slit light. This focusing mechanism, for example, moves the objective lens along the illumination optical axis 2a. Alternatively, the focusing mechanism moves a focusing lens disposed between the objective lens and the slit forming unit.

[0257] 23 is a top view, in which the direction along the axis of the subject's eye E is the Z direction, the direction perpendicular to this that is the left-right direction for the subject is the X direction, and the direction perpendicular to both the X and Z directions (the up-down direction, the body axis direction) is the Y direction. In this embodiment, the slit lamp microscope system 1 can be aligned with the subject's eye E so that the illumination optical axis 2a coincides with the axis of the subject's eye E, or more broadly, alignment can be performed so that the illumination optical axis 2a is positioned parallel to the axis of the subject's eye E.

[0258] The imaging system 3 images the anterior segment of the eye onto which the slit light from the illumination system 2 is projected. Reference symbol 3a denotes the optical axis of the imaging system 3 (referred to as the imaging optical axis). The imaging system 3 includes an optical system 3A and an image sensor 3B. The optical system 3A guides light from the anterior segment of the subject's eye E onto which the slit light is projected to the image sensor 3B. The optical system 3A may have a configuration similar to that of an imaging system of a conventional slit lamp microscope. For example, the optical system 3A includes, in order from the side closest to the subject's eye E, an objective lens, a variable magnification optical system, and an imaging lens. The light from the anterior segment of the subject's eye E onto which the slit light is projected passes through the objective lens and the variable magnification optical system and is imaged by the imaging lens on the imaging surface of the image sensor 3B. The image sensor 3B receives the light guided by the optical system 3A on its imaging surface. The image sensor 3B includes an area sensor having a two-dimensional imaging area. The area sensor may be, for example, a charge-coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor. The imaging system 3 may include a focusing mechanism for changing its focus position. The focusing mechanism may, for example, move an objective lens along the imaging optical axis 3a. Alternatively, the focusing mechanism may move a focusing lens disposed between the objective lens and the imaging lens along the imaging optical axis 3a.

[0259] The illumination system 2 and the imaging system 3 function as a Scheimpflug camera. That is, the illumination system 2 and the imaging system 3 are configured so that the object plane along the illumination optical axis 2a, the optical system 3A, and the imaging plane of the imaging element 3B satisfy the so-called Scheimpflug condition. More specifically, the YZ plane (including the object plane) passing through the illumination optical axis 2a, the principal plane of the optical system 3A, and the imaging plane of the imaging element 3B intersect on the same straight line. This allows imaging to be performed while focusing on all positions within the object plane (all positions in the direction along the illumination optical axis 2a).

[0260] In this embodiment, the illumination system 2 and the imaging system 3 are configured so that the imaging system 3 is focused on at least the range from the posterior surface of the cornea C to the anterior surface of the crystalline lens CL (anterior chamber). Taking practicality into consideration, the illumination system 2 and the imaging system 3 may be configured so that the imaging system 3 is focused on at least the range from the anterior surface of the cornea C to the posterior surface of the crystalline lens CL. This allows the slit lamp microscope system 1 to capture an image of the anterior segment of the subject's eye E while the imaging system 3 is focused on the entire range from the apex of the anterior surface of the cornea C (Z=Z1) to the apex of the posterior surface of the crystalline lens CL (Z=Z2). The intersection of the illumination optical axis 2a and the imaging optical axis 3a is located at the coordinate Z=Z0. This condition is realized according to the configuration and arrangement of the elements included in the illumination system 2, the configuration and arrangement of the elements included in the imaging system 3, and the relative positions of the illumination system 2 and the imaging system 3. The parameters indicating the relative positions of the illumination system 2 and the imaging system 3 include, for example, the angle θ formed between the illumination optical axis 2a and the imaging optical axis 3a. The angle θ is set to, for example, 17.5 degrees, 30 degrees, or 45 degrees. The angle θ may be variable.

[0261] The moving image capturing system 4 captures a moving image of the anterior segment of the subject's eye E in parallel with the imaging of the subject's eye by the illumination system 2 and the imaging system 3. The moving image capturing system 4 functions as a video camera.

[0262] The optical path combining element 5 combines the optical path (illumination optical path) of the illumination system 2 with the optical path (video imaging optical path) of the moving image capturing system 4. The optical path combining element 5 may be, for example, a beam splitter such as a half mirror or a dichroic mirror.

[0263] A specific example of an optical system including an illumination system 2, an imaging system 3, a video imaging system 4, and an optical path coupling element 5 is shown in FIG. 24. In this example, the imaging system 3 includes two imaging systems (a first imaging system and a second imaging system). In some exemplary embodiments, the optical system of the slit lamp microscope system 1 may include other elements (e.g., any element in the description of the ophthalmic apparatus 1000, any element of a known slit lamp microscope, or any element of a known ophthalmic apparatus) in addition to or instead of the elements shown in FIG. 24.

[0264] 24 includes an illumination system 20, a left imaging system 30L, a right imaging system 30R, and a video imaging system 40. The illumination system 20 is an example of the illumination system 2. The combination of the left imaging system 30L and the right imaging system 30R is an example of the imaging system 3, and is an example of the combination of the first imaging system 1014 and the second imaging system 1015 of the ophthalmic apparatus 1000. The video imaging system 40 is an example of the video imaging system 4. The beam splitter 47 is an example of the optical path coupling element 5.

[0265] In FIG. 24, reference numeral 20a denotes the optical axis of the illumination system 20 (referred to as the illumination optical axis), reference numeral 30La denotes the optical axis of the left imaging system 30L (referred to as the left imaging optical axis), and reference numeral 30Ra denotes the optical axis of the right imaging system 30R (referred to as the right imaging optical axis). The orientation of the left imaging optical axis 30La and the orientation of the right imaging optical axis 30Ra are different from each other. The angle formed by the illumination optical axis 20a and the left imaging optical axis 30La is represented by θL, and the angle formed by the illumination optical axis 20a and the right imaging optical axis 30Ra is represented by θR. The angles θL and θR may be equal to or different from each other. Each of the angles θL and θR may be variable. The illumination optical axis 20a, the left imaging optical axis 30La, and the right imaging optical axis 30Ra intersect at a single point. As in FIG. 23, the Z coordinate of this intersection is represented by Z0.

[0266] The movement mechanism 6 in this example is configured to move the illumination system 20, the left imaging system 30L, and the right imaging system 30R in the direction indicated by the arrow 49 (X direction). In some exemplary embodiments, the illumination system 20, the left imaging system 30L, and the right imaging system 30R are placed on a stage that is movable at least in the X direction, and the movement mechanism 6 moves this movable stage in the X direction in accordance with a control signal from the control unit 7.

[0267] The illumination system 20 projects slit light onto the anterior segment of the subject's eye E. Similar to the illumination system of a conventional slit lamp microscope, the illumination system 20 includes, in order from the side farthest from the subject's eye E, an illumination light source 21, a positive lens 22, a slit forming unit 23, and objective lens groups 24 and 25.

[0268] Illumination light (e.g., visible light) output from the illumination light source 21 is refracted by the positive lens 22 and projected onto the slit forming unit 23. A part of the projected illumination light passes through the slit formed by the slit forming unit 23 to become slit light. The generated slit light is refracted by the objective lens groups 24 and 25, and then reflected by the beam splitter 47 and projected onto the anterior segment of the subject's eye E.

[0269] The left imaging system 30L includes a reflector 31L, an imaging lens 32L, and an image sensor 33L. The reflector 31L and the imaging lens 32L guide light from the anterior segment onto which the slit light is projected by the illumination system 20 (light traveling in the direction of the left imaging system 30L) to the image sensor 33L.

[0270] The light traveling from the anterior segment toward the left imaging system 30L is light from the anterior segment onto which the slit light is projected, and is light traveling in a direction away from the illumination optical axis 20a. The reflector 31L reflects this light in a direction approaching the illumination optical axis 20a. The imaging lens 32L refracts the light reflected by the reflector 31L and forms an image on the imaging surface 34L of the imaging element 33L. The imaging element 33L receives this light on the imaging surface 34L.

[0271] The left imaging system 30L repeatedly captures images in parallel with the movement of the illumination system 20, the left imaging system 30L, and the right imaging system 30R by the movement mechanism 6. This allows a plurality of anterior eye images (a series of Scheimpflug images) to be obtained.

[0272] The object plane along the illumination optical axis 20a, the optical system including the reflector 31L and the imaging lens 32L, and the imaging plane 34L satisfy the Scheimpflug condition. More specifically, considering the deflection of the optical path of the imaging system 30L by the reflector 31L, the YZ plane (including the object plane) passing through the illumination optical axis 20a, the principal plane of the imaging lens 32L, and the imaging plane 34L intersect on the same straight line. This allows the left imaging system 30L to capture images by focusing on all positions within the object plane (for example, in the range from the anterior surface of the cornea to the posterior surface of the crystalline lens).

[0273] The right photographing system 30R includes a reflector 31R, an imaging lens 32R, and an image sensor 33R. The reflector 31R and the imaging lens 32R guide light (light traveling in the direction of the right photographing system 30R) from the anterior segment onto which the slit light is projected by the illumination system 20 to the image sensor 33R. The right photographing system 30R acquires multiple images of the anterior segment (a series of Scheimpflug images) by repeatedly photographing in parallel with the movement of the illumination system 20, the left photographing system 30L, and the right photographing system 30R by the movement mechanism 6. The object plane along the illumination optical axis 20a, the optical system including the reflector 31R and the imaging lens 32R, and the image sensor 34R satisfy the Scheimpflug condition.

[0274] The Scheimpflug image acquisition by the left imaging system 30L and the Scheimpflug image acquisition by the right imaging system 30R are performed in parallel with each other. The combination of the series of Scheimpflug images acquired by the left imaging system 30L and the series of Scheimpflug images acquired by the right imaging system 30R corresponds to the combination of the first Scheimpflug image group and the second Scheimpflug image group.

[0275] The control unit 7 can synchronize the repeated photographing by the left imaging system 30L and the repeated photographing by the right imaging system 30R. This allows a correspondence relationship to be obtained between the series of Scheimpflug images obtained by the left imaging system 30L and the series of Scheimpflug images obtained by the right imaging system 30R. This correspondence relationship is a temporal correspondence relationship, and more specifically, pairs images acquired substantially simultaneously.

[0276] Alternatively, the control unit 7 or the data processing unit 8 can execute a process for determining a correspondence relationship between a plurality of anterior-segment images acquired by the left imaging system 30L and a plurality of anterior-segment images acquired by the right imaging system 30R. For example, the control unit 7 or the data processing unit 8 can pair the anterior-segment images sequentially input from the left imaging system 30L and the anterior-segment images sequentially input from the right imaging system 30R based on the timing of their input.

[0277] The video imaging system 40 captures video of the anterior segment of the subject's eye E from a fixed position in parallel with imaging by the left imaging system 30L and imaging by the right imaging system 30R. Here, the video imaging system 40 does not need to be moved by the moving mechanism 6. The video imaging system 40 is disposed coaxially with the illumination system 20, but the arrangement is not limited thereto. In some exemplary embodiments, the video imaging system can be disposed non-coaxially with the illumination system 20.

[0278] The light that has passed through the beam splitter 47 is reflected by a reflector 48 and enters the moving image capturing system 40. The light that has entered the moving image capturing system 40 is refracted by an objective lens 41, and then an image is formed on the imaging surface of an imaging element 43 by an imaging lens 42. The imaging element 43 is an area sensor.

[0279] The video capture system 40 can be used for monitoring, aligning, tracking, etc. the movement of the subject's eye E. Furthermore, the video capture system 40 can be used to process a series of Scheimpflug images.

[0280] Returning to Fig. 23, the movement mechanism 6 is configured to move the illumination system 2 and the imaging system 3 together in the X direction.

[0281] The control unit 7 is configured to control each part of the slit lamp microscope system 1. For example, the control unit 7 controls elements of the illumination system 2 (illumination light source, slit forming unit, focusing mechanism, etc.), elements of the imaging system 3 (focusing mechanism of optical system 3A, image sensor 3B, etc.), elements of the video imaging system 4 (focusing mechanism, image sensor, etc.), movement mechanism 6, data processing unit 8, communication unit 9, user interface 10, etc.

[0282] The control unit 7 can control the illumination system 2, the imaging system 3, and the moving mechanism 6, and the video imaging system 4, in parallel. This allows slit scanning (collection of a series of Scheimpflug images) and video imaging (collection of a series of time-series images) by the image acquisition unit 1010 of the ophthalmic apparatus 1000 to be performed in parallel. Furthermore, the control unit 7 can control the illumination system 2, the imaging system 3, and the moving mechanism 6, and the video imaging system 4, in synchronization with each other. This allows slit scanning and video imaging by the image acquisition unit 1010 of the ophthalmic apparatus 1000 to be synchronized with each other.

[0283] In a configuration in which the imaging system 3 includes a left imaging system 30L and a right imaging system 30R, the control unit 7 can synchronize repeated imaging by the left imaging system 30L (collection of a first Scheimpflug image group) and repeated imaging by the right imaging system 30R (collection of a second Scheimpflug image group).

[0284] The control unit 7 includes a processor, a main memory device, an auxiliary memory device, etc. The auxiliary memory device stores computer programs such as various control programs. These computer programs may be stored in a computer or storage device accessible to the slit lamp microscope system 1. The functions of the control unit 7 are realized by cooperation between software such as the control programs and hardware such as the processor.

[0285] The control unit 7 can apply the following control to the illumination system 2, the imaging system 3, and the moving mechanism 6 in order to scan the three-dimensional area of ​​the anterior segment of the eye E with slit light.

[0286] First, the control unit 7 controls the movement mechanism 6 to position the illumination system 2 and the imaging system 3 at a predetermined scan start position (alignment control). The scan start position is, for example, a position corresponding to an end (first end) of the cornea C in the X direction, or a position further away from the axis of the subject's eye E. The symbol X0 in FIG. 25A indicates the scan start position corresponding to the first end of the cornea C in the X direction. Furthermore, the symbol X0' in FIG. 25B indicates a scan start position further away from the axis EA of the subject's eye E than the position corresponding to the first end of the cornea C in the X direction.

[0287] The control unit 7 controls the illumination system 2 to start projecting slit light onto the anterior segment of the subject's eye E (slit light projection control). The control unit 7 also controls the imaging system 3 to start capturing video of the anterior segment of the subject's eye E (imaging control). After executing alignment control, slit light projection control, and imaging control, the control unit 7 controls the movement mechanism 6 to start moving the illumination system 2 and the imaging system 3 (movement control). The movement control moves the illumination system 2 and the imaging system 3 integrally. That is, the illumination system 2 and the imaging system 3 are moved while maintaining the relative position (angle θ, etc.) between the illumination system 2 and the imaging system 3 (while satisfying the Scheimpflug condition). The illumination system 2 and the imaging system 3 are moved from the aforementioned scan start position to a predetermined scan end position. The scan end position is, for example, a position corresponding to the end (second end) of the cornea C opposite the first end in the X direction, similar to the scan start position, or a position further away from the axis of the subject's eye E.

[0288] In this example, the slit scan is applied to a range from the scan start position to the scan end position. This slit scan is realized by projecting a slit light onto the anterior segment of the eye, with the X direction as the width direction and the Y direction as the length direction, moving the illumination system 2 and the imaging system 3 in the X direction in unison, and capturing video using the imaging system 3 in parallel (linked and synchronized). The length of the slit light (i.e., the dimension of the beam cross section of the slit light in the Y direction) is set to, for example, equal to or greater than the diameter of the cornea C on the surface of the subject's eye E. Furthermore, the movement distance of the illumination system 2 and the imaging system 3 by the movement mechanism 6 is set to equal to or greater than the diameter of the cornea C in the X direction. This makes it possible to apply the slit scan to a three-dimensional region including the entire cornea C, thereby enabling imaging of a wide range of the anterior chamber.

[0289] This type of slit scanning produces multiple anterior segment images (a series of Scheimpflug images) with different projection positions of the slit light. In other words, a moving image is produced depicting the movement of the projection position of the slit light in the X direction. An example of such multiple anterior segment images (i.e., a group of frames constituting a moving image) is shown in FIG. 26.

[0290] FIG. 26 shows multiple anterior-segment images (frame group) F1, F2, F3, . . . , FN. The subscript n in each of these anterior-segment images Fn (n = 1, 2, . . . , N) represents the chronological order. That is, the nth anterior-segment image acquired is represented by the symbol Fn. The anterior-segment image Fn includes a slit light image An. As shown in FIG. 26, the slit light images A1, A2, A3, . . . , AN move rightward in time sequence. In the example shown in FIG. 26, the scan start position and scan end position correspond to both ends of the cornea C in the X direction. Note that the scan start position and / or scan end position are not limited to this example and may be, for example, a position farther from the axis of the subject's eye E than the corneal edge. Furthermore, the scan direction and number of scans can be set arbitrarily.

[0291] The data processing unit 8 is configured to perform various types of data processing. The data to be processed may be either data acquired by the slit lamp microscope system 1 or data input from an external source.

[0292] The data processing unit 8 includes a processor, a main memory device, an auxiliary memory device, etc. The auxiliary memory device stores computer programs such as various data processing programs. These computer programs may be stored in a computer or storage device accessible to the slit lamp microscope system 1. The functions of the data processing unit 8 are realized by cooperation between software such as the data processing programs and hardware such as a processor.

[0293] The data processing unit 8 may have any of the configurations described for the data processing unit 1020 of the ophthalmologic apparatus 1000 (see FIG. 2C and FIGS. 3 to 18). The configuration of the data processing unit 8 is not limited thereto.

[0294] A case will be described in which the data processing unit 8 includes an image selection unit 1021 (see FIG. 2C). The image selection unit 1021 of this embodiment selects a new series of Scheimpflug images corresponding to the slit scan from two series of Scheimpflug images collected by the left imaging system 30L and the right imaging system 30R based on the correspondence between these two series of Scheimpflug images (the correspondence between the first Scheimpflug image group and the second Scheimpflug image group). The data processing unit 8 generates inflammation state information based on the new series of Scheimpflug images selected by the image selection unit 1021, similar to the inflammation state information generation unit 1022 of FIG. 2C.

[0295] The configuration of the data processing unit 8 is not limited to these examples. The data processing unit 8 in some exemplary embodiments may have any data processing function related to the technology disclosed by any of the applicants of the present application, such as any data processing function disclosed in Patent Document 3 (JP 2019-213733 A).

[0296] The communication unit 9 performs data communication between the slit lamp microscope system 1 and other devices. That is, the communication unit 9 transmits data to other devices and receives data transmitted from other devices.

[0297] The data communication method performed by the communication unit 9 may be any method. For example, the communication unit 9 includes one or more of various communication interfaces, such as a communication interface compatible with the Internet, a communication interface compatible with a dedicated line, a communication interface compatible with a LAN, and a communication interface compatible with short-range communication. The data communication may be wired communication or wireless communication.

[0298] The data transmitted and received by the communication unit 9 may be encrypted. In this case, for example, the control unit 7 and / or the data processing unit 8 includes at least one of an encryption processing unit that encrypts the data transmitted by the communication unit 9 and a decryption processing unit that decrypts the data received by the communication unit 9.

[0299] The user interface 10 includes any user interface device such as a display device, an operation device, etc. Users such as doctors, subjects, and assistants can use the user interface 10 to operate the slit lamp microscope system 1 and input information to the slit lamp microscope system 1.

[0300] The display device displays various types of information under the control of the control unit 7. The display device may include a flat panel display such as a liquid crystal display (LCD). The operation device includes devices for operating the slit lamp microscope system 1 and devices for inputting information. The operation device includes, for example, buttons, switches, levers, dials, handles, knobs, mice, keyboards, trackballs, operation panels, etc. A device in which the display device and operation device are integrated, such as a touch screen, may also be used.

[0301] At least a part of the user interface may be arranged as a peripheral device of the slit lamp microscope system 1.

[0302] The elements of the slit lamp microscope system 1 are not limited to those described above. The slit lamp microscope system 1 may include any element that can be combined with a slit lamp microscope, or more generally, any element that can be combined with an ophthalmic device. Furthermore, the slit lamp microscope system 1 may include any element for processing data of the subject's eye acquired by the slit lamp microscope, or more generally, any element for processing any ophthalmic data.

[0303] For example, the slit lamp microscope system 1 may include a fixation system that outputs light (fixation light) for fixating the subject's eye E. The fixation system typically includes at least one visible light source (fixation light source) or a display device that displays an image such as a landscape chart or a fixation target. The fixation system is arranged, for example, coaxially or non-coaxially with the illumination system 2 or the imaging system 3.

[0304] The ophthalmic apparatus 1000 and the slit lamp microscope system 1 described above have a function of photographing the subject's eye (photographing function, image acquisition unit 1010), but the ophthalmic apparatus according to the present disclosure is not limited to such a device (ophthalmic photographing apparatus). Some exemplary embodiments of the ophthalmic apparatus include a computer (information processing device) having a function of externally receiving an image of the subject's eye instead of (or in addition to) the photographing function.

[0305] An example of the configuration of an ophthalmic apparatus as such an information processing device is shown in Fig. 27. The ophthalmic apparatus 3000 of this example includes an image acquisition unit 3010 and a data processing unit 3020. The image acquisition unit 3010 includes an image reception unit 3011. The image acquisition unit 3010 may further include the same configuration as the image acquisition unit 1010 of the ophthalmic apparatus 1000. The data processing unit 3020 may include any of the configurations described above for the data processing unit 1020 of the ophthalmic apparatus 1000, but is not limited thereto.

[0306] The image receiving unit 3011 is configured to receive a Scheimpflug image of the subject's eye that has been acquired in advance (in other words, a Scheimpflug image of the subject's eye that has been acquired by previous imaging). The image receiving unit 3011 includes, for example, a communication device and / or a media drive. The communication device is configured to receive data stored in an external storage device, similar to the communication unit 9 of the slit lamp microscope system 1, for example. The media drive is configured to read data recorded on a recording medium.

[0307] The data processing unit 3020 is configured to execute processing for generating inflammation state information indicating the inflammation state of the subject's eye from the Scheimpflug image received by the image receiving unit 3011. For the processing that can be executed by the data processing unit 3020, please refer to the description of the ophthalmologic apparatus 1000 and the description of the slit lamp microscope system 1.

[0308] This disclosure presents some exemplary aspects of the embodiments. These aspects are merely examples of the present invention. Therefore, any modifications (omissions, substitutions, additions, etc.) within the scope of the gist of the present invention can be applied to this disclosure.

[0309] It is possible to configure a program that causes a computer to execute any one or more of the processes described in this disclosure. It is also possible to create a recording medium on which such a program is recorded. The recording medium is a non-transitory recording medium that can be read by a computer. Such a recording medium may take any form, and examples thereof include a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0310] The present invention may include a method including any one or more steps described in this disclosure. A method according to some exemplary embodiments is a method for controlling an ophthalmic apparatus (e.g., the ophthalmic apparatus 1000, the slit lamp microscope system 1, or the ophthalmic apparatus 3000) including a processor, and includes a step of causing the ophthalmic apparatus to acquire a Scheimpflug image of a subject's eye (referred to as a first acquisition step), and a step of causing the processor to execute processing for generating inflammation state information indicating an inflammation state of the subject's eye from the Scheimpflug image (referred to as a first generation step).

[0311] The first acquisition step and / or the first generation step may be embodied by any of the features in the description of the ophthalmic apparatus 1000, any of the features in the description of the slit lamp microscope system 1, and any of the features in the description of the ophthalmic apparatus 3000. Furthermore, any of the features in the description of the ophthalmic apparatus 1000, any of the features in the description of the slit lamp microscope system 1, and any of the features in the description of the ophthalmic apparatus 3000 may be combined with the first acquisition step and the first generation step.

[0312] In addition, a method according to some exemplary embodiments is a method for processing an image of an eye, and includes a step of acquiring a Scheimpflug image of the test eye (referred to as a second acquisition step) and a step of performing processing to generate inflammation state information indicating the inflammatory state of the test eye from the Scheimpflug image (referred to as a second generation step).

[0313] The second acquisition step and / or the second generation step may be embodied by any of the features in the description of the ophthalmic apparatus 1000, any of the features in the description of the slit lamp microscope system 1, and any of the features in the description of the ophthalmic apparatus 3000. Furthermore, any of the features in the description of the ophthalmic apparatus 1000, any of the features in the description of the slit lamp microscope system 1, and any of the features in the description of the ophthalmic apparatus 3000 may be combined with the second acquisition step and the second generation step.

[0314] The present invention may include a program (referred to as a first program) that causes a computer to execute a method for controlling an ophthalmic device. The present invention may also include a program (referred to as a second program) that causes a computer to execute a method for processing eye images. Furthermore, the present invention may include a computer-readable non-transitory recording medium on which the first program is recorded. The present invention may also include a computer-readable non-transitory recording medium on which the second program is recorded. Such a non-transitory recording medium may be in any form, and examples thereof include a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0315] In the above embodiment, the automatic evaluation of the density of inflammatory cells present in the anterior chamber has been described in particular detail. There are various points to consider when automatically evaluating the density of inflammatory cells. In addition to the various points described in the above embodiment, the inventors have also considered the following points: (1) distinguishing between artifacts and cellular regions in a Scheimpflug image (particularly the anterior chamber region); (2) achieving consistency with conventional evaluation methods performed using a slit lamp microscope (e.g., the evaluation method proposed by the SUN Working Group); and (3) ensuring the quality of the evaluation (e.g., stability, robustness, reproducibility, accuracy, precision, etc.) regardless of adjustments or changes to the imaging conditions (e.g., camera gain).

[0316] Regarding (1), several solutions have been proposed in the above embodiments, but it is also possible to use any known artifact detection or artifact removal technique, such as detection or removal of artifacts caused by eyelashes.

[0317] Regarding (2), several solutions have been proposed in the above embodiments. In addition, for example, a correspondence relationship between a data group obtained by a conventional evaluation method and a data group obtained by the evaluation method according to the present disclosure can be determined, and consistency with the conventional evaluation method can be improved based on this correspondence relationship. The correspondence relationship between the data groups can be created using machine learning-based processing and / or non-machine learning-based processing. In machine learning-based processing, for example, machine learning is performed using training data including multiple pairs of data groups obtained by the conventional evaluation method and data groups obtained by the evaluation method according to the present disclosure. The inference model constructed in this way includes a neural network that receives data obtained by the evaluation method according to the present disclosure as input and outputs data that resembles data obtained by the conventional evaluation method.

[0318] Similarly, for (3), correspondences between various data obtained in response to various shooting conditions (e.g., various camera gain values) can be found, and the evaluation quality can be stabilized based on these correspondences. The correspondences between data can be created using machine learning-based processing and / or non-machine learning-based processing. In machine learning-based processing, for example, machine learning is performed using training data including multiple pairs of data obtained under a first condition and data obtained under a second condition. The inference model constructed in this way includes a neural network that receives data obtained under the first condition (or the second condition) as input and outputs data that resembles the data obtained under the second condition (or the first condition).

[0319] The ophthalmic device, method for controlling the ophthalmic device, method for processing eye images, program, and recording medium disclosed herein make it possible to at least partially automate the evaluation of inflammatory conditions based on eye images, which has traditionally been done manually.

[0320] In the present disclosure, the inflammatory state is evaluated based on a Scheimpflug image, and therefore the evaluation can be performed based on a high-quality image in focus over a wide range, making it possible to perform a high-quality evaluation over a wide range of the subject's eye.

[0321] In addition, by combining slit scanning, it is possible to rapidly obtain a high-quality Scheimpflug image group (a series of Scheimpflug images) that is in focus over a wide three-dimensional area of ​​the subject's eye, and evaluation can be performed based on this Scheimpflug image group, making it possible to evaluate a very wide area of ​​the subject's eye with high quality. For example, it becomes possible to evaluate a wide area of ​​the anterior chamber, and it is also possible to add the lens and cornea to the evaluation targets.

[0322] In the invention described in Patent Document 5 (International Publication No. 2018 / 003906), scattered light cannot be detected if the exposure time required to capture one frame is short, such as at video rate, so the exposure time required to capture one frame is set to approximately 100 milliseconds to 1 second. However, considering the effects of eye movement and blinking of the subject's eye, slit scanning as disclosed in the present disclosure cannot be performed.

[0323] Furthermore, in the invention described in Patent Document 5 (WO 2018 / 003906), the dimensions of the projected image of the slit light on the cornea are set to 0.2 mm x 2 mm, making it difficult to image a wide area of ​​the anterior segment. In contrast, in this embodiment, the dimensions of the projected image of the slit light on the cornea can be set to, for example, approximately 0.05 mm x 8 to 12 mm, making it possible to image a wide area of ​​the anterior segment.

[0324] Furthermore, in this embodiment, white LEDs can be used instead of blue LEDs as in the invention described in Patent Document 5 (International Publication No. 2018 / 003906), and evaluation can be performed using color information (R signal, G signal, B signal) using a color camera instead of a monochrome camera.

[0325] As described above, according to this embodiment, it is possible to image a wide area of ​​the anterior segment, and in addition to obtaining images of the anterior segment, images of inflammatory cells, and information on the inflammatory state, it is also possible to present and analyze the shape of the anterior segment, which has the advantage of being able to provide a variety of information to doctors. [Explanation of symbols]

[0326] 1000 ophthalmology equipment 1010 Image acquisition unit 1011 Lighting system 1012 Photography 1020 Data Processing Unit 1021 Image selection section 1022 Inflammation state information generation unit 1031 First Segmentation Section 1032 Second Segmentation Section 1033 Cell evaluation information generation processing unit 3000 Ophthalmology equipment 3010 Image acquisition unit 3011 Image Reception Department 3020 Data Processing Unit

Claims

1. an image acquisition unit that acquires a Scheimpflug image of the subject's eye; a data processing unit that executes processing for generating inflammation state information indicating an inflammation state of the subject's eye from the Scheimpflug image; Including, the image acquisition unit includes an imaging system configured to satisfy the Scheimpflug condition, which images the subject's eye, and an illumination system configured to project slit light onto the subject's eye, and scans a three-dimensional region of the subject's eye with the slit light to collect a series of Scheimpflug images; the data processing unit generates the inflammation state information indicating an inflammation state in a three-dimensional region in the anterior chamber of the subject's eye based on the series of Scheimpflug images. Ophthalmology equipment.

2. the inflammation state information includes cell evaluation information that is evaluation information regarding inflammatory cells in the anterior chamber of the subject's eye, the data processing unit executes at least one of a first segmentation for identifying an anterior chamber region corresponding to the anterior chamber, a second segmentation for identifying a cellular region corresponding to the inflammatory cells, and a cell evaluation information generation process for generating the cell evaluation information. The ophthalmic device of claim 1.

3. the data processing unit executes the first segmentation for identifying the anterior chamber region from the Scheimpflug image, the second segmentation for identifying the cellular region from the anterior chamber region identified by the first segmentation, and the cell evaluation information generation process for generating the cell evaluation information from the cellular region identified by the second segmentation. The ophthalmic device of claim 2.

4. the data processing unit performs the first segmentation using a pre-constructed first inference model; the first inference model includes a first neural network constructed by machine learning using training data including at least an eye image; the first neural network is configured to receive an input of a Scheimpflug image and output an anterior chamber region. The ophthalmic apparatus of claim 3.

5. the data processing unit performs the second segmentation using a second inference model constructed in advance; the second inference model includes a second neural network constructed by machine learning using training data including at least an eye image; the second neural network is configured to receive an input of an anterior chamber region in a Scheimpflug image and output a cellular region.

5. The ophthalmic apparatus according to claim 3 or 4.

6. the data processing unit further performs a conversion process of converting the anterior chamber region of the Scheimpflug image into image data having a structure corresponding to the input layer of the second neural network; The second neural network is configured to receive the image data generated by the conversion process as an input and output a cell region. The ophthalmic device of claim 5.

7. the data processing unit executes the cell evaluation information generation process using a third inference model constructed in advance; the third inference model includes a third neural network constructed by machine learning using training data including at least an eye image; the third neural network is configured to receive an input of a cell region in a Scheimpflug image and output cell evaluation information; The ophthalmic device according to any one of claims 3 to 6.

8. the data processing unit executes the second segmentation for identifying the cell region from the Scheimpflug image, and the cell evaluation information generation process for generating the cell evaluation information from the cell region identified by the second segmentation. The ophthalmic device of claim 2.

9. the data processing unit performs the second segmentation using a fourth inference model constructed in advance; the fourth inference model includes a fourth neural network constructed by machine learning using training data including at least an eye image; the fourth neural network is configured to receive a Scheimpflug image as an input and output a cell region. The ophthalmic device of claim 8.

10. the data processing unit executes the cell evaluation information generation process using a fifth inference model constructed in advance; the fifth inference model includes a fifth neural network constructed by machine learning using training data including at least an eye image; The fifth neural network is configured to receive an input of a cell region in a Scheimpflug image and output cell evaluation information.

10. The ophthalmic apparatus according to claim 8 or 9.

11. the data processing unit executes the cell evaluation information generation process for generating the cell evaluation information from the Scheimpflug image. The ophthalmic device of claim 2.

12. the data processing unit executes the cell evaluation information generation process using a sixth inference model constructed in advance; the sixth inference model includes a sixth neural network constructed by machine learning using training data including at least an eye image; the sixth neural network is configured to receive an input of a Scheimpflug image and output cell evaluation information; The ophthalmic device of claim 11.

13. the data processing unit executes the first segmentation for identifying the anterior chamber region from the Scheimpflug image, and the cell evaluation information generation process for generating the cell evaluation information from the anterior chamber region identified by the first segmentation. The ophthalmic device of claim 2.

14. the data processing unit performs the first segmentation using a seventh inference model constructed in advance; the seventh inference model includes a seventh neural network constructed by machine learning using training data including at least an eye image; the seventh neural network is configured to receive an input of a Scheimpflug image and output an anterior chamber region. The ophthalmic device of claim 13.

15. the data processing unit executes the cell evaluation information generation process using a pre-constructed eighth inference model; the eighth inference model includes an eighth neural network constructed by machine learning using training data including at least an eye image; the eighth neural network is configured to receive an input of an anterior chamber region in a Scheimpflug image and output cell evaluation information; 15. An ophthalmic apparatus according to claim 13 or 14.

16. the data processing unit further performs a conversion process of converting the anterior chamber region of the Scheimpflug image into image data having a structure corresponding to the input layer of the eighth neural network; the eighth neural network is configured to receive the image data generated by the conversion processing as input and output cell evaluation information. The ophthalmic device of claim 15.

17. the data processing unit executes the first segmentation for specifying the anterior chamber region by applying a first analysis process to the Scheimpflug image, the second segmentation for specifying the cellular region by applying a second analysis process to the anterior chamber region specified by the first segmentation, and the cell evaluation information generation process for generating the cell evaluation information by applying a third analysis process to the cellular region specified by the second segmentation. The ophthalmic device of claim 2.

18. the data processing unit, in the third analysis process, identifies a partial region of the anterior chamber region identified by the first segmentation, calculates the number of the cellular regions belonging to the partial region, and calculates the density of the inflammatory cells based on the number and a dimension of the partial region; The cell evaluation information includes the density.

18. The ophthalmic device of claim 17.

19. the data processing unit processes the series of Scheimpflug images to generate processed image data, and generates the inflammation state information from the processed image data. An ophthalmic device according to any one of claims 1 to 18.

20. the imaging system includes a first imaging system and a second imaging system that image the subject's eye from different directions, An ophthalmic device according to any one of claims 1 to 19.

21. an optical axis of the first imaging system and an optical axis of the second imaging system are arranged to be inclined in opposite directions with respect to an optical axis of the illumination system; the data processing unit selects one of a first Scheimpflug image acquired by the first imaging system and a second Scheimpflug image acquired by the second imaging system, and generates the inflammation state information based on the selected Scheimpflug image. The ophthalmic device of claim 20.

22. the data processing unit selects a Scheimpflug image that does not include a corneal reflection artifact from among the first Scheimpflug image and the second Scheimpflug image.

22. The ophthalmic device of claim 21.

23. the image acquisition unit includes an image reception unit that receives the Scheimpflug image acquired in advance. An ophthalmic device according to any one of claims 1 to 22.

24. A method for controlling an ophthalmic apparatus including a processor and an image acquisition unit that acquires a Scheimpflug image of a subject's eye, comprising: the image acquisition unit includes an imaging system configured to satisfy the Scheimpflug condition, the imaging system capturing an image of the subject's eye, and an illumination system projecting slit light onto the subject's eye; causing the image acquisition unit to scan a three-dimensional region of the subject's eye with the slit light to collect a series of Scheimpflug images; causing the processor to perform processing for generating inflammation state information indicating an inflammation state in a three-dimensional region within the anterior chamber of the subject's eye based on the series of Scheimpflug images; method.

25. A program that causes a computer to execute the method of claim 24.

26. A computer-readable non-transitory recording medium on which the program of claim 25 is recorded.

Images