Information processing system, information processing method, and program

Abstract

[Problem] To make it possible to easily measure refractive power. [Solution] An information processing system characterized by comprising: a learning model storage unit that stores a learning model trained through machine learning with images of eyes each captured with the cornea irradiated with light emitted and refraction values of the eyes; an image acquisition unit that acquires a captured image of an eye of a user, which is captured by a user terminal while emitting light; and an estimation processing unit that feeds the acquired captured image into the learning model and estimates the refraction value of the user.

Description

Information processing system, information processing method and program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] In Patent Document 1, the refractive power of the eye is measured using photorefraction.

[0003] U.S. Patent No. 9,408,535

[0004] To provide a technique for measuring eye refractive power more easily than with conventional eye refractive power measuring devices.

[0005] The main invention of the present invention for solving the above problem is an information processing system comprising: a learning model memory unit that stores a learning model that has been trained through machine learning from an image of an eye photographed with a light emitted and irradiated onto the cornea and the refraction value of the eye; an image acquisition unit that acquires an image of a user's eye photographed with a light emitted by a user terminal; and an estimation processing unit that provides the acquired image to the learning model and estimates the refraction value of the user.

[0006] Other problems and solutions disclosed in this application will be made clear in the section on preferred embodiments of the invention and the drawings.

[0007] According to the present invention, refractive power can be easily measured.

[0008] FIG. 1 is a diagram illustrating an example of the overall configuration of a refractive power measurement system, which is an information processing system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating the state of measurement in this embodiment. FIG. 3 is a diagram illustrating a user's eye 20 included in a captured image. FIG. 4 is a diagram illustrating an example of the hardware configuration of a management server 2. FIG. 5 is a diagram illustrating an example of the software configuration of the management server 2. FIG. 6 is a diagram illustrating the flow of a learning process. FIG. 7 is a diagram illustrating the flow of a refraction value prediction process.

[0009] <Summary of the Invention> The details of embodiments of the present invention will be listed and described below. The present invention, for example, comprises the following configuration: [Item 1] An information processing system comprising: a learning model storage unit that stores a learning model trained by machine learning on images of an eye photographed with a light emitted and irradiated onto the cornea and the refraction value of the eye; an image acquisition unit that acquires images of a user's eye photographed with a light emitted by a user terminal; and an estimation processing unit that provides the acquired images to the learning model to estimate the refraction value of the user. [Item 2] The information processing system according to item 1, wherein the refraction value includes at least one of spherical power, astigmatism power, and astigmatism axis. [Item 3] The information processing system according to item 1, wherein the learning model has learned the light area and the refraction value included in the image, and the information processing system includes a light extraction unit that extracts a light area where the light is included from the captured image, and the estimation processing unit provides the light area to the learning model to estimate the refraction value. [Item 4] The information processing system according to item 1, wherein the learning model has learned the cornea area and the refraction value included in the image captured so that the light is on the cornea, and the information processing system includes a cornea extraction unit that extracts a cornea area where the cornea is included from the captured image, and the estimation processing unit provides the cornea area to the learning model to estimate the refraction value. [Item 5] An information processing method characterized in that a computer that stores a learning model that has been trained by machine learning from images of human eyes photographed with a light turned on and the refraction value of the eyes executes the following steps: acquiring images of a user's eyes photographed with a light turned on by a user terminal; and providing the acquired images to the learning model to estimate the refraction value of the user.[Item 6] A program for causing a computer that stores a learning model that has been trained by machine learning on images of human eyes photographed with a light turned on and the refraction value of the eyes, to execute the steps of: acquiring images of a user's eyes photographed with a light turned on by a user terminal; and providing the acquired images to the learning model to estimate the refraction value of the user.

[0010] <System Overview> Fig. 1 is a diagram showing an example of the overall configuration of a refractive power measurement system, which is an information processing system according to one embodiment of the present invention. The refractive power measurement system of this embodiment is configured to include a management server 2. The management server 2 is communicably connected to a user terminal 1 via a communication network. The communication network is, for example, the Internet, and is constructed using a public telephone network, a mobile phone network, a wireless communication path, Ethernet (registered trademark), etc.

[0011] The user terminal 1 is a computer operated by a user. The user terminal 1 may be, for example, a smartphone, a tablet computer, or a personal computer. In this embodiment, the user terminal 1 is assumed to be a smartphone or a tablet computer equipped with a camera (not shown) and a flashlight (not shown).

[0012] The management server 2 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0013] The refractive power measurement system of this embodiment estimates the refractive power of a user's eyes. Fig. 2 is a diagram illustrating the measurement process in this embodiment. In the refractive power measurement system of this embodiment, a flashlight of the user terminal 1 is turned on so that light from the flashlight is irradiated onto the cornea (iris of the eye), and an image of the user (which may be the upper body, the face, or just the eyes) is captured with the camera of the user terminal 1, so that the captured image includes light reflected by the cornea.

[0014] FIG. 3 is a diagram illustrating a user's eye 20 included in a captured image. Reflected light 22 from a flashlight is captured on a cornea 21 in the eye 20. The shape of the flashlight is assumed to be circular. The shape of this reflected light 22 may be deformed into an ellipse, as shown in FIG. 3. The specific deformation varies depending on the refractive index.

[0015] In the refractive power measurement system of this embodiment, a value indicating the refractive power of a user's eye (refractive value) is measured using a general measuring device. The general measuring device can measure the refractive power by, for example, irradiating light using a photorefraction method and detecting light reflected from the fundus. A learning model is created by machine learning using the refractive value measured using the general measuring device and an image of light reflected from the cornea 21. The image of reflected light may be an image of the user's entire body, an image of the user's upper body, an image of the user's face, or an image of the user's eye. The image of reflected light may also be an image of the user with the eye or cornea extracted from an image of the user. In the refractive power measurement system, the user's refractive value can be estimated by providing the learning model with an image of the user photographed with a flashlight aimed at their eye.

[0016] <Management Server> FIG. 4 is a diagram illustrating an example of the hardware configuration of the management server 2. Note that the illustrated configuration is an example, and other configurations may also be used. The management server 2 includes a CPU 201, memory 202, storage device 203, communication interface 204, input device 205, and output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, solid state drive, or flash memory. The communication interface 204 is an interface for connecting to a communication network, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 is used to input data, and is, for example, a keyboard, mouse, touch panel, button, microphone, etc. The output device 206 is used to output data, and is, for example, a display, printer, speaker, etc. Each functional unit of the management server 2 described below is realized by the CPU 201 reading a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit of the management server 2 is realized as part of the storage area provided by the memory 202 and the storage device 203.

[0017] 5 is a diagram illustrating an example of the software configuration of the management server 2. The management server 2 includes a learning model storage unit 231, an image acquisition unit 211, an estimation processing unit 212, a light extraction unit 213, and a cornea extraction unit 214.

[0018] <Storage Unit> The learning model storage unit 231 stores a learning model that has been trained by machine learning using captured images of a person and the refraction value of the person's eyes. As described above, the captured image is an image captured by activating the flashlight of the user terminal 1 so that the light from the flashlight is irradiated onto the cornea of ​​the person, and then capturing an image of the person with the camera of the user terminal 1 so that the captured image includes light reflected from the cornea. The refraction value of the eye includes at least one of spherical power, astigmatism power, and astigmatism axis.

[0019] The learning model may be created by learning, through machine learning, the area of ​​reflected light from a flashlight (hereinafter referred to as a light area) included in a photographed image and the refractive value.The learning model may also be created by learning, through machine learning, the area of ​​the cornea (hereinafter referred to as a corneal area) included in a photographed image and the refractive value.

[0020] <Functional Unit> The image acquisition unit 211 acquires a captured image of the user captured by the user terminal 1 using a flashlight.

[0021] The estimation processing unit 212 provides the acquired photographed image to a learning model to estimate the refraction value of the user.

[0022] The light extraction unit 213 can extract a light area where a flashlight is captured from the captured image. The estimation processing unit 212 can provide the light area to a learning model to estimate a refraction value.

[0023] The cornea extraction unit 214 can extract a corneal region showing the cornea from the captured image. The estimation processing unit 212 can provide the corneal region to a learning model to estimate the refractive index. If the cornea extraction unit 214 cannot extract the corneal region from the captured image (for example, if the eyes are closed), the image acquisition unit 211 may send a message to the user terminal 1 instructing it to capture the image again, so as to reacquire the captured image.

[0024] The light extraction unit 213 may extract a light region from the corneal region of the captured image. If the light extraction unit 213 cannot extract a light region from the captured image (for example, if the flashlight is not emitted or the light from the flashlight is not irradiated onto the corneal region), the image acquisition unit 211 may send a message to the user terminal 1 instructing the user to capture the image again, and reacquire the captured image.

[0025] <Operation> Fig. 6 is a diagram illustrating the flow of the learning process. The learning model may be created by the user terminal 1 or the management server 2, or by another computer.

[0026] First, the refraction value of the subject is measured using a general measuring device such as an autorefractometer (S301). Next, the flashlight of the user terminal 1 (smartphone) is turned on and illuminated to the subject's eyes, while the subject is photographed with the camera of the user terminal 1 (S302). The measured refraction value and the photographed image are associated and stored in a computer (S303). The computer then uses machine learning to learn the pair of refraction value and photographed image, creating a learning model that infers the refraction value from the photographed image (S304).

[0027] The learning model created in this manner is stored in the learning model storage unit 231 of the management server 2.

[0028] FIG. 7 is a diagram illustrating the flow of the refraction value prediction process.

[0029] The user terminal 1 emits a flashlight (S321), photographs the user (S322), and transmits the photographed image to the management server 2 (S323). Here, it is assumed that the user terminal 1 is positioned so that light emitted from the flashlight is reflected by the cornea and the reflected light appears in the photographed image.

[0030] The management server 2 receives the photographed image from the user terminal 1 (S324), provides the received photographed image to the learning model stored in the learning model storage unit 231 to infer the refractive value (S325), and outputs the refractive value (for example, by sending the refractive value to the user terminal 1 to display it, or by sending the refractive value to the doctor's terminal to display it) (S326). Note that an image of the eye or cornea extracted from the photographed image may be provided to the learning model.

[0031] The management server 2 may identify the corneal region from the received photographed image, and infer the refraction value by providing a learning model with an extracted image obtained by extracting the corneal region from the photographed image or with the photographed image and information specifying the corneal region. The management server 2 may also identify the light region from the received photographed image, and infer the refraction value by providing a learning model with an extracted image obtained by extracting the light region from the photographed image or with the photographed image and information specifying the light region.

[0032] As described above, the refractive power measurement system of this embodiment allows a user's refractive power to be estimated simply by using a user terminal 1 such as a smartphone to take a picture of the user with a flashlight and providing the captured image to the learning model. Therefore, the user's refractive power can be easily measured.

[0033] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0034] For example, in this embodiment, a flashlight is irradiated onto the user's eyes, but this is not limiting. For example, the display of the user terminal 1 may be illuminated, and an in-camera (not shown) may be used to photograph the user. In this case, a guide for aligning the position of the user's face (or eyes) may be displayed on the display of the user terminal 1, thereby fixing the position of the face (or eyes). The user terminal 1 can control a part of the display to emit bright white light, so that this light is irradiated onto the user's eyes.

[0035] In addition, although the flashlight is circular in this embodiment, it may be shaped like a cross, for example. For example, a bright white cross may be displayed on the display, and the user may be photographed using the front camera after the cross is positioned within the cornea.

[0036] 1 User terminal 2 Management server

Claims

1. An information processing system comprising: a learning model memory unit that stores a learning model trained by machine learning on an image of an eye, including light reflected from the cornea, when a light is emitted and shone on the cornea, and the refractive value of the eye; an image acquisition unit that acquires an image of a user's eye, when a user terminal emits light and shone on the user's cornea, and an estimation processing unit that provides the acquired image to the learning model and estimates the refractive value of the user.

2. An information processing system according to claim 1, characterized in that the refractive value includes at least one of a spherical power, a cylindrical power, and an axis of the cylindrical power.

3. An information processing system as described in claim 1, wherein the learning model has learned the area in which the light is reflected and the refraction value contained in the image, the information processing system further comprising a light extraction unit that extracts the area in which the light is reflected from the captured image, and the estimation processing unit provides the extracted area to the learning model to estimate the refraction value.

4. An information processing system as described in claim 1, wherein the learning model has been trained to learn the area of ​​the cornea and the refractive value contained in the image captured so that the light falls on the cornea, and the information processing system further comprises a cornea extraction unit that extracts a corneal area in which the cornea is shown from the captured image, and the estimation processing unit provides the corneal area to the learning model to estimate the refractive value.

5. An information processing method, characterized in that a computer executes the steps of: acquiring an image of a user's eye while a user terminal emits light and shines it on the user's cornea; and estimating the refraction value of the user by providing the acquired image to a learning model that has been trained by machine learning on an image of the eye, including the light reflected from the cornea while the light is emitted and shined on the cornea, and the refraction value of the eye.

6. A program for causing a computer to execute the steps of: acquiring an image of a user's eye while a user terminal emits light and shines it on the user's cornea; and estimating the refraction value of the user by providing the acquired image to a learning model that has been trained by machine learning on an image of the eye, including the light reflected from the cornea while the light is emitted and shined on the cornea, and the refraction value of the eye.

Images