Information processing device, information processing method, and program

Abstract

An information processing device (1) comprises: an eye state determination unit (30) that determines the state of the eyes of a user by inputting an eye image (P2) of the user to a first trained model (31) generated in advance by machine learning on the basis of training data obtained by adding annotations indicating the state of the eyes to each of a plurality of eye images acquired in advance; and a score unit (40) that scores the state of the eyes of the user on the basis of the state of the eyes of the user determined by the eye state determination unit (30).

Description

Information Processing Apparatus, Information Processing Method, and Program

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program, and particularly to an information processing apparatus, an information processing method, and a program capable of determining the state of eyes using AI (Artificial Intelligence) technology.

[0002] In recent years, with the increase in the number of users who use VDTs (Visual Display Terminals) such as smartphones or personal computers, the number of users complaining of eye problems has been increasing. In order to improve their eye problems, users visit an ophthalmologist or instill eye drops.

[0003] Eye problems include various symptoms such as eye fatigue (such as eye strain and mental eye fatigue), blurred vision, itching, or dry eyes. However, since it is difficult for users to accurately grasp the state of their own eyes, it is difficult for users to accurately identify the causes of their eye problems by themselves. Therefore, conventionally, an ophthalmic diagnosis support apparatus capable of identifying the causes of eye problems has been proposed (for example, Patent Document 1).

[0004] Japanese Unexamined Patent Application Publication No. 2020-36835

[0005] However, conventional ophthalmic diagnosis support apparatuses are used in medical institutions, and it is difficult for users themselves to accurately and easily grasp the state of their own eyes. Therefore, there are many cases where users visit an ophthalmologist after the eye disease or symptoms have worsened.

[0006] The present invention has been made to solve such problems, and an object thereof is to provide an information processing apparatus or the like that can easily and appropriately grasp the state of a user's eyes.

[0007] To achieve the above objective, one aspect of the information processing apparatus according to the present invention includes: an eye state determination unit that determines the state of a user's eyes when a user's eye image is input to a first learning model that has been pre-generated by machine learning based on training data obtained by assigning annotations indicating the state of the eyes to each of a plurality of eye images acquired in advance; and a scoring unit that scores the state of the user's eyes based on the state of the user's eyes determined by the eye state determination unit, wherein the eye image is an image that includes only the eye and the peripheral portion of the eye.

[0008] Furthermore, one aspect of the information processing method according to the present invention includes an eye state determination step in which the user's eye image is input to a first learning model generated by machine learning based on training data obtained by assigning annotations indicating the state of the eye to each of a plurality of eye images acquired in advance, thereby determining the state of the user's eye, and a scoring step in which the user's eye state is scored based on the state of the user's eye determined in the eye state determination step.

[0009] Furthermore, one aspect of the program according to the present invention is a program that causes a computer to execute the above-described information processing method.

[0010] According to the present invention, the condition of a user's eyes can be easily and appropriately assessed. This makes it easy to identify users who require self-care for their eyes, allowing them to treat their own eyes before eye diseases or symptoms become severe.

[0011] Figure 1 is a block diagram showing the configuration of an information processing device according to Embodiment 1. Figure 2 is a diagram showing an example of a face image and an eye image. Figure 3 is a flowchart of the information processing method according to Embodiment 1. Figure 4 is a diagram showing an example of a screen displayed on the display unit of the information processing device according to Embodiment 1. Figure 5 is a block diagram showing the configuration of an information processing device according to Embodiment 2. Figure 6 is a flowchart of the information processing method according to Embodiment 2. Figure 7 is a diagram showing an example of a screen displayed on the display unit of the information processing device according to Embodiment 2. Figure 8 is a block diagram showing the configuration of an information processing device according to a modified example of Embodiment 2.

[0012] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below are all general or specific examples of the present invention. Therefore, the numerical values, shapes, materials, components, arrangement positions and connection configurations of components, as well as the steps and order of steps shown in the following embodiments, are examples only and are not intended to limit the present invention. Accordingly, among the components in the following embodiments, those not described in an independent claim will be described as optional components.

[0013] Furthermore, each figure is a schematic diagram and not necessarily a strictly accurate representation. In each figure, substantially identical components are denoted by the same reference numerals, and redundant explanations are omitted or simplified.

[0014] (Embodiment 1) An information processing device 1 according to Embodiment 1 will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the information processing device 1 according to Embodiment 1.

[0015] The information processing device 1 shown in Figure 1 is a device that determines the condition of the user's eyes from the user's eye image and scores the condition of the user's eyes. The information processing device 1 is, for example, a computer with a CPU or a mobile terminal such as a smartphone.

[0016] As shown in Figure 1, the information processing device 1 comprises a face image acquisition unit 10, an eye image extraction unit 20, an eye state determination unit 30, a scoring unit 40, and a display unit 50. The eye image extraction unit 20, the eye state determination unit 30, and the scoring unit 40 are control units in the information processing device 1. In other words, the processing in each of the eye image extraction unit 20, the eye state determination unit 30, and the scoring unit 40 is executed by the control unit.

[0017] The face image acquisition unit 10 acquires a face image of the user. The face image is an image that includes the user's face. As shown in Figure 2, the face image P1 is, for example, an image that includes the entire face of the user from the neck up. However, the face image P1 is not limited to this; it may be an image of the user's whole body as long as it includes the face. In addition, the face image P1 includes at least the user's eyes.

[0018] The face image acquisition unit 10 is an imaging device that acquires data of a user's face image P1 by, for example, imaging the user. For example, the face image acquisition unit 10 is a camera. If the information processing device 1 is a smartphone, the face image acquisition unit 10 is a camera mounted on the smartphone.

[0019] Furthermore, the face image acquisition unit 10 may acquire the face image P1 not by taking an image itself, but by receiving the face image P1 taken by another imaging device. For example, the face image acquisition unit 10 may be a communication unit that wirelessly receives data of the face image P1 taken by another imaging device, or it may be a USB port or media slot to which a recording medium containing data of the face image P1 taken by another imaging device can be detachably attached. In this way, the face image acquisition unit 10 is an interface to the information processing device 1.

[0020] The face image P1 acquired by the face image acquisition unit 10 is input to the eye image extraction unit 20. In other words, the face image P1 becomes input data to the eye image extraction unit 20. As shown in Figure 1, the eye image extraction unit 20 extracts the eye image P2 from the face image P1 acquired by the face image acquisition unit 10. The eye image P2 is an image that includes only the eye and the surrounding area of ​​the eye (a peripheral eye image). In other words, the eye image P2 is not an image of the eye itself (i.e., the entire surface of the eye from the inner corner to the outer corner), but an image consisting of the eye and the surrounding area of ​​the eye, as shown in Figure 2. The peripheral area of ​​the eye is the part that exists around the eye. The peripheral area of ​​the eye includes, for example, the upper eyelid and lower eyelid, but does not include the eyebrows. The eye image P2 may include dark circles under the eyes, but does not include the entire dark circles under the eyes. Also, the eye image P2 is a rectangular image as an example, but is not limited to this.

[0021] The eye image extraction unit 20, for example, identifies the position of the eyes from the face image P1 input to the face image acquisition unit 10, and extracts the eye image P2 by cutting out the eye image P2 from the face image P1. In this case, the eye image extraction unit 20 may use machine learning to extract the eye image P2 from the face image P1. When using machine learning to extract the eye image P2 from the face image P1, the eye image P2 can be extracted from the face image P1 by using a learning model that has been pre-trained to determine the position of the eyes in the face image using machine learning.

[0022] The eye image P2 extracted by the eye image extraction unit 20 is input to the eye state determination unit 30. In other words, the eye image P2 becomes input data to the eye state determination unit 30.

[0023] The eye condition determination unit 30 determines the user's eye condition based on the eye image P2 extracted by the eye image extraction unit 20. Specifically, the eye condition determination unit 30 has a first learning model 31, and the user's eye condition is determined when the eye image P2 extracted by the eye image extraction unit 20 is input to the first learning model 31 of the eye condition determination unit 30. The first learning model 31 is stored, for example, in a storage medium such as memory mounted on the information processing device 1.

[0024] The first learning model 31 in the eye state determination unit 30 is pre-generated by machine learning based on training data obtained by adding annotations indicating the state of the eye to each of a plurality of previously acquired eye images (images including only the eye and the peripheral part of the eye). In this embodiment, the state of the eye is the degree of eye fatigue. Therefore, the first learning model 31 is generated based on training data obtained by adding annotations indicating the degree of eye fatigue to each of a plurality of eye images. Hereinafter, an example of how to generate the first learning model 31 is described below.

[0025] The training data that forms the basis of the first learning model 31 is obtained from eye images of multiple eye patients who complain of eye problems. In this case, first, multiple eye images are obtained by imaging the area around each eye of multiple eye patients. Then, the multiple eye images obtained are labeled. Specifically, for each of the multiple eye images obtained, the patient's known degree of eye fatigue is added as an annotation. For example, the degree of eye fatigue is divided into multiple stages, and one of the stages of fatigue is added as an annotation to each of the multiple eye images. In this way, training data (a labeled dataset) is obtained in which the degree of eye fatigue is added as an annotation to each of the multiple eye images. In other words, the training data is data in which the degree of eye fatigue is tagged for each eye image.

[0026] Then, based on this training data, a first learning model 31 is generated by machine learning. In other words, the first learning model 31 is a pre-trained model generated by machine learning using a predetermined algorithm. The first learning model 31 includes, for example, a neural network. Specifically, the first learning model 31 includes a CNN (Convolutional Neural Network). The first learning model 31 may be a single model or an ensemble model consisting of multiple AI groups. If the first learning model 31 is composed of multiple AI groups, it is preferable to use AI groups with high accuracy (for example, accuracy of 60% or more). If the first learning model 31 is composed of multiple AI groups, for example, four AI groups with high accuracy can be used.

[0027] The eye condition determination unit 30 analyzes the eye image P2 extracted by the eye image extraction unit 20 using the first learning model 31 that was generated in advance. Based on this, the eye condition determination unit 30 determines the degree of eye fatigue of the user based on the eye image P2 extracted by the eye image extraction unit 20. In this case, the eye condition determination unit 30 may combine EfficientNet as the image recognition model. This allows for accurate determination of the user's eye condition.

[0028] The scoring unit 40 scores the user's eye condition based on the eye condition determination unit 30. In this embodiment, the eye condition is the degree of eye fatigue, so the scoring unit 40 scores the user's eye fatigue based on the degree of eye fatigue determined by the eye condition determination unit 30. In other words, the scoring unit 40 calculates the user's eye fatigue as a score (eye fatigue score). For example, the scoring unit 40 calculates a score indicating the user's eye fatigue as a score out of 100 points. In this case, the score (score) indicating the user's eye fatigue may be in increments of 1 point, 5 points, or 10 points. The score indicating the user's eye fatigue indicates the degree of eye damage.

[0029] Furthermore, the score indicating the user's eye fatigue level may be a graded evaluation rather than a numerical score. In this case, the score indicating the user's eye fatigue level may be a three-level evaluation such as A, B, C, or a five-level evaluation such as A, B, C, D, E, etc. In the case of a three-level evaluation, the degree of eye fatigue can be divided into mild, moderate, and severe. Also, when evaluating with multiple levels, the score calculated on a 100-point scale may be divided into multiple levels. Conversely, multiple levels calculated by graded evaluation may be converted into a score on a 100-point scale using a function-based algorithm.

[0030] The display unit 50 displays a score indicating the user's eye condition, which has been scored by the scoring unit 40. In this embodiment, the display unit 50 displays a score indicating the user's degree of eye fatigue, which has been scored by the scoring unit 40. For example, the display unit 50 displays a score calculated by the scoring unit 40 (e.g., "80 points") or a graded rating (e.g., "A") as a score indicating the user's degree of eye fatigue.

[0031] In addition, the display unit 50 may also display information other than the score. For example, the display unit 50 may display the processing results processed by the control unit in the information processing device 1. Specifically, the eye image P2 extracted by the eye image extraction unit 20 may be displayed.

[0032] The display unit 50 is a display device having a display screen. In this embodiment, the display unit 50 is mounted on the information processing device 1 as part of the information processing device 1, but it is not limited to this. In other words, the display unit 50 may be a separate unit from the information processing device 1. In this case, the display unit 50 is a monitor connected to the information processing device 1 wirelessly or by wire.

[0033] Next, the information processing method using the information processing device 1 will be explained with reference to Figures 1 and 2, and with reference to Figure 3. Figure 3 is a flowchart of the information processing method according to Embodiment 1.

[0034] As shown in Figure 3, in the information processing method according to this embodiment, first, a face image P1, which is an image including the user's face, is acquired (face image acquisition step: step S11). Specifically, the face image acquisition unit 10 acquires the user's face image. For example, the face image acquisition unit 10, which is a camera, captures an image of the user to acquire data of the user's face image P1.

[0035] Next, the eye image P2 is extracted from the face image P1 acquired in step S11 (eye image extraction step: step S12). Specifically, the eye image extraction unit 20 extracts the eye image P2 from the face image P1 acquired by the face image acquisition unit 10. For example, as shown in Figure 2, the eye image P2 is cut out from the face image P1 as a horizontal rectangular image including the eye. Specifically, the eye images P2 of the right eye and the left eye are extracted from the face image P1. Note that if the first learning model 31 is generated using the image of the right eye, only the eye image P2 of the right eye may be extracted from the face image P1.

[0036] Next, the eye image P2 extracted in step S12 is input to the first learning model 31 to determine the user's eye condition (eye condition determination step: step S13). Specifically, the eye condition determination unit 30 determines the user's eye condition by inputting the eye image P2 extracted by the eye image extraction unit 20 into the first learning model 31. In this embodiment, the degree of eye fatigue of the user is determined by analyzing the eye image P2 using the first learning model 31.

[0037] Furthermore, if the first learning model 31 is generated using the image of the right eye, and the right eye image P2 has not been extracted, the left eye image P2 may be flipped horizontally and then input to the first learning model 31 as the eye image P2.

[0038] Next, the user's eye condition is scored based on the condition of the user's eyes determined in step S13 (scoring step: step S14). Specifically, the scoring unit 40 scores the user's eye condition based on the condition of the user's eyes determined by the eye condition determination unit 30. In this embodiment, the user's eye fatigue level is scored based on the degree of eye fatigue determined in step S13.

[0039] Next, the score indicating the user's eye condition, which was scored in step S14, is displayed (score display step: step S15). Specifically, the score indicating the user's degree of eye fatigue, which was scored in the scoring unit 40, is displayed on the display unit 50.

[0040] Figure 4 shows an example of a screen displayed on the display unit 50 of the information processing device 1 according to Embodiment 1. In Figure 4, the information processing device 1 is shown as a smartphone.

[0041] As shown in Figure 4, the display unit 50 displays a score (65 points in Figure 4) as a score indicating the user's degree of eye fatigue, calculated based on the eye image P2. The display unit 50 also displays information on eye drops suitable for the user's eye condition from among several types of eye drops. In this case, the control unit of the information processing device 1 selects one eye drop from among several types that can improve the user's eye condition based on the eye condition determination unit 30, and displays that selected eye drop on the display unit 50. In other words, the display unit 50 displays a recommended eye drop suitable for the user's eye condition. Specifically, the display unit 50 displays an image of the recommended eye drop product and its product name. The display unit 50 may also display foods or nutrients suitable for the user's eye condition.

[0042] As described above, the information processing device 1 according to this embodiment includes a face image acquisition unit 10 that acquires a face image P1, an eye image extraction unit 20 that extracts an eye image P2 from the face image P1 acquired by the face image acquisition unit 10, an eye state determination unit 30 that determines the state of the user's eyes when the eye image P2 extracted by the eye image extraction unit 20 is input to a first learning model 31 which is generated by machine learning based on training data obtained by adding annotations indicating the state of the eyes to previously acquired eye images, and a scoring unit 40 that scores the state of the user's eyes based on the state of the user's eyes determined by the eye state determination unit 30.

[0043] As described above, the information processing device 1 according to this embodiment determines the condition of the user's eyes based on an eye image P2 extracted from the user's face image P1, and scores the user's eye condition. This allows the user to easily and appropriately understand the condition of their own eyes. Therefore, it is easy to identify users who need self-care for their eyes, and they can treat their own eyes before eye diseases or symptoms become severe. In other words, they can practice self-medication. As a result, it is also possible to suppress the increase in medical expenses.

[0044] Moreover, by grasping the state of their own eyes, users can easily select and purchase one type of eye drops suitable for themselves from multiple types of eye drops. That is, conventionally, it has been difficult for users to accurately and easily grasp the state of their own eyes. Therefore, even when users try to purchase eye drops to improve eye discomfort, it has been difficult for them to select the eye drops suitable for themselves. In particular, for eye drops, there are many commercially available products to address various symptoms such as tired eyes, blurriness, itching, or dry eyes. For example, there is an eye drop manufacturer that manufactures and sells more than 50 types of eye drops by itself, and in total among multiple eye drop manufacturers, there are more than 100 types of commercially available eye drops. For this reason, even if users can grasp the state of their eyes to some extent from their self-awareness symptoms, it has been extremely difficult for them to select the most suitable eye drops for themselves from the numerous eye drops. In contrast, by using the information processing device 1 according to the present embodiment, users can easily and appropriately grasp the state of their own eyes. Therefore, even if there are more than 100 types of eye drops, users can easily select and purchase one type of eye drops suitable for themselves from the multiple types of eye drops.

[0045] In addition, as pharmaceuticals for treating eye discomfort (such as eye fatigue like eye tiredness and eye strain, blurriness, itching, or dry eyes), there are not only eye drops but also oral medications. Therefore, by grasping the state of the eyes using the information processing device 1 according to the present embodiment, users can accurately and easily select suitable ones from various pharmaceuticals including oral medications.

[0046] Also, in the information processing device 1 according to the present embodiment, the degree of eye fatigue of the user is judged from the eye image P2 extracted from the face image P1 of the user, and the state of the user's eyes is scored.

[0047] As a result, users can easily and appropriately grasp the degree of their own eye fatigue. Therefore, users can perform appropriate self-medication according to the degree of eye fatigue.

[0048] Further, in the information processing device 1 according to the present embodiment, a display unit 50 for displaying a score indicating the state of the user's eyes scored by the score unit 40 is provided.

[0049] With this configuration, the user can visually and easily recognize the fatigue level of their own eyes.

[0050] At this time, not only the score but also products and / or messages suitable for the user's eye condition may be displayed on the display unit 50.

[0051] Specifically, useful products for the user may be displayed on the display unit 50 by characters and / or images. For example, pharmaceuticals such as recommended eye drops or oral medications may be displayed on the display unit 50. Thereby, the user can easily purchase pharmaceuticals such as eye drops or oral medications suitable for themselves. Also, foods and / or nutrients suitable for the user's eye condition may be displayed on the display unit 50. Thereby, the user can purchase foods suitable for themselves or purchase food ingredients or supplements containing nutrients suitable for themselves. Also, messages recommending glasses suitable for the user or recommending a change in the power of contact lenses may be displayed on the display unit 50. Thereby, the user can purchase glasses suitable for themselves or purchase contact lenses suitable for themselves.

[0052] Also, useful messages for the user may be displayed on the display unit 50 by characters / or images. For example, messages recommending eye rest or recommending a visit to a hospital such as an ophthalmology department may be displayed on the display unit 50. Thereby, the user can rest their eyes or visit an ophthalmology department. Also, methods of eye massage etc. may be displayed on the display unit 50. Thereby, the user can massage their eyes to relieve eye fatigue. Also, character information such as "Don't overwatch your smartphone" or "Get enough sleep" may be displayed on the display unit 50.

[0053] Thus, by displaying products or messages useful to the user on the display unit 50, various advices can be given to the user.

[0054] (Embodiment 2) Next, the information processing device 2 according to Embodiment 2 will be described with reference to Figure 5. Figure 5 is a block diagram showing the configuration of the information processing device 2 according to Embodiment 2.

[0055] As shown in Figure 5, the information processing device 2 according to this embodiment further includes an age acquisition unit 60, a medical questionnaire response acquisition unit 70, and an age estimation unit 80, in addition to the information processing device 1 according to the first embodiment.

[0056] The age acquisition unit 60 acquires the actual age of the user (target person) from whom the facial image P1 is acquired. In other words, the age acquisition unit 60 is a reception unit that accepts the user's actual age.

[0057] Furthermore, the medical questionnaire response acquisition unit 70 acquires the answers to the eye questionnaire of the user (subject) from whom the facial image P1 is acquired. In other words, the medical questionnaire response acquisition unit 70 is a reception unit that accepts the answers to the user's eye questionnaire.

[0058] The age acquisition unit 60 and the medical questionnaire response acquisition unit 70 are the user interface of the information processing device 2, and are, for example, a touch panel, keyboard, numeric keypad, or mouse. If the information processing device 2 is a smartphone, the age acquisition unit 60 and the medical questionnaire response acquisition unit 70 are the smartphone's display screen (touch panel). In this case, the user operates the smartphone's display screen to input their actual age. This allows the information processing device 2 to acquire the user's actual age. The user also operates the smartphone's display screen to answer the medical questionnaire displayed on the screen. This allows the information processing device 2 to acquire the user's answers to the eye questionnaire.

[0059] The eye examination questionnaire includes at least one item selected from the group consisting of the degree of blurred vision, the degree of dry eyes, the degree of digital stress, the degree of drowsiness, and the degree of glare. In this embodiment, the eye examination questionnaire includes five items: the degree of blurred vision, the degree of dry eyes, the degree of digital stress, the degree of drowsiness, and the degree of glare. Digital stress refers to the stress placed on the eyes when watching television or using personal computers or mobile devices such as smartphones.

[0060] These five factors contribute significantly to the degree of eye fatigue (especially tired eyes) in humans. The fact that these five factors contribute significantly to the degree of eye fatigue was discovered through experiments conducted by the inventors prior to filing this application. Specifically, data was collected from approximately 230 eye patients aged 20 to 80 years, with an equal ratio of men and women and age groups, by conducting various eye examinations and asking questions about subjective symptoms of eye strain. The contribution of the factors to eye fatigue was then examined. The examination items included microscopic measurement of blood flow around the eye, visual acuity, contrast sensitivity, intraocular pressure, aberrations, focusing ability, and fine adjustment of focus.

[0061] As a result, we found that among the many factors related to eye fatigue, five factors—the degree of blurred vision, the degree of dry eyes, the degree of digital stress, the degree of drowsiness, and the degree of glare—contributed significantly more to eye fatigue than other factors, and that their contributions increased in this order. In other words, among these five, the degree of blurred vision had the highest contribution, and the degree of glare had the lowest contribution. In particular, among these five, the degree of blurred vision, the degree of dry eyes, and the degree of digital stress contributed significantly, with these three factors accounting for approximately 80% of the total contribution. Furthermore, among these three factors, the degree of blurred vision and the degree of dry eyes contributed particularly high and similarly.

[0062] When using the information processing device 2, the user answers these five questionnaire items, and in this case, the user answers the five questionnaire items using, for example, a five-point scale. For example, the questionnaire item for the degree of blurred vision is, "Do you see double or have blurred vision?", and the answer options are, "0 Very clear", "1 No particular problem", "2 Slightly blurry", "3 Blurry", and "4 Very blurry". The questionnaire item for the degree of dry eye is, "Do your eyes feel dry?", and the answer options are, "0 No", "1 Occasionally", "2 Sometimes", "3 Often", and "4 Always". The questionnaire item for the degree of digital stress is, "Do your eye symptoms worsen when you are watching TV or using a computer or mobile phone?", and the answer options are, "0 No", "1 Occasionally", "2 Sometimes", "3 Often", and "4 Always". Furthermore, the questionnaire item regarding the degree of sleepiness is "Do you feel sleepy?", and the answer options are "0 I am wide awake", "1 Not particularly", "2 A little sleepy", "3 Sleepy", and "4 Very sleepy". Additionally, the questionnaire item regarding the degree of sensitivity to light is "Do you find light dazzling?", and the answer options are "0 No", "1 Occasionally", "2 Sometimes", "3 Often", and "4 Always".

[0063] The age estimation unit 80 estimates the user's age based on the eye image P2 extracted by the eye image extraction unit 20. Specifically, the age estimation unit 80 has a second learning model 81, and when the eye image P2 extracted by the eye image extraction unit 20 is input to the second learning model 81 of the age estimation unit 80, the age of the user from whom the face image P1 was taken is estimated. The second learning model 81 is stored, for example, in a storage medium such as a memory mounted on the information processing device 2.

[0064] The second learning model 81 in the age estimation unit 80 is pre-generated by machine learning based on training data obtained by adding annotations indicating the actual age to each of a plurality of eye images (images including only the eye and the surrounding area of ​​the eye) that have been acquired in advance. In other words, the second learning model 81 is a pre-trained model generated by machine learning using a predetermined algorithm. The second learning model 81 includes, for example, a neural network. Specifically, the second learning model 81 includes a CNN. The second learning model 81 may be a single model or an ensemble model consisting of multiple AI groups.

[0065] The age estimation unit 80 analyzes the eye image P2 extracted by the eye image extraction unit 20 using the second learning model 81 that was generated in advance. This allows the age estimation unit 80 to estimate the user's eye age from the eye image P2 extracted by the eye image extraction unit 20.

[0066] The user's actual age obtained by the age acquisition unit 60 and the eye age (estimated age) estimated by the age estimation unit 80 are input to the score unit 40. The score unit 40 then scores the user's eye condition based on the eye age gap (gap from actual age), which is the difference between the user's actual age obtained by the age acquisition unit 60 and the user's eye age (estimated age) estimated by the age estimation unit 80, and the user's eye condition determined by the eye condition determination unit 30. In other words, in this embodiment, the score unit 40 does not score the user's eye condition based solely on the user's eye condition determined by the eye condition determination unit 30, but also takes into account the eye age gap, which is the difference between the user's actual age and the user's eye age (estimated age), when scoring the user's eye condition.

[0067] Furthermore, in this embodiment, the user's eye questionnaire answers obtained by the questionnaire response acquisition unit 70 are also input to the score unit 40. Therefore, the score unit 40 scores the user's eye condition by also considering the user's eye questionnaire answers obtained by the questionnaire response acquisition unit 70. Specifically, the score unit 40 scores the user's eye condition based on the eye age gap, which is the difference between the user's actual age obtained by the age acquisition unit 60 and the user's eye age estimated by the age estimation unit 80, the user's eye condition determined by the eye condition determination unit 30, and the user's eye questionnaire answers obtained by the questionnaire response acquisition unit 70.

[0068] The scoring unit 40 can score the user's eye condition using a function or algorithm that takes three variables: (i) the user's eye condition determined by the eye condition judgment unit 30, (ii) the eye age gap which is the difference between the user's actual age obtained by the age acquisition unit 60 and the user's eye age estimated by the age estimation unit 80, and (iii) the user's answers to the eye questionnaire. In this case, it is advisable to weight the three variables. This can improve the accuracy of the calculated score. Furthermore, if the items in the user's eye questionnaire are the five items mentioned above (degree of blurred vision, degree of dry eyes, degree of digital stress, degree of drowsiness, and degree of glare), these five items should also be added as variables. In this case, it is advisable to weight these five items as well. For example, the weight of the items that contribute significantly to the degree of eye fatigue should be increased. This can further improve the accuracy of the calculated score.

[0069] The scoring unit 40 may, instead of using a function, use a machine learning model to score the user's eye condition based on three factors: (i) the user's eye condition determined by the eye condition determination unit 30, (ii) the eye age gap which is the difference between the user's actual age obtained by the age acquisition unit 60 and the user's eye age estimated by the age estimation unit 80, and (iii) the user's answers to the eye questionnaire.

[0070] Furthermore, the scoring unit 40 may score the user's eye condition based on two factors: (i) the user's eye condition as determined by the eye condition determination unit 30, (ii) the eye age gap which is the difference between the user's actual age obtained by the age acquisition unit 60 and the user's eye age estimated by the age estimation unit 80, and (iii) the user's answers to the eye questionnaire, rather than (i) the user's eye condition as determined by the eye condition determination unit 30 and (ii) the user's answers to the eye questionnaire.

[0071] Next, the information processing method using the information processing device 2 will be explained with reference to Figure 5 and using Figure 6. Figure 6 is a flowchart of the information processing method according to Embodiment 2.

[0072] As shown in Figure 6, in the information processing method according to this embodiment, a face image P1, which is an image including the user's face, is acquired (face image acquisition step: step S11), and an eye image P2 is extracted from the face image P1 acquired in step S11 (eye image extraction step: step S12).

[0073] In this embodiment, the user's actual age is further acquired (age acquisition step: step S21). Specifically, the user inputs their actual age using the age acquisition unit 60.

[0074] Furthermore, the system also obtains the user's answers to an eye questionnaire (questionnaire response acquisition step: step S22). Specifically, the user inputs their answers to a questionnaire about their eyes using the questionnaire response acquisition unit 70. For example, the user inputs answers to five questionnaire items: the degree of blurred vision, the degree of dry eyes, the degree of digital stress, the degree of drowsiness, and the degree of glare.

[0075] Next, similar to Embodiment 1 described above, the eye image P2 extracted in step S12 is input to the first learning model 31 to determine the user's eye condition (eye condition determination step: step S13). Specifically, the user's degree of eye fatigue is determined by analyzing the eye image P2 using the first learning model 31.

[0076] In this embodiment, the eye image P2 extracted in step S12 is further input to the second learning model 81 to estimate the user's eye age (eye age estimation step: step S23). Specifically, the eye age of the user is estimated by analyzing the eye image P2 with the age estimation unit 80 using the second learning model 81.

[0077] Next, the user's eye condition is scored based on the eye age gap, which is the difference between the user's actual age obtained in step S21 and the user's eye age estimated in step S23, the user's responses to the eye questionnaire obtained in step S22, and the user's eye condition determined in step S13 (scoring step: step S14). Specifically, the user's eye fatigue level is scored as part of the user's eye condition.

[0078] Next, similar to the first embodiment described above, a score indicating the user's eye condition, which was scored in step S14, is displayed (score display step: step S15). Specifically, the score indicating the user's degree of eye fatigue, which was scored in the scoring unit 40, is displayed on the display unit 50.

[0079] Figure 7 shows an example of a screen displayed on the display unit 50 of the information processing device 2 according to Embodiment 2. In Figure 7, the information processing device 2 is shown as a smartphone.

[0080] As shown in Figure 7, in this embodiment as well, the display unit 50 displays a score (65 points in Figure 7) as a score indicating the user's degree of eye fatigue, and also displays information on medicines suitable for the user's eye condition from among several types of eye drops, oral medications, and other medicines. In addition, as in the first embodiment described above, the display unit 50 may also display products suitable for the user's eye condition other than medicines, or it may display a message suitable for the user's eye condition.

[0081] In this embodiment, the eye age gap (gap from actual age), which is the difference between the user's actual age obtained by the age acquisition unit 60 and the user's eye age estimated by the age estimation unit 80, is also displayed on the display unit 50. In addition, the results of the user's eye-related questionnaire obtained by the questionnaire response acquisition unit 70 are also displayed on the display unit 50. The questionnaire response results are displayed for the four questionnaire items that have the highest contribution to the degree of eye fatigue out of the five questionnaire items: degree of blurred vision, degree of dry eyes, degree of digital stress, degree of drowsiness, and degree of glare (specifically, the degree of blurred vision, degree of dry eyes, degree of digital stress, and degree of drowsiness).

[0082] Specifically, as shown in Figure 7, the five items—eye age and the results of the four questionnaire questions—are displayed in five stages using a radar chart. By displaying the eye age and the user's answers to the eye questionnaire in this way, the user's eye condition can be easily understood visually. In Figure 7, "eye age" is a score of the eye age gap, which is the difference between the user's actual age and the user's eye age (estimated age). In Figure 7, the eye age gap is divided into five stages and displayed as "eye age". For example, Level 1 indicates an actual age of -9 years or more, Level 2 indicates an actual age of -6 to -8 years, Level 3 indicates an actual age of -5 to +7 years, Level 4 indicates an actual age of +8 to +10 years, and Level 5 indicates an actual age of +11 years or more.

[0083] As described above, the information processing device 2 according to this embodiment also includes a face image acquisition unit 10 that acquires a face image P1, an eye image extraction unit 20 that extracts an eye image P2 from the face image P1 acquired by the face image acquisition unit 10, an eye state determination unit 30 that determines the state of the user's eyes when the eye image P2 extracted by the eye image extraction unit 20 is input to the first learning model 31, and a scoring unit 40 that scores the state of the user's eyes based on the state of the user's eyes determined by the eye state determination unit 30.

[0084] This configuration allows for easy and accurate assessment of the user's eye condition. This makes it easy to identify users who require self-care for their eyes, enabling them to treat their own eyes before eye diseases or symptoms become severe. Furthermore, users can easily select and purchase the most suitable eye drops from a wide variety of options available.

[0085] Furthermore, in the information processing device 2 of this embodiment, the eye condition is not scored solely based on the eye condition determination unit 30, but rather the user's eye condition is scored by adding the user's answers to the eye questionnaire obtained by the questionnaire answer acquisition unit 70 to the user's eye condition determined by the eye condition determination unit 30. This makes it possible to increase the accuracy of the score indicating the user's eye condition calculated by the score unit 40. For example, as in the information processing device 1 of the first embodiment described above, when the eye condition was scored solely based on the user's eye condition determined by the eye condition determination unit 30, the accuracy of the score was about 60%. However, as in the information processing device 2 of this embodiment, when the user's eye condition is scored by considering the user's answers to the eye questionnaire obtained by the questionnaire answer acquisition unit 70 in addition to the user's eye condition determined by the eye condition determination unit 30, the accuracy of the score was 80% or more.

[0086] Furthermore, in the information processing device 2 of this embodiment, the user's eye condition is scored by adding the eye age gap, which is the difference between the user's actual age obtained by the age acquisition unit 60 and the user's eye age estimated by the age estimation unit 80, to the user's eye condition determined by the eye condition determination unit 30 and the user's eye condition answers obtained by the eye condition answer acquisition unit 70. This makes it possible to further improve the accuracy of the score indicating the user's eye condition calculated by the score unit 40.

[0087] Furthermore, as shown in Figure 8, the accuracy of the score calculated by the score unit 40 can be further improved by providing a focus adjustment ability acquisition unit 90 that acquires information on the user's eye's focusing ability, as shown in the information processing device 2A. In this case, the score unit 40 scores the user's eye condition based on the user's eye condition determined by the eye condition determination unit 30, the user's answers to the eye medical questionnaire acquired by the medical questionnaire answer acquisition unit 70, and the information on the user's eye's focusing ability acquired by the focus adjustment ability acquisition unit 90. In Figure 8, the score unit 40 may score the user's eye condition without using the user's answers to the eye medical questionnaire. In other words, the score unit 40 may score the user's eye condition based on the user's eye condition determined by the eye condition determination unit 30 and the information on the user's eye's focusing ability acquired by the focus adjustment ability acquisition unit 90. The focus adjustment ability acquisition unit 90 is, for example, a focus adjustment ability measuring device that can measure the user's eye's focusing ability.

[0088] Furthermore, in this embodiment, the degree of eye fatigue in the eye region (the region corresponding to the eye image P2), consisting of the eye and its surrounding area, may be visualized and displayed visually using a heat map, either by using only the eye image P2 extracted by the eye image extraction unit 20, or by using the eye image P2 and the user's eye questionnaire answers (questionnaire data) obtained by the questionnaire answer acquisition unit 70. In other words, the display unit 50 may display, using a heat map, which parts of the eye region, consisting of the eye and its surrounding area, contribute to eye fatigue. Furthermore, based on these results, appropriate countermeasures may be suggested to the user for the areas that contribute most to eye fatigue. For example, if the heat map shows that the lower eyelids contribute significantly to fatigue, it can be considered that dark circles under the eyes may be influencing the fatigue. In this case, oral medication or skincare products that promote blood flow may be suggested to the user in order to suppress the decrease in blood flow, which is one of the causes of dark circles under the eyes. For example, the suggested oral medication or skincare product may be displayed on the display unit 50.

[0089] (Modification) The information processing apparatus and information processing method according to the present invention have been described above based on embodiments, but the present invention is not limited to the above embodiments.

[0090] For example, in embodiments 1 and 2 described above, the entire face image P1 is acquired by the face image acquisition unit 10, and then the eye image extraction unit 20 extracts the eye image P2 from the face image P1. However, the system is not limited to this. Specifically, the eye image P2 may be extracted without acquiring the face image P1. For example, the eye image P2 may be extracted from the beginning while displaying the face on the display screen (i.e., the face image P1 is not captured). In this case, the face may be displayed on the display screen, and the eye image P2 may be automatically extracted and captured from the display screen. Thus, the information processing devices 1, 2, and 2A do not necessarily have to be equipped with a face image acquisition unit 10.

[0091] Furthermore, information processing devices 1, 2, and 2A do not only lack a face image acquisition unit 10, but also do not need to have an eye image extraction unit 20. In this case, information processing devices 1, 2, and 2A only need to be able to acquire a separately extracted face image P1. For example, information processing devices 1 and 2 can acquire the face image P1 by transmitting the separately extracted face image P1 to information processing devices 1, 2, and 2A via wireless communication. In this case, the separately extracted face image P1 may be stored in an external device such as a server.

[0092] Furthermore, in embodiments 1 and 2 described above, the score calculated by the score unit 40 was presented to the user by being displayed on the display unit 50, but this is not limited to this. For example, the score calculated by the score unit 40 may be presented to the user by being announced by the voice announcement unit. In this case, the score calculated by the score unit 40 may be presented to the user by being displayed on the display unit 50 and announced by the voice announcement unit, or the score may be presented to the user by the voice announcement unit alone.

[0093] The processes described for the operation of the eye image extraction unit 20, the eye state determination unit 30, and the scoring unit 40 can be executed by a computer. For example, a computer can execute each of the above processes by running a program using hardware resources such as a processor (CPU), memory, and input / output circuits. Specifically, the processor can execute each process by acquiring the data to be processed from memory or input / output circuits, performing calculations on the data, and outputting the calculation results to memory or input / output circuits. The processor may consist of a single semiconductor chip or may consist of multiple semiconductor chips physically. When the processor consists of multiple semiconductor chips, each control of each embodiment may be implemented on a separate semiconductor chip. The eye image extraction unit 20, the eye state determination unit 30, and the scoring unit 40 may also consist of circuits. These circuits may constitute a single circuit as a whole, or they may be separate circuits. These circuits may also be general-purpose circuits or dedicated circuits.

[0094] Furthermore, the information processing method in the above embodiment may be implemented as a computer program executed by a computer, or as a computer-readable recording medium storing the program. For example, the present invention may be a program that causes a computer to execute the information processing method.

[0095] Furthermore, the present invention also includes forms obtained by applying various modifications to the above embodiments that a person skilled in the art could conceive, and forms realized by arbitrarily combining the components and functions of the embodiments without departing from the spirit of the present invention. In addition, the present invention also includes any combination of two or more claims from among the multiple claims described in the claims of the present application, provided that they are not technically contradictory. For example, if the cited claims described in the claims of the present application are made into a multi-claim or multi-multi-claim so as to refer to all of the higher-level claims without technically contradictory, then all combinations of claims included in that multi-claim or multi-multi-claim are also included in the present invention.

[0096] 1, 2, 2A Information Processing Device 10 Face Image Acquisition Unit 20 Eye Image Extraction Unit 30 Eye State Judgment Unit 31 First Learning Model 40 Score Unit 50 Display Unit 60 Age Acquisition Unit 70 Questionnaire Response Acquisition Unit 80 Age Estimation Unit 81 Second Learning Model 90 Focus Adjustment Power Acquisition Unit P1 Face Image P2 Eye Image

Claims

1. An information processing device comprising: an eye state determination unit that determines the state of a user's eyes when a user's eye image is input to a first learning model that has been pre-generated by machine learning based on training data obtained by assigning annotations indicating the state of the eye to each of a plurality of eye images acquired in advance; and a scoring unit that scores the state of the user's eyes based on the state of the user's eyes determined by the eye state determination unit, wherein the eye image is an image that includes only the eye and the peripheral portion of the eye.

2. The information processing apparatus according to claim 1, comprising: a face image acquisition unit that acquires a face image which is an image including the face of the user; and an eye image extraction unit that extracts the eye image from the face image acquired by the face image acquisition unit, wherein the eye image extracted by the eye image extraction unit is input to the first learning model.

3. An information processing device according to claim 1, comprising: an age acquisition unit that acquires the actual age of the user; and an age estimation unit that estimates the eye age of the user by inputting the user's eye images into a second learning model that has been pre-generated by machine learning based on training data obtained by assigning an annotation indicating age to each of a plurality of eye images acquired in advance, wherein the score unit scores the eye condition of the user based on the eye age gap, which is the difference between the actual age of the user acquired by the age acquisition unit and the eye age of the user estimated by the age estimation unit, and the eye condition of the user determined by the eye condition determination unit.

4. The information processing apparatus according to claim 3, comprising a questionnaire response acquisition unit that acquires the user's response to an eye examination, wherein the scoring unit scores the user's eye condition based on the difference between the user's actual age acquired by the age acquisition unit and the user's eye age estimated by the age estimation unit, the user's eye condition determined by the eye condition determination unit, and the user's response to the eye examination acquired by the questionnaire response acquisition unit.

5. The information processing device according to claim 1, comprising a medical questionnaire response acquisition unit that acquires the user's response to a medical questionnaire about the user's eyes, wherein the scoring unit scores the user's eye condition based on the user's eye condition determined by the eye condition determination unit and the user's response to the medical questionnaire acquired by the medical questionnaire response acquisition unit.

6. The information processing device according to claim 4 or 5, wherein the items of the medical questionnaire include at least one selected from the group consisting of the degree of blurred vision, the degree of dry eyes, the degree of digital stress, the degree of drowsiness, and the degree of glare.

7. The information processing apparatus according to claim 1, comprising a focus adjustment ability acquisition unit for acquiring information on the user's eye's focusing ability, wherein the scoring unit scores the user's eye's condition based on the user's eye's condition determined by the eye condition determination unit and the user's eye's focusing ability information acquired by the focus adjustment ability acquisition unit.

8. The information processing apparatus according to any one of claims 1 to 5, further comprising a display unit that displays a score indicating the state of the user's eyes, which has been scored by the scoring unit.

9. The information processing apparatus according to claim 8, wherein the display unit displays products or messages suitable for the user's eye condition.

10. The information processing apparatus according to any one of claims 1 to 5, wherein the eye condition is the degree of eye fatigue.

11. An information processing method comprising: an eye state determination step in which the user's eye image is input to a first learning model generated by machine learning based on training data obtained by assigning annotations indicating the state of the eye to each of a plurality of eye images acquired in advance, thereby determining the state of the user's eye; and a scoring step in which the user's eye state is scored based on the state of the user's eye determined in the eye state determination step.

12. A program that causes a computer to execute the information processing method described in claim 11.

Images