Image processing device, image processing method and program

Abstract

An image processing device 1X mainly includes a convergence measuring means 15X, a determining means 16X, and a pupil measuring means 17X. The convergence measuring means 15X measures the degree of convergence of the eyes of a subject after the eyes are opened on the basis of an image capturing the subject who opens the eyes after the eyes have been closed. The determining means 16X determines whether there is a change in the focal point of the subject on the basis of the degree of convergence. The pupil measuring means 17X measures the pupils of the subject on the basis of an image capturing the subject when no change in the focal point is determined.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical field of an image processing device, an image processing method, and a storage medium that perform processing related to measurement of pupillary light reflex using an image.

[0002] There are known systems that measure the pupils of a subject based on a photographed facial image of the subject. For example, Patent Literature 1 discloses a system that detects the area of ​​the subject's pupils based on video signals from two cameras and analyzes the pupil area, pupil diameter, and pupil position.

[0003] International Publication WO2002 / 003853

[0004] Generally, measuring pupillary light reflex requires a dedicated device or system. On the other hand, when measuring pupillary light reflex on a daily basis for self-care purposes, it is desirable for the subject to be able to easily perform the measurement themselves.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that can suitably generate information regarding a subject's pupillary reflex from an image of the subject.

[0006] One aspect of the image processing device is an image processing device having: a convergence measurement means for measuring the degree of convergence of a subject after opening their eyes based on an image of the subject who has closed their eyes and then opened them; a determination means for determining whether or not there has been a change in focus of the subject based on the degree of convergence; and a pupil measurement means for measuring the pupil of the subject based on the image of the subject when it is determined that there has been no change in focus.

[0007] One aspect of the image processing method is an image processing method in which a computer measures a degree of convergence of a subject after opening their eyes based on an image of the subject who has closed and then opened their eyes, determines whether or not there is a change in focus of the subject based on the degree of convergence, and measures the pupil of the subject based on the image of the subject when it is determined that there is no change in focus. Note that the term "computer" includes any electronic device (which may be a processor included in an electronic device) and may be composed of multiple electronic devices.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following processes: based on an image of a subject who has closed their eyes and then opened them, measure the degree of convergence of the subject after the subject has opened their eyes; based on the degree of convergence, determine whether or not there has been a change in focus of the subject; and if it is determined that there has been no change in focus, measure the pupil of the subject based on the image of the subject.

[0009] It is possible to suitably generate information regarding the subject's pupillary reflex from an image of the subject.

[0010] 1 shows a schematic configuration of a light reflex measurement system according to a first embodiment. FIG. 2 shows a state during measurement of the light reflex when the light reflex measurement system is a single terminal device. FIG. 3 shows an example of the hardware configuration of an image processing device common to all embodiments. FIG. 4 shows an example of a functional block of an image processing device related to light reflex measurement processing in the first embodiment. FIG. 5 shows an example of a camera image capturing a subject's face. FIG. 6 shows an example of a graph showing temporal changes in pupil size associated with light reflex. FIG. 7 shows an example of a flowchart related to light reflex measurement processing in the first embodiment. FIG. 8 shows a schematic configuration of a light reflex measurement system according to a modified example. FIG. 9 shows a schematic configuration of a light reflex measurement system according to a second embodiment. FIG. 10 is a block diagram of an image processing device according to a third embodiment. FIG. 11 shows an example of a flowchart executed by the image processing device in the third embodiment.

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] <First Embodiment> (1) System Configuration Fig. 1 shows a schematic configuration of a light reflex measurement system 100 according to the first embodiment. The light reflex measurement system 100 is a system that simply measures the light reflex of a subject 6 based on an image of the face of the subject 6 captured by a visible light camera, and mainly includes an image processing device 1, an input device 2, an output device 3, a storage device 4, and a measurement device 5 including a camera (imaging device) 51. The subject 6 simply measures his or her own light reflex using the light reflex measurement system 100, for the purpose of, for example, managing his or her health condition (including self-care).

[0013] The image processing device 1 measures the pupillary light reflex of the subject 6 based on facial images of the subject 6 generated by the camera 51 (including a video that is a sequence of a predetermined number of images obtained in time series; the same applies hereinafter), and outputs the measurement results. The image processing device 1 communicates data with the input device 2, output device 3, storage device 4, and measurement device 5 via a communication network or by direct wireless or wired communication.

[0014] The input device 2 is an interface that accepts user input (manual input). The user who inputs information using the input device 2 may be the subject 6 himself / herself, or a person managing or supervising the subject 6. The input device 2 may be, for example, various user input interfaces such as a touch panel, buttons, a keyboard, a mouse, or a voice input device. The input device 2 supplies an input signal generated based on the user input to the image processing device 1.

[0015] The output device 3 outputs predetermined information based on an output signal supplied from the image processing device 1. In this case, the output signal includes at least one of a display signal and an audio signal. The output device 3 displays information based on the display signal supplied from the image processing device 1, and outputs audio information based on the audio signal supplied from the image processing device 1. The output device 3 includes at least one of a display device such as a display or a projector, and an audio output device such as a speaker, for example.

[0016] The storage device 4 is a memory that stores various information necessary for measuring the pupillary light reflex, etc. The storage device 4 may be an external storage device such as a hard disk connected to or built into the image processing device 1, or may be a storage medium such as a flash memory. The storage device 4 may also be a server device that performs data communication with the image processing device 1. The storage device 4 may also be composed of multiple devices.

[0017] The measurement device 5 is one or more sensors including a camera 51, which is a visible light camera. For example, the measurement device 5 may include an illuminance sensor for detecting changes in the amount of light in the measurement environment of the subject's 6 pupillary reflex. The measurement device 5 supplies signals measured by each sensor to the image processing device 1. Hereinafter, the image generated by the camera 51 will also be referred to as a "camera image." The camera 51 is an example of an "imaging means."

[0018] 1 is an example, and various modifications may be made to the configuration. For example, the image processing device 1, input device 2, output device 3, storage device 4, and measurement device 5 may be implemented by a single terminal device such as a smartphone or tablet terminal.

[0019] 2 shows a state in which the light reflex measurement system 100 is a single terminal device (e.g., a smartphone) during measurement of the light reflex. As shown in Fig. 2, the subject 6 holds the light reflex measurement system 100, which is a terminal device, and adjusts the orientation of the terminal device so that the subject's face is included in the shooting range of the camera 51. The light reflex measurement system 100 may be fixed to a tripod or the like. In the state shown in Fig. 2, the subject 6 closes his or her eyes for a predetermined period of time and then opens them in accordance with instructions (guidance) output by the light reflex measurement system 100, causing the light reflex measurement system 100 to measure the light reflex.

[0020] (2) Hardware Configuration Fig. 3 shows the hardware configuration of the image processing device 1. The image processing device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 90.

[0021] The processor 11 executes programs stored in the memory 12 to function as a controller (arithmetic unit) that performs overall control of the image processing device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0022] The memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The memory 12 also stores programs for executing processes performed by the image processing device 1. Some of the information stored in the memory 12 may be stored in one or more external storage devices capable of communicating with the image processing device 1, or in a storage medium that is detachable from the image processing device 1. The memory 12 may also function as at least a part of the storage device 4.

[0023] The interface 13 is an interface for electrically connecting the image processing device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0024] The hardware configuration of the image processing device 1 is not limited to the configuration shown in Fig. 3. For example, the image processing device 1 may include at least one of an input device 2, an output device 3, a storage device 4, and a measurement device 5.

[0025] (3) Overview of the Light Reflex Measurement Process Next, the light reflex measurement process, which is a process related to measuring the light reflex, will be described. In summary, the image processing device 1 sequentially instructs the subject 6 to close and open their eyes, and measures the subject 6's pupil size over time based on camera images generated after the eyes are opened. At this time, the image processing device 1 measures the degree of convergence to determine whether or not there is a focus change during the pupil size measurement period, and uses the measurement results of the pupil size when no focus change occurs to calculate an index of the light reflex. This enables the image processing device 1 to accurately measure the light reflex based on the camera images.

[0026] 4 shows an example of functional blocks of the image processing device 1 related to light reflex measurement processing. Functionally, the processor 11 of the image processing device 1 includes an instruction unit 14, a convergence measurement unit 15, a focus change determination unit 16, a pupil size measurement unit 17, and a light reflex output unit 18. Note that in FIG. 4, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to that shown. The same applies to other functional block diagrams described below.

[0027] When the instructing unit 14 detects a user input via the input device 2 instructing the start of measurement of the pupillary light reflex, or when a pre-planned timing for measurement of the pupillary light reflex arrives, the instructing unit 14 determines that measurement of the pupillary light reflex should be started, and instructs the subject 6 to close and open their eyes, in that order. In this case, the instructing unit 14 instructs the subject 6 to open their eyes after a predetermined time has elapsed since instructing the subject 6 to close their eyes. The predetermined time is, for example, the length of time the eyes are closed necessary to observe the pupillary light reflex, and a default value is stored in the storage device 4, the memory 12, or the like. In this case, the instructing unit 14 may use any facial recognition technology to detect the subject 6's eyes being closed from a camera image. In this case, the instructing unit 14 instructs the subject 6 to open their eyes after a predetermined time has elapsed since detecting the subject 6's eyes being closed from a camera image.

[0028] Furthermore, the instructing unit 14 preferably instructs the subject 6 to direct their gaze (i.e., focus) at a location a predetermined distance away before issuing an instruction to close or open their eyes (also referred to as a "pre-eye-closing gaze instruction") in order to stabilize the subject 6's focus. The above-mentioned "predetermined distance" is, for example, a distance of 6 meters or more at which the focal distance is essentially infinity. Note that the location a predetermined distance away is not limited to a location 6 meters or more away, and may be any location where the distance to the subject 6 does not change. After issuing the pre-eye-closing gaze instruction, if the focus change determination unit 16 determines that there is no focus change, the instructing unit 14 issues instructions to close and open their eyes in sequence. The instructing unit 14 notifies the convergence measurement unit 15, focus change determination unit 16, and pupil size measurement unit 17, respectively, that it has issued the instructions to close and open their eyes, and notifies the convergence measurement unit 15 and focus change determination unit 16, respectively, that it has issued the pre-eye-closing gaze instruction.

[0029] The convergence measurement unit 15 measures the degree of eye convergence (i.e., the state of cross-eyedness) (also referred to as the "convergence degree") of the subject 6's eyes based on camera images acquired from the camera 51 via the interface 13. Here, the convergence measurement unit 15 measures the convergence degree based on camera images generated during a period in which the subject 6 is estimated to have a pupillary reflex (also referred to as the "pupil's light reflex estimated period"). The target period is, for example, set to a period from when the subject 6 opens his eyes in response to an eye-opening instruction from the instruction unit 14 until a predetermined time has elapsed. The predetermined time is the length of time in which a pupillary reflex is estimated to occur after the eyes are opened, and a default value is stored in the storage device 4, the memory 12, or the like. In this case, the convergence measurement unit 15 detects the opening of the subject 6's eyes using any image recognition technology based on the latest camera images acquired from the camera 51 after the eye-opening instruction from the instruction unit 14, and recognizes the period from when the subject 6 detects the opening of his eyes until a predetermined time has elapsed as the pupillary reflex estimated period. The estimated light reflex period is an example of a "predetermined period."

[0030] Furthermore, when a gaze instruction before closing the eyes is given, the convergence measurement unit 15 measures the degree of convergence based on the camera image obtained after the gaze instruction before closing the eyes. In this case, for example, the convergence measurement unit 15 measures the degree of convergence during the period from the gaze instruction before closing the eyes to the instruction to close the eyes. The convergence measurement unit 15 supplies the measured degree of convergence to the focus change determination unit 16.

[0031] The focus change determination unit 16 determines whether or not there is a change in focus of the subject 6 during the pupillary reflex estimation period, and based on the determination result, determines whether or not measurement and output of the pupillary reflex is required by the pupillary reflex output unit 18. That is, the focus change determination unit 16 determines that the focus of the subject 6 is stable (i.e., there is constancy) during the pupillary reflex estimation period as a necessary condition for measuring and outputting the pupillary reflex. Therefore, the focus change determination unit 16 instructs the pupillary reflex output unit 18 to measure and output the pupillary reflex (i.e., calculate the pupillary reflex index, which will be described later) only when the focus of the subject 6 is stable during the pupillary reflex estimation period.

[0032] Generally, the presence or absence of a focus change can be determined by the presence or absence of a change in the convergence level, and if there is no change in the convergence level, it can be assumed that there is no focus change. Therefore, the focus change determination unit 16 determines whether or not there is a focus change of the subject 6 during the pupillary reflex estimation period based on the time-series convergence level measured by the convergence measurement unit 15 from the camera images generated during the pupillary reflex estimation period. For example, the focus change determination unit 16 determines whether or not there is a substantial change in the convergence level based on a comparison of a predetermined threshold value with the variance, the difference between the maximum and minimum values, or other statistics representing variance of a predetermined number of convergence levels obtained during the pupillary reflex estimation period. The above-mentioned threshold value is pre-stored, for example, in the storage device 4 or memory 12.

[0033] If the focus change determination unit 16 determines that the convergence has substantially changed, it determines that a focus change occurred during the pupillary reflex estimation period. In this case, the focus change determination unit 16 requests the instruction unit 14 to re-output an instruction. Here, in the camera image obtained during the pupillary reflex estimation period in which the focus change occurred, a pupil change due to a focus change known as near reflex occurs, making it highly likely that the pupillary reflex cannot be accurately measured. Therefore, if it determines that a focus change occurred during the pupillary reflex estimation period, the focus change determination unit 16 requests the instruction unit 14 to re-measure the pupillary reflex. Note that if a pre-eye-closing gaze instruction has been given, the instruction unit 14 outputs a pre-eye-closing gaze instruction and instructions to close and open the eyes, in that order. If a pre-eye-closing gaze instruction has not been given, the instruction unit 14 outputs instructions to close and open the eyes, in that order.

[0034] On the other hand, if the focus change determination unit 16 determines that the convergence has not substantially changed, it determines that there has been no focus change during the pupillary reflex estimation period. In this case, the focus change determination unit 16 instructs the pupillary reflex output unit 18 to measure and output the pupillary reflex. Here, if there has been no focus change during the pupillary reflex estimation period, no change in the pupil due to the focus change has occurred, and it is possible to accurately measure the pupillary reflex of the subject 6 from the camera image obtained during the pupillary reflex estimation period. Therefore, if it is determined that there has been no focus change during the pupillary reflex estimation period, the pupillary reflex output unit 18 measures and outputs the pupillary reflex, thereby making it possible to output accurate measurement results of the pupillary reflex.

[0035] Preferably, the focus change determination unit 16 may determine whether or not there is a change in the focus of the subject 6 before closing and after opening the eyes of the subject 6, in addition to determining whether or not there is a change in the focus of the subject 6 during the pupillary light reflex estimation period. That is, the focus change determination unit 16 may determine, as a condition (necessary condition) for measuring and outputting the pupillary light reflex, that the focus of the subject 6 is stable before closing the eyes based on an eye-closing instruction and after opening the eyes based on an eye-opening instruction. In this case, the focus change determination unit 16 determines whether or not there is a change in the focus of the subject 6 based on, for example, the difference between the degree of convergence measured by the convergence measurement unit 15 immediately before closing the eyes of the subject 6 and the degree of convergence measured by the convergence measurement unit 15 after opening the eyes of the subject 6. If the difference is greater than a threshold, the focus change determination unit 16 determines that there is a change in the focus of the subject 6 before and after closing and opening the eyes, and requests the instruction unit 14 to re-output instructions regardless of whether or not there is a change in focus during the pupillary light reflex estimation period. On the other hand, if there is no change in focus of the subject 6 before closing and after opening their eyes, and if there is no change in focus during the pupillary light reflex estimation period, the focus change determination unit 16 instructs the pupillary light reflex output unit 18 to measure and output the pupillary light reflex. Note that even if there is no change in focus of the subject 6 before closing and after opening their eyes, if there is a change in focus during the pupillary light reflex estimation period, the focus change determination unit 16 requests the instruction unit 14 to re-output an instruction. In this way, by making stable focus before and after instructions to close and open your eyes a necessary condition for measuring the pupillary light reflex, more accurate pupillary light reflex measurement results can be obtained.

[0036] Furthermore, when a pre-eye-closing gaze instruction is given, the focus change determination unit 16 may determine whether or not the subject 6's focus has changed after the pre-eye-closing gaze instruction as a necessary condition for issuing the eye-closing instruction, and may determine whether or not the subject 6's focus has changed after the pre-eye-closing gaze instruction. In this case, for example, the focus change determination unit 16 determines whether or not the convergence level has changed in the time series supplied from the convergence measurement unit 15 at a predetermined time immediately preceding the predetermined time. In this case, for example, the focus change determination unit 16 determines whether or not the convergence level has substantially changed within the predetermined time based on a comparison of a predetermined threshold with a variance, a difference between a maximum value and a minimum value, or other statistical quantity representing variation for a predetermined number of convergence levels obtained within the predetermined time. Then, if the focus change determination unit 16 determines that the convergence level has not substantially changed, it considers that the subject 6's focus has not changed (i.e., the focus is stable) and instructs the instruction unit 14 to output an eye-closing instruction. On the other hand, if the focus change determination unit 16 determines that the degree of convergence has substantially changed, it assumes that there has been a change in the focus of the subject 6 (i.e., the focus is not stable), and subsequently determines whether there has been a change in the degree of convergence in the time series supplied at the specified time immediately before.

[0037] The pupil size measurement unit 17 measures the pupil size of the subject 6 during the pupillary light reflex estimation period. In this case, the pupil size measurement unit 17 measures the pupil size of the subject 6 from each of the camera images generated during the pupillary light reflex estimation period, thereby generating time-series measurement results of the pupil size of the subject 6 during the pupillary light reflex estimation period. Then, the pupil size measurement unit 17 supplies the pupil size measurement results to the pupillary light reflex output unit 18.

[0038] The pupil size measurement unit 17 may perform processing to measure the pupil size of the subject 6 from each of the camera images generated during the pupillary reflex estimation period, regardless of the determination result of the focus change determination unit 16 regarding the presence or absence of a focus change of the subject 6. In this case, for example, the pupil size measurement unit 17 may output the measurement result of the pupil size generated during the pupillary reflex estimation period to the pupillary reflex output unit 18 only when it receives a determination result indicating that there has been no focus change from the focus change determination unit 16. In another example, the pupil size measurement unit 17 may receive the determination result of the focus change determination unit 16 regarding the presence or absence of a focus change, and output the determination result and the measurement result of the pupil size generated during the pupillary reflex estimation period to the pupillary reflex output unit 18.

[0039] Instead of the above example, the pupil size measurement unit 17 may perform processing to measure the pupil size of the subject 6 from each of the camera images generated during the pupillary light reflex estimation period and stored in the storage device 4, memory 12, or the like, after the focus change determination unit 16 determines that there is no focus change in the subject 6. That is, in this case, when the focus change determination unit 16 determines that there is a focus change in the subject 6, the pupil size measurement unit 17 does not measure the pupil size based on the camera images generated during the relevant pupillary light reflex estimation period.

[0040] The pupillary reflex output unit 18 calculates an index related to pupillary reflex (also referred to as "pupillary reflex index") based on the measurement result of the pupil size by the pupil size measurement unit 17 during the pupillary reflex estimation period, and displays or outputs information related to the calculation result of the pupillary reflex index by sound via the output device 3. Here, the pupillary reflex output unit 18 calculates the pupillary reflex index when the focus change determination unit 16 instructs it to measure and output the pupillary reflex, that is, when it is determined that there has been no change in the focus of the subject 6 during the pupillary reflex estimation period (and before and after eye closing and eye opening). The type of pupillary reflex index to be calculated varies depending on the application to which the pupillary reflex measurement system 100 is applied, and specific examples will be described later in sections "(4) Convergence, Pupil Size, and Pupillary Reflex Index" and "(7) Application."

[0041] Here, the information regarding the calculation result of the pupillary light reflex index that the pupillary light reflex output unit 18 outputs to the output device 3 may be information indicating the calculated pupillary light reflex index itself, or may be information regarding the state of the subject 6 estimated based on the pupillary light reflex index. An example of the latter will be specifically described in section "(7) Application." The pupillary light reflex output unit 18 generates an output signal (i.e., at least one of a display signal and an audio signal) for outputting information regarding the calculation result of the pupillary light reflex index, and supplies the output signal to the output device 3 via the interface 13. As a result, the pupillary light reflex output unit 18 causes the output device 3 to output information regarding the calculation result of the pupillary light reflex index.

[0042] Furthermore, the image processing device 1 may calculate a predetermined number of times of pupillary reflex indices by executing the processes of the instruction unit 14, the convergence measurement unit 15, the focus change determination unit 16, the pupil size measurement unit 17, and the pupillary reflex output unit 18 a predetermined number of times. In this case, the pupillary reflex output unit 18 calculates a time average or other representative value of the pupillary reflex indices for the predetermined number of times and outputs the calculation result to the output device 3. This makes it possible to output a highly accurate pupillary reflex index that has been subjected to statistical processing.

[0043] The components of the instruction unit 14, convergence measurement unit 15, focus change determination unit 16, pupil size measurement unit 17, and pupillary light reflex output unit 18 described in FIG. 4 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not necessarily be realized by software programs, but may also be realized by any combination of hardware, firmware, and software. At least some of these components may also be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0044] (4) Convergence, Pupil Size, and Light Reflex Index First, specific examples of the convergence and pupil size will be described. Fig. 5 shows an example of a camera image of the face of the subject 6.

[0045] The image processing device 1 uses any image recognition technology to recognize the areas of the eyes (both eyes in this case) 9a, iris 9b, and pupil 9c of the subject 6 from a camera image capturing the face of the subject 6. Then, the image processing device 1 calculates the pupil size and degree of convergence based on each of the recognized areas.

[0046] First, calculation of pupil size will be described. The image processing device 1 calculates the pupil size (i.e., pupil diameter) corresponding to the length of the arrow 90. Here, the image processing device 1 may calculate the pupil size for each eye and further generate an average of the calculated pupil sizes.

[0047] Next, calculation of the degree of convergence will be described. The image processing device 1 calculates the interpupillary distance, which corresponds to the length of the arrow 91, as an example of the degree of convergence. In this way, the image processing device 1 can accurately recognize convergence by calculating the degree of convergence based on both eyes. In this case, the camera image is an image capturing at least both eyes of the subject 6, and the image processing device 1 calculates the degree of convergence based on the positions of the pupils 9c of both eyes in the camera image.

[0048] The image processing device 1 may calculate pupil information based on a camera image capturing only one eye of the subject. In this case, for example, the image processing device 1 may calculate the relative position of the center of the pupil 9c with respect to both ends of the eye 9a (i.e., the distance corresponding to the lengths of the arrows 93 and 94) as the degree of convergence. In this way, the image processing device 1 may calculate a value indicating the relative position of the pupil 9c in the eye 9a as the degree of convergence.

[0049] Here, the image processing device 1 may perform a normalization process to convert the pupil size, interpupillary distance, etc., from a size based on the number of pixels in the camera image to a size of a predetermined scale (e.g., actual size), thereby generating normalized pupil size and interpupillary distance. In this case, for example, the image processing device 1 may normalize the pupil size and interpupillary distance based on the size of the iris 9b, taking advantage of the fact that the size of the iris 9b varies little among individuals and does not change over time. In this case, for example, information indicating the relationship between the size of the iris 9b in the image and the size of the iris 9b after normalization is pre-stored in the storage device 4, memory 12, etc., and the image processing device 1 performs the normalization described above by referring to this information. In another example, if the light reflex measurement system 100 includes multiple cameras with different viewpoints as the cameras 51, the image processing device 1 may recognize the actual size of the pupil size and interpupillary distance by three-dimensionally reconstructing the face of the subject 6 from camera images from the multiple cameras based on any three-dimensional reconstruction technology, such as SfM (Structure from Motion).

[0050] Next, a specific example of the light reflex index will be described. Fig. 6 is an example of a graph showing the change over time in pupil size due to the light reflex.

[0051] In this case, after a time lag (i.e., latency) has elapsed from time "t1" when the opening of the eyes of the subject 6 based on the eye-opening instruction is detected until the pupil starts to contract, the pupil size (pupil radius here) of the subject 6 gradually decreases from a maximum value "Dmax" to a minimum value "Dmin" at time "t2." Thereafter, the pupil size expands again and reaches a steady state around time "t3," a predetermined time after time t2.

[0052] In this case, for example, the image processing device 1 calculates, as light reflex indices, the minimum value Dmin, the maximum value Dmax, the maximum amount of pupil contraction corresponding to the length of arrow 95 (i.e., Dmax - Dmin), the maximum pupil contraction rate corresponding to the ratio of the maximum value Dmax to the minimum value Dmin, the pupil contraction speed corresponding to the slope of line 96, and the re-dilation speed corresponding to the slope of line 97. Note that the slope of line 96 corresponds to the speed of pupil size decrease during the period in which the pupil size decreases, and the slope of line 97 corresponds to the speed of pupil size increase during the period in which the pupil size increases (re-dilation).

[0053] The light reflex index is not limited to the index described above, but may be any index based on the time-series pupil size measured during the light reflex estimation period.

[0054] (5) Processing Flow Fig. 7 is an example of a flowchart relating to the light reflex measurement process executed by the image processing device 1. Here, as an example, the light reflex measurement process when a gaze instruction before closing the eyes is performed will be described.

[0055] First, the image processing device 1 outputs a pre-eye closure gaze instruction that indicates the gaze of the subject 6 via the output device 3 (step S11). Then, the image processing device 1 measures the degree of convergence of the subject 6 based on each of the time-series camera images generated by the camera 51 after step S11 (step S12).

[0056] Then, the image processing device 1 determines whether or not the focus of the subject 6 is constancy (step S13). In this case, the image processing device 1 determines whether or not the focus of the subject 6 is constancy based on whether or not there is a change in the convergence over time obtained in step S12. Then, when the image processing device 1 determines that the focus of the subject 6 is not constancy (step S13; No), the process returns to step S12 and measures the convergence.

[0057] Then, when it is determined that the focus of the subject 6 is stable (step S13; Yes), the image processing device 1 outputs instructions to close and open the eyes of the subject 6 via the output device 3 (step S14). In this case, for example, after instructing the subject 6 to close their eyes, the image processing device 1 outputs an instruction to open their eyes a predetermined time after detecting that the subject 6 has closed their eyes.

[0058] Next, the image processing device 1 determines whether the subject 6 has opened their eyes (step S15). In this case, the image processing device 1 determines whether the subject 6 has opened their eyes based on, for example, a camera image. If the image processing device 1 determines that the subject 6 has not opened their eyes (step S15; No), it continues the determination in step S15.

[0059] If the image processing device 1 determines that the subject 6 has opened their eyes (step S15; Yes), it measures the pupil size and convergence for a predetermined time based on the latest camera images (step S16). In this case, the image processing device 1 regards the period from when it is determined that the subject 6 has opened their eyes until the predetermined time has elapsed as an estimated light reflex period, and continuously measures the pupil size and convergence based on the camera images generated during the estimated light reflex period. This allows a predetermined number of measurement results (time-series data) of pupil size and convergence to be obtained.

[0060] Next, the image processing device 1 determines whether or not there is focus constancy (step S17). In this case, in a first example, the image processing device 1 determines whether or not there is focus constancy during the execution period of step S16 (i.e., the pupillary light reflex estimation period) based on whether or not there is a change in the time-series convergence obtained in step S16. In a second example, the image processing device 1 makes the determination in step S17 by taking into account the focus before closing the eyes in addition to whether or not there is focus constancy during the pupillary light reflex estimation period. In this case, the image processing device 1 determines whether or not there is focus constancy based on the convergence obtained in step S12 immediately before it was determined in step S13 that there is focus constancy, and the time-series convergence obtained in step S16.

[0061] If the image processing device 1 determines that there is no focus constancy (step S17; No), the process returns to step S11. Note that, in step S17, when the image processing device 1 determines the focus constancy without taking into account the measurement result of the degree of convergence in step S12 (i.e., in the case of the first example), the process may return to step S14 instead of returning to step S11.

[0062] If it is determined that the focus is stationary (step S17; Yes), one or more types of pupillary reflex indices are calculated based on the time-series pupil size measurement results measured in step S16 (step S18). Note that the image processing device 1 may calculate a time average (or other representative value) of the pupillary reflex indices based on a plurality of pupillary reflex indices samples obtained by repeating steps S11 to S18 a plurality of times.

[0063] Then, the image processing device 1 outputs information related to the calculation result of the pupillary light reflex index via the output device 3 (step S19). In this case, the image processing device 1 may determine the state of the subject 6, such as the presence or absence of aging phenomena, fatigue, or nervous system disease, from the calculated pupillary light reflex index according to the application to be applied, and output the determination result via the output device 3. Examples of applications will be described later.

[0064] (6) Modifications Modifications of the above-described embodiment will now be described. The following modifications may be applied to the above-described embodiment in any combination.

[0065] (Variation 1) The image processing device 1 may output, via the output device 3, guide information (i.e., guidance information) regarding the shooting distance so that the distance (shooting distance) between the subject 6 and the camera 51 is a predetermined distance, based on the size of the iris of the subject 6 recognized from the camera image.

[0066] In this case, for example, before issuing the gaze instruction before closing the eyes in step S11, the image processing device 1 recognizes the iris size of the subject 6 from the camera image. The image processing device 1 then determines whether the recognized iris size falls within an appropriate size range. The appropriate size range is an iris size range corresponding to a shooting distance range that is within a distance range preferable for the pupillary light reflex measurement process, and is a range pre-stored in, for example, the storage device 4 or the memory 12. If the recognized iris size falls outside the appropriate size range, the image processing device 1 causes the output device 3 to output guidance information that advises the subject 6 to adjust the shooting distance. Specifically, if the recognized iris size is smaller than the appropriate size range, the image processing device 1 outputs guidance information suggesting that the shooting distance be shortened. If the recognized iris size is larger than the appropriate size range, the image processing device 1 outputs guidance information suggesting that the shooting distance be increased. If the recognized iris size is within the appropriate size range, the image processing device 1 issues the gaze instruction before closing the eyes in step S11.

[0067] According to this modification, the image processing device 1 can suitably adjust the shooting distance to a distance suitable for measuring the pupillary light reflex.

[0068] (Modification 2) The image processing device 1 may further execute a process of controlling the light source so that the illuminance of the subject 6 during the light reflex estimation period is constant.

[0069] Fig. 8 shows a schematic configuration of the light reflex measurement system 100 in Modification 2. The light reflex measurement system 100 in Modification 2 has a light source 7. The light source 7 is a light source that illuminates the subject 6, and when the light reflex measurement system 100 is a smartphone or the like as shown in Fig. 2, the light source 7 is a light or display provided on the side of the camera used by the smartphone or the like.

[0070] In this modification, the image processing device 1 controls the light intensity of the light source 7. For example, the image processing device 1 causes the light source 7 to emit light at the same light intensity during a period from a predetermined timing after an instruction to close the eyes and before an instruction to open the eyes until a predetermined time has elapsed. In this case, the period is set to a length that includes at least the entire estimated light reflex period. Note that, if the light source 7 is a light, the image processing device 1 may control the on / off of the light source 7 so that it is on during the period and off outside the period.

[0071] According to this modified example, the image processing device 1 controls the illuminance of the subject 6 to be constant during the pupillary light reflex estimation period, and can effectively suppress a decrease in the measurement accuracy of the pupillary light reflex due to uneven illuminance.

[0072] The image processing device 1 may define a condition for measuring and outputting the pupillary light reflex (i.e., calculating the pupillary light reflex index) as a constant illuminance of the subject 6 during the pupillary light reflex estimation period, and may perform measurement and output of the pupillary light reflex only if the illuminance of the subject 6 is stable during the pupillary light reflex estimation period. In this case, the image processing device 1 determines whether the illuminance of the subject 6 during the pupillary light reflex estimation period is constant, for example, based on the time series of illuminance measured by the illuminance sensor of the measurement device 5 during the pupillary light reflex estimation period. In this case, for example, the image processing device 1 determines whether the illuminance has substantially changed based on a comparison of a predetermined threshold with the variance, the difference between the maximum and minimum values, or other statistical quantities representing variation of a predetermined number of illuminance measurements obtained during the pupillary light reflex estimation period. Even in this case, the image processing device 1 can effectively suppress a decrease in the measurement accuracy of the pupillary light reflex due to uneven illuminance. Instead of an illuminance sensor, for example, the brightness value of a camera image may be used.

[0073] (7) Applications Next, examples of applications of the light reflex measurement system 100 will be described. In the following applications, the light reflex measurement system 100 estimates the state (particularly the state related to health) of the subject 6 based on the calculated light reflex index, and outputs the estimated state of the subject 6. Here, as specific examples of applications of the light reflex measurement system 100, an application relating to quantitative understanding of functional decline due to aging and an application relating to quantitative understanding of eye strain will be described.

[0074] (7-1) Quantitative Assessment of Age-Related Functional Decline In an application related to quantitative assessment of age-related functional decline, for example, the image processing device 1 calculates, as light reflex indices, the pupil size measured immediately after the subject's eyes are opened (i.e., the maximum pupil size Dmax), the pupil size when the subject's eyes are opened and a light reflex occurs (i.e., the minimum pupil size Dmin or the pupil size after re-dilation), and the pupil contraction rate. The image processing device 1 then compares the calculated values ​​of various light reflex indices with reference values ​​of the light reflex indices to quantitatively estimate the degree of age-related functional decline in the subject 6. The image processing device 1 then outputs the estimation results to the output device 3. The reference values ​​may be general reference values ​​of various light reflex indices for the subject 6's age, or may be past calculated values ​​of the light reflex indices for the subject 6. The reference values ​​may also be thresholds for determining the presence or level of age-related functional decline.

[0075] Here, the image processing device 1 may use a model that estimates the degree of functional decline due to aging of the subject 6, and output information output by the model to the output device 3. The above-mentioned model is a machine learning model such as an equation, a lookup table, or a neural network, and outputs an estimation result regarding the degree of functional decline due to aging (e.g., an estimated age of the subject 6) when, for example, calculated values ​​of various pupillary reflex indices (or differences between the calculated values ​​and reference values, etc.) are input. Parameters of the above-mentioned model, etc., are stored in advance in the storage device 4, memory 12, etc. Furthermore, the image processing device 1 may display a graph or the like that allows comparison between the calculated values ​​of various pupillary reflex indices and the corresponding reference values.

[0076] With such an application, the subject 6 can easily measure his / her own light reflex using his / her own smartphone or the like, and quantitatively grasp the functional decline due to aging. The image processing device 1 can also suggest preventive activities for functional decline due to aging and visualize the preventive effects of functional decline due to aging, thereby increasing the user (subject 6)'s awareness of continuing to take preventive activities for functional decline due to aging.

[0077] (7-2) Quantitative Assessment of Eye Strain In an application related to quantitative assessment of eye strain, the image processing device 1 calculates, for example, the maximum pupil contraction rate, pupil contraction velocity, and re-dilation velocity as pupillary light reflex indices. The image processing device 1 then compares the calculated values ​​of various pupillary light reflex indices with reference values ​​of pupillary light reflex indices to quantitatively estimate the degree of eye strain of the subject 6. The image processing device 1 then outputs the estimation result to the output device 3. The reference values ​​may be general reference values ​​of various pupillary light reflex indices for the age of the subject 6, or may be previously calculated values ​​of the pupillary light reflex indices of the subject 6. The reference values ​​may also be thresholds for determining the presence or absence or level of eye strain of the subject 6.

[0078] Here, the image processing device 1 may use a model for estimating the degree of eye strain of the subject 6 and cause the output device 3 to output information output by the model. The above-mentioned model may be a machine learning model such as an equation, a lookup table, or a neural network, and may output an estimation result regarding the degree of eye strain of the subject 6 when, for example, calculated values ​​of various pupillary reflex indices (or differences between the calculated values ​​and reference values, etc.) are input. Parameters of the above-mentioned model, etc., are stored in advance in the storage device 4, memory 12, etc. Furthermore, the image processing device 1 may display, as information indicating the degree of eye strain of the subject 6, a graph or the like that compares the calculated values ​​of various pupillary reflex indices with the corresponding reference values.

[0079] According to such an application, the subject 6 can easily measure his / her own light reflex using his / her own smartphone or the like, and quantitatively grasp his / her eye strain.

[0080] <Second embodiment> Fig. 9 shows a schematic configuration of a light reflex measurement system 100A in the second embodiment. The light reflex measurement system 100A according to the second embodiment has an image processing device 1A that functions as a server, and a terminal device 8 that is used by a subject and functions as a client. The image processing device 1A and the terminal device 8 communicate data via a network 99. Hereinafter, the same components as those in the first embodiment will be appropriately designated by the same reference numerals, and their description will be omitted.

[0081] The terminal device 8 is a terminal used by a user who will be the subject, and has input, display, communication, and imaging functions, and functions as the input device 2, output device 3, and measurement device 5 including camera 51 shown in Fig. 1. The terminal device 8 may be, for example, a personal computer, a tablet terminal such as a smartphone, or a PDA (Personal Digital Assistant). The terminal device 8 transmits the facial image of the subject output by the camera 51 to the image processing device 1A via the network 99.

[0082] The image processing device 1A has the same hardware configuration as the image processing device 1 shown in Fig. 2, and the processor 11 of the image processing device 1A has the functional blocks shown in Fig. 4 described in the first embodiment. The image processing device 1A receives camera images from the terminal device 8 via the network 99 and executes a process for measuring the subject's light reflex. In addition, the image processing device 1A transmits an output signal for outputting the processing results to the terminal device 8 via the network 99 based on a display request from the terminal device 8.

[0083] In this way, the image processing device 1A in the second embodiment performs processing related to measuring the pupillary light reflex of the subject who is the user of the terminal device 8, and can present the measurement results of the pupillary light reflex to the subject via the terminal device 8 in an appropriate manner.

[0084] 10 is a block diagram of an image processing device 1X according to a third embodiment. The image processing device 1X mainly includes a convergence measurement unit 15X, a determination unit 16X, and a pupil measurement unit 17X. Note that the image processing device 1X may be configured by a plurality of devices.

[0085] The convergence measurement unit 15X measures the degree of convergence of the subject after opening their eyes based on an image of the subject opening their eyes after closing them. The convergence measurement unit 15X can be, for example, the convergence measurement unit 15 in the first or second embodiment.

[0086] The determining unit 16X determines whether or not the subject has experienced a change in focus based on the degree of convergence. The determining unit 16X can be, for example, the focus change determining unit 16 in the first or second embodiment.

[0087] The pupil measurement unit 17X measures the pupil of the subject based on an image of the subject when it is determined that there is no focus change. The pupil measurement unit 17X can be, for example, the pupil size measurement unit 17 in the first or second embodiment.

[0088] 11 is an example of a flowchart executed by the image processing device 1X in the third embodiment. First, the convergence measurement unit 15X measures the degree of convergence after the subject opens their eyes based on an image of the subject who has closed and then opened their eyes (step S21). The determination unit 16X determines whether or not the subject has changed their focus based on the degree of convergence (step S22). If it is determined that there has been no change in focus, the pupil measurement unit 17X measures the pupil of the subject based on an image of the subject (step S23).

[0089] According to the third embodiment, the image processing device 1X can suitably measure the pupil, which changes in accordance with the subject's light reflex.

[0090] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0091] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0092] [Supplementary Note 1] An image processing device comprising: a convergence measurement means for measuring a degree of convergence of a subject after the subject has opened their eyes after closing them, based on images of the subject, the subject opening the eyes; a determination means for determining whether or not there has been a change in focus of the subject based on the degree of convergence; and a pupil measurement means for measuring the pupil of the subject based on the images of the subject when it has been determined that there has been no change in focus. [Supplementary Note 2] The image processing device according to Supplementary Note 1, further comprising: a light reflex calculation means for calculating an index related to the subject's light reflex based on time-series measurement results of the pupil. [Supplementary Note 3] The image processing device according to Supplementary Note 1, wherein the determination means determines whether or not there has been a change in focus during a predetermined period based on images of the subject taken during the predetermined period after the eyes have been opened, and the pupil measurement means, when it has been determined that there has been no change in focus during the predetermined period, measures the pupil based on the images of the subject taken during the predetermined period. [Supplementary Note 4] The image processing device according to Supplementary Note 3, wherein the determination means further determines whether or not there is a change in focus before closing the eyes and after opening the eyes, and the pupil measurement means measures the pupil based on the images of the subject captured during the predetermined period when it is determined that there is no change in focus during the predetermined period and that there is no change in focus before closing the eyes and after opening the eyes. [Supplementary Note 5] The image processing device according to Supplementary Note 3, further comprising light source control means for controlling the amount of light illuminating the subject so that the amount of light from the light source is constant during the predetermined period. [Supplementary Note 6] The image processing device according to Supplementary Note 1, wherein the images are images of at least both of the subject's eyes, and the convergence measurement means calculates the degree of convergence based on the positions of the pupils of both eyes in the images. [Supplementary Note 7] The image processing device according to Supplementary Note 1, further comprising guidance information output means for outputting guidance information regarding the distance between the subject and the image capture means for generating the images by an output device, based on a measurement result of measuring the iris of the subject based on the images. [Supplementary Note 8] The image processing device according to Supplementary Note 1, further comprising an instruction unit that instructs the subject to close his / her eyes and to open his / her eyes after a predetermined time has elapsed since closing his / her eyes.[Supplementary Note 9] The image processing device according to Supplementary Note 8, wherein the instructing means gives instructions regarding the subject's line of sight, and instructs the subject to close their eyes if it is determined that there has been no change in the subject's focus after the instruction. [Supplementary Note 10] The image processing device according to Supplementary Note 1, wherein the pupil measurement means measures the size of the pupil. [Supplementary Note 11] The image processing device according to Supplementary Note 2, wherein the pupillary reflex calculation means outputs information regarding the subject's condition estimated based on the index by an output device. [Supplementary Note 12] The image processing device according to Supplementary Note 11, wherein the pupillary reflex calculation means outputs the degree of functional decline due to aging of the subject estimated based on the index as information regarding the condition. [Supplementary Note 13] The image processing device according to Supplementary Note 11, wherein the pupillary reflex output means outputs the degree of eye strain of the subject estimated based on the index as information regarding the condition. [Supplementary Note 14] An image processing method in which a computer measures a degree of convergence of a subject's eyes after they have been opened, based on an image of the subject who has closed their eyes and then opened them, determines whether or not there has been a change in focus of the subject based on the degree of convergence, and measures the pupil of the subject based on the image of the subject when it has been determined that there has been no change in focus. [Supplementary Note 15] A storage medium having stored therein a program that causes a computer to execute processes of measuring a degree of convergence of the subject's eyes after they have been opened, based on an image of the subject who has closed their eyes and then opened them, determines whether or not there has been a change in focus of the subject based on the degree of convergence, and measures the pupil of the subject based on the image of the subject when it has been determined that there has been no change in focus.

[0093] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0094] REFERENCE SIGNS LIST 1, 1A, 1X Image processing device 2 Input device 3 Output device 4 Storage device 5 Measurement device 7 Light source 8 Terminal device 11 Processor 12 Memory 13 Interface 51 Camera 90 Data bus 99 Network 100, 100A Light reflex measurement system

Claims

1. a convergence measuring means for measuring a degree of convergence of a subject after opening their eyes based on an image of the subject who has closed their eyes and then opened them; a determination means for determining whether or not the subject has experienced a change in focus based on the degree of convergence; a pupil measurement means for measuring the pupil of the subject based on the image of the subject when it is determined that there is no focus change; An image processing device having:

2. 2. The image processing apparatus according to claim 1, further comprising: a light reflex calculation unit that calculates an index relating to the subject's light reflex based on the time-series measurement results of the pupil.

3. the determining means determines whether or not the focus has changed during the predetermined period based on the image of the subject captured during the predetermined period after the eye has been opened; 2. The image processing device according to claim 1, wherein, when it is determined that there is no change in focus during the predetermined period, the pupil measurement means measures the pupil based on the image of the subject captured during the predetermined period.

4. The determination means further determines whether or not there is a change in focus between before the eye is closed and after the eye is opened, 4. The image processing device according to claim 3, wherein the pupil measurement means measures the pupil based on the image of the subject taken during the predetermined period when it is determined that there is no change in the focus during the predetermined period and that there is no change in the focus between before closing the eye and after opening the eye.

5. 4. The image processing apparatus according to claim 3, further comprising a light source control means for controlling the amount of light emitted from the light source illuminating the subject so that the amount of light is constant during the predetermined period.

6. the image is an image of at least both eyes of the subject, The image processing device according to claim 1 , wherein the convergence measuring means calculates the degree of convergence based on positions of the pupils of the eyes in the image.

7. 2. The image processing device according to claim 1, further comprising a guidance information output means for outputting guidance information regarding a distance between the subject and an imaging means that generates the image, based on a measurement result of measuring the iris of the subject based on the image, by an output device.

8. The image processing apparatus according to claim 1 , further comprising an instruction unit for instructing the subject to close his / her eyes and to open his / her eyes after a predetermined time has elapsed since closing his / her eyes.

9. The computer measuring a degree of eye convergence of the subject after the subject has opened their eyes based on an image of the subject who has closed their eyes and then opened their eyes; determining whether or not the subject has had a change in focus based on the degree of convergence; measuring the pupil of the subject based on the image of the subject when it is determined that there is no focus change; Image processing methods.

10. measuring a degree of eye convergence of the subject after the subject has opened their eyes based on an image of the subject who has closed their eyes and then opened their eyes; determining whether or not the subject has had a change in focus based on the degree of convergence; A program that causes a computer to execute a process of measuring the pupil of the subject based on the image of the subject when it is determined that there is no focus change.