Information processing system, eye condition measurement system, information processing method and program

Abstract

An information processing system (10) comprises a first acquisition unit (11), a second acquisition unit (12), a movement control unit (13), and a state evaluation unit (15). The first acquisition unit (11) acquires, from a first imaging unit (30), image data pertaining to a first image in which the head of a subject is imaged with a first angle of view. The second acquisition unit (12) acquires, from a second imaging unit (40), image data pertaining to a second image in which the eye region of the subject is imaged with a second angle of view narrower than the first angle of view. The movement control unit (13) moves the field of view range of the second imaging unit (40) on the basis of position information which is for the head of the subject and which is obtained on the basis of the first image. The state evaluation unit (15) evaluates a change in the state of the eyes of the subject on the basis of time-series data of the second image.

Description

[Technical field]

[0001] The present disclosure relates to an information processing system, an eye condition measuring system, an information processing method, and program In particular, an information processing system for evaluating changes in the condition of a subject's eyes, an eye condition measuring system, an information processing method, and program Regarding. [Background technology]

[0002] A technology has been proposed for evaluating changes in the condition of a subject's eyes based on captured images of the subject's eyes. For example, Patent Document 1 discloses a method for calculating pupil movement from changes in pupil position detected based on captured images of the driver's face. However, the method described in Patent Document 1 above cannot detect the pupil position if the head moves and the eyeball moves out of the field of view of the camera. Therefore, the subject needs to fix the head so that the eyeball does not move out of the field of view of the camera.

[0003] Patent Document 2 also discloses a method for detecting the amount of eye movement based on the difference between a reference position indicating the position of the blood vessel image in the white of the eye in a reference image and the position of the blood vessel image in the white of the eye in a detection image. In this method, the eye image in the detection image is returned to the position of the reference image by Affine transformation from three points including the positions of the inner corner and outer corner of the eye, so that image blurring can be prevented. However, even in this method, the subject is required to keep his / her head fixed to some extent so that the eye does not move out of the field of view of the camera.

[0004] On the other hand, a method is known in which a specified area is enlarged and photographed using cameras with different angles of view. Patent Document 3 discloses an information processing system in which a target area specified in an image photographed by a first wide-angle camera is photographed by a second narrow-angle camera. However, Patent Document 3 above only photographs a person's face, and does not disclose photographing the eye area, which is a minute area. Patent Document 3 above also does not disclose a technique for measuring and evaluating minute changes in the state of the eyes. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-195377 A [Patent Document 2] International Publication No. 2016 / 195066 [Patent Document 3] International Publication No. 2012 / 001755 Summary of the Invention [Problem to be solved by the invention]

[0006] In view of the above-mentioned problems, an object of the present disclosure is to provide an information processing system, an eye condition measurement system, an information processing method, and a non-transitory computer-readable medium that can appropriately evaluate changes in the condition of a subject's eyes while the head is not fixed and the subject is in a relaxed state. [Means for solving the problem]

[0007] An information processing system according to an aspect of the present disclosure includes a first acquisition means, a second acquisition means, a movement control means, and a condition evaluation means. The first acquisition means acquires image data related to a first image obtained by capturing an image of a subject's head at a first angle of view from a first imaging means. The second acquisition means acquires image data related to a second image obtained by capturing an image of an eye region of the subject at a second angle of view narrower than the first angle of view from a second imaging means. The movement control means moves the field of view range of the second imaging means based on position information of the subject's head acquired based on the first image. The condition evaluation means evaluates a change in the condition of the subject's eyes based on time-series data of the second image.

[0008] An eye condition measuring system according to an aspect of the present disclosure includes a first imaging means, a second imaging means, and an information processing device. The first imaging means images the subject's head at a first angle of view. The second imaging means images the subject's eye region at a second angle of view narrower than the first angle of view. The information processing device includes a first acquisition means, a second acquisition means, a movement control means, and a condition evaluation means. The first acquisition means acquires image data related to a first image obtained by imaging the subject's head at a first angle of view from the first imaging means. The second acquisition means acquires image data related to a second image obtained by imaging the subject's eye region at a second angle of view narrower than the first angle of view from the second imaging means. The movement control means moves the field of view range of the second imaging means based on position information of the subject's head acquired based on the first image. The condition evaluation means evaluates a change in the condition of the subject's eyes based on time-series data of the second image.

[0009] An information processing method according to one aspect of the present disclosure includes acquiring, from a first imaging means, image data relating to a first image in which a subject's head is imaged at a first angle of view, acquiring, from a second imaging means, image data relating to a second image in which the subject's eye area is imaged at a second angle of view narrower than the first angle of view, moving the field of view range of the second imaging means based on position information of the subject's head acquired based on the first image, and evaluating changes in the condition of the subject's eyes based on time series data of the second image.

[0010] According to one aspect of the present disclosure program causes a computer to execute a first acquisition process, a second acquisition process, a movement control process, and a state evaluation process. do.The first acquisition process is a process of acquiring image data related to a first image obtained by capturing an image of the subject's head at a first angle of view from a first imaging means. The second acquisition process is a process of acquiring image data related to a second image obtained by capturing an image of the subject's eye region at a second angle of view narrower than the first angle of view from a second imaging means. The movement control process is a process of moving the field of view range of the second imaging means based on position information of the subject's head acquired based on the first image. The condition evaluation process is a process of evaluating a change in the condition of the subject's eyes based on time-series data of the second image. Effect of the Invention

[0011] The present disclosure provides an information processing system, an eye condition measuring system, an information processing method, and an image processing method for appropriately evaluating changes in the eye condition of a subject in a relaxed state without fixing the head. program can be provided. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a functional configuration of an information processing system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a hardware configuration of an information processing system according to a first embodiment. [Diagram 3] FIG. 11 is a system configuration diagram of an eye condition measuring system to which an information processing system according to a second embodiment can be applied. [Figure 4] FIG. 11 is a block diagram showing a functional configuration of an information processing device according to a second embodiment. [Diagram 5] FIG. 11 is a diagram for explaining a movement control process performed by a movement control unit according to the second embodiment. [Figure 6] 13 is a diagram for explaining a focal position control process performed by a movement control unit according to the second embodiment. FIG. [Figure 7] FIG. 11 is a diagram for explaining a state evaluation process performed by a state evaluation unit according to the second embodiment. [Figure 8] 10 is a flowchart showing the procedure of information processing in the information processing device according to the second embodiment. [Figure 9] FIG. 11 is a system configuration diagram of an eye condition measuring system according to a third embodiment. [Figure 10] FIG. 11 is a system configuration diagram of an eye condition measuring system according to a fourth embodiment. [Figure 11] FIG. 13 is a block diagram showing a functional configuration of an information processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the present disclosure will be described through embodiments, but the invention according to the claims is not limited to the following embodiments. In addition, all of the configurations described in the embodiments are not necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals.

[0014] <Embodiment 1> First, a first embodiment of the present disclosure will be described with reference to Figs. 1 and 2. Fig. 1 is a block diagram showing a functional configuration of an information processing system 10 according to the first embodiment. The information processing system 10 is a computer device that evaluates changes in the condition of the eyes of a subject. The information processing system 10 includes a first acquisition unit 11, a second acquisition unit 12, a movement control unit 13, and a condition evaluation unit 15.

[0015] The first acquisition unit 11 is also called a first acquisition means. The first acquisition unit 11 is communicably connected to the first imaging unit 30, and acquires image data (first image data) related to a first image obtained by capturing an image of the subject's head at a first angle of view from the first imaging unit 30. Here, the first imaging unit 30 is a camera that captures an image of the subject's head at the first angle of view. The first imaging unit 30 is also called a first imaging means.

[0016] The second acquisition unit 12 is also called a second acquisition means. The second acquisition unit 12 is communicatively connected to the second imaging unit 40, and acquires image data (second image data) related to a second image obtained by capturing an eye region of the subject at a second angle of view from the second imaging unit 40. Here, the second imaging unit 40 is a camera that captures an image of the eye region of the subject at the second angle of view. The second imaging unit 40 is also called a second imaging means. The second angle of view is narrower than the first angle of view. The eye region may be an eyeball or a surrounding area including the eyeball.

[0017] The movement control unit 13 is also called a movement control means. The movement control unit 13 moves the field of view of the second imaging unit 40 based on position information of the subject's head acquired based on the first image. The field of view of the second imaging unit 40 is the range imaged by the second imaging unit 40, and is also called a capture volume. The field of view is determined based on the angle of view and the camera optical axis, and the larger the angle of view, the wider the field of view.

[0018] The condition evaluation unit 15 is also called a condition evaluation means. The condition evaluation unit 15 evaluates a change in the condition of the subject's eyes based on the time-series data of the second images.

[0019] FIG. 2 is a block diagram showing a hardware configuration of the information processing system 10 according to the first embodiment.

[0020] The information processing system 10 has, as its main hardware components, a processor 100, a ROM 101 (Read Only Memory), a RAM 102 (Random Access Memory), and an interface unit 103 (IF; Interface). The processor 100, the ROM 101, the RAM 102, and the interface unit 103 are connected to each other via a data bus or the like.

[0021] The processor 100 has a function as a calculation device that performs control processing and calculation processing. The processor 100 may be a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a combination of these. The ROM 101 has a function for storing a control program and a calculation program executed by the processor 100. The RAM 102 has a function for temporarily storing processing data and the like. The interface unit 103 inputs and outputs signals to and from the outside via a wired or wireless connection. The interface unit 103 also accepts a data input operation by a user and displays information to the user. For example, the interface unit 103 communicates with the first imaging unit 30 and the second imaging unit 40.

[0022] In this way, according to the information processing system 10 of the first embodiment, the field of view of the narrow-angle camera is moved based on the position information of the subject's head acquired based on the first wide-angle image, so that the eye area is enlarged and the change in the condition of the eye is evaluated. Therefore, even if the subject's head is not fixed, it is possible to prevent the eye area from moving out of the field of view of the narrow-angle camera. This allows the information processing system 10 to appropriately evaluate the change in the condition of the subject's eyes while the head is not fixed and the subject is in a relaxed state.

[0023] <Embodiment 2> 3 is a system configuration diagram of an eye condition measuring system 1a to which an information processing system (hereinafter, referred to as an information processing device) according to embodiment 2 can be applied. In this diagram, the left-right direction of the subject P is defined as the X-axis direction, the front-back direction of the subject P is defined as the Y-axis direction, and the up-down direction of the subject P is defined as the Z-axis direction.

[0024] Here, the eye condition measuring system 1a is a computer system that measures and evaluates changes in the eye condition of a subject. In this embodiment 2, the eye condition measuring system 1a measures and evaluates the vibration state of the pupil caused by the saccade phenomenon. The eye condition measuring system 1a includes an information processing device 10a, a first imaging unit 30a, a second imaging unit 40, a movable mirror 50, a driving unit 51, and a light source 61. The distance between the first imaging unit 30a and the subject P in the Y-axis direction is D. D is, for example, 2±0.5 [m].

[0025] The information processing device 10a corresponds to the information processing system 10 in FIG. 1. The information processing device 10a is connected to the first imaging section 30a, the second imaging section 40, and the driving section 51. In response to receiving the first image data from the first imaging section 30a, the information processing device 10a generates a control signal required for the eye area of ​​the subject P to fall within the field of view of the second imaging section 40 and a control signal required for focusing the second imaging section 40 on the eye area of ​​the subject P. The information processing device 10a then transmits the control signal for the field of view in the X-axis and Z-axis directions to the driving section 51 and the control signal for the focal position to the second imaging section 40.

[0026] The first imaging unit 30a is a camera having the same function as the first imaging unit 30 in FIG. 1. In the present embodiment 2, the first imaging unit 30a has a wide-angle camera 31. The wide-angle camera 31 captures at least the face of the subject P at a first angle of view and generates first image data. The focal length of the wide-angle camera 31 is set in advance so that the face of the subject P, which is located a distance of D [m], can be captured. For example, the focal length of the wide-angle camera 31 may be less than 200 [mm], preferably less than 70 [mm], and is 28 [mm] in the present embodiment 2. The focal length of the wide-angle camera 31 may be 12 [mm].

[0027] The frame rate of the wide-angle camera 31 is set in advance so as to be able to track the shaking of the eye area accompanying the shaking of the head of the subject P and to suppress an excessive increase in the amount of data. For example, the frame rate of the wide-angle camera 31 may be 120 [fps] or more and 1200 [fps] or less, preferably 240 [fps] or more and 1000 [fps] or less, more preferably 480 [fps] or more and 1000 [fps] or less, and is 500 [fps] in the second embodiment. Here, the wide-angle camera 31 may capture an image of a subject at a frame rate equal to or higher than the frame rate of the second imaging unit 40 described later. This makes it easy to suitably control the field of view of the second imaging unit 40 according to the shaking of the eye area. In response to generating the first image data, the wide-angle camera 31 transmits the first image data to the information processing device 10a.

[0028] The second imaging unit 40 is a camera that captures the subject's eye region at a second angle of view narrower than the first angle of view and generates second image data. In order to facilitate pupil detection, which will be described later, the second imaging unit 40 may be a near-infrared camera. If the detection wavelength of the light receiving element of the second imaging unit 40 is called the pupil detection wavelength, the pupil detection wavelength is, for example, a wavelength of 940 [nm]. The second imaging unit 40 also has a telephoto lens and a liquid lens. The liquid lens is a lens that determines the focal position (focus point) and may be controlled based on a control signal for the focal position by the information processing device 10a. The focal length of the second imaging unit 40 is set so that the eye region of the subject P, which is located D [m] away from the movable mirror 50, can be captured, and is longer than the focal length of the wide-angle camera 31. For example, the focal length of the second imaging unit 40 may be 100 [mm] or more, preferably 150 [mm] or more, and is 200±50 [mm] in the second embodiment. Here, the second imaging unit 40 has a very narrow depth of field because of its long focal length. Therefore, the second imaging unit 40 controls the focal position based on the distance information in order to focus on the eye region. The frame rate of the second imaging unit 40 is set in advance so that the state change of the eye, in this example, the saccade phenomenon, can be observed and the amount of data can be suppressed from increasing excessively. For example, the frame rate of the second imaging unit 40 may be 120 [fps] or more and 1000 [fps] or less, preferably 240 [fps] or more and 1000 [fps] or less, more preferably 500 [fps] or more and 1000 [fps] or less, and is 500 [fps] in the second embodiment. The second imaging unit 40 transmits the second image data to the information processing device 10a in response to generating the second image data.

[0029] The movable mirror 50 is a pair of mirrors that move the optical axis of the second imaging unit 40. The movable mirror 50 includes a first movable mirror 50-1 and a second movable mirror 50-2. In the following, when the first movable mirror 50-1 and the second movable mirror 50-2 are not distinguished from each other, they are simply called the movable mirror 50. Each movable mirror 50 is fixedly connected to the driving unit 51 via a support unit (not shown) so as to have a predetermined tilt angle, and is configured so that the tilt angle can be changed by rotating the support unit. For example, the first movable mirror 50-1 is configured so that the support unit to which it is connected rotates around the Z axis, and the second movable mirror 50-2 is configured so that the support unit to which it is connected rotates around the X axis. As a result, the movable mirror 50 can move the optical axis of the second imaging unit 40 in the X-axis direction and the Z-axis direction, that is, the field of view range of the second imaging unit 40 in the X-axis direction and the Z-axis direction.

[0030] In the present embodiment 1, the movable mirror 50 is a galvanometer mirror having a relatively small mass and high responsiveness. This makes it easy for the eye condition measuring system 1a to image the eye region by linking the field of view of the second imaging unit 40 with minute and high-speed movement of the head of the subject P.

[0031] The driving unit 51 is also called a driving means. The driving unit 51 is a driving motor that rotates each movable mirror 50 via a support unit. The driving unit 51 has a first driving unit 51-1 corresponding to the first movable mirror 50-1 and a second driving unit 51-2 corresponding to the second movable mirror 50-2. The first driving unit 51-1 and the second driving unit 51-2 may also be simply called the driving unit 51. Here, the driving unit 51 rotates the movable mirror 50 according to a control signal of the field of view range in the X-axis and Z-axis directions of the information processing device 10a.

[0032] The light source 61 is a light source that illuminates the face of the subject P. The light source 61 is a light source having a wavelength range corresponding to the wavelength for pupil detection, and is a near-infrared light source of 940 [nm] in the second embodiment.

[0033] That is, the second imaging section 40 captures an image of the eye region of the subject P by receiving light from the path of the eye region of the subject P, the first movable mirror 50-1, the second movable mirror 50-2, and then the second imaging section 40.

[0034] 4 is a block diagram showing a functional configuration of an information processing device 10a according to embodiment 2. The information processing device 10a in FIG. 4 has an element position estimation unit 14 and an output unit 16 in addition to the components of the information processing device 10 in FIG.

[0035] The first acquisition unit 11 is connected to the first imaging unit 30a (wide-angle camera 31) and receives and acquires the first image data from the first imaging unit 30a. The first acquisition unit 11 may acquire the first image data at a frame rate equal to or higher than the frame rate at which the second acquisition unit 12 acquires the second image data. The first acquisition unit 11 supplies the acquired first image data to the movement control unit 13.

[0036] The second acquisition unit 12 is connected to the second imaging unit 40, and receives and acquires second image data from the second imaging unit 40. The second acquisition unit 12 supplies the acquired second image data to the element position estimation unit 14. The second acquisition unit 12 may also supply the acquired second image data to the movement control unit 13.

[0037] The movement control unit 13 generates change information of the position of the face of the subject P in the X-axis and Z-axis directions based on the time series data of the first image. The movement control unit 13 calculates the amount of rotation of each of the first movable mirror 50-1 and the second movable mirror 50-2 based on the change information. The movement control unit 13 may use the second image data in addition to the change information to calculate the amount of rotation. Then, the movement control unit 13 generates a control signal of the field of view range in the X-axis and Z-axis directions based on each amount of rotation, and transmits the control signal of the field of view range in the Z-axis direction to the first driving unit 51-1 and the control signal of the field of view range in the X-axis direction to the second driving unit 51-2. The movement control unit 13 also performs a focus position control process based on the time series data of the first image and the second image, generates change information of the position of the face of the subject P in the Y-axis direction, and generates a control signal of the focus position. Then, the movement control unit 13 controls the liquid lens by transmitting the control signal to the second imaging unit 40. In this way, the second imaging unit 40 moves its field of view range and focus position (focus point) based on the change information of the position of the face.

[0038] The element position estimation unit 14 is also called element position estimation means. The element position estimation unit 14 estimates the position of the eye element in the second image. In the second embodiment, the position of the eye element estimated by the element position estimation unit 14 includes the position of the center of gravity of the pupil of the subject P and the positions of the outer corner and inner corner of the eye. Note that the positions of the outer corner and inner corner of the eye may be either one of them, or may be any other fixed point position instead. The element position estimation unit 14 supplies position information of the eye element to the state evaluation unit 15.

[0039] The state evaluation unit 15 generates difference information of the positions of the eye elements in the second image, and evaluates the vibration state of the pupil of the subject P based on the difference information. The vibration state of the pupil may be at least one of the vibration amount, vibration direction, vibration frequency, and vibration duration of the pupil. The state evaluation unit 15 supplies information on the vibration state of the pupil to the output unit 16.

[0040] The output unit 16 is also called an output means. The output unit 16 outputs information about the vibration state of the pupil as an evaluation result. The output unit 16 may have a display unit (not shown) that displays the evaluation result. The output unit 16 may also have a transmission unit (not shown) that transmits the evaluation result to an external device (not shown).

[0041] Here, the movement control process will be described in detail with reference to Fig. 5. Fig. 5 is a diagram for explaining the movement control process by the movement control unit 13 according to the second embodiment. In this figure, a second image IMG_b is shown. In the second image IMG_b, an image of the eye region including the inner corner Ib and the outer corner Ob of the eye of the subject P is captured.

[0042] First, the movement control unit 13 calculates the position coordinates of the eye region in the first image (referred to as eye position coordinates). In the second embodiment, the eye position coordinates are the position coordinates of the center of gravity of the eye region in the first image when the eye region in the first image is approximated to a substantially elliptical region, but are not limited thereto and may be the position coordinate range of the eye region in the first image. Next, the movement control unit 13 calculates the position coordinates of the center of gravity VC of the image region (projection region) VA in the second image corresponding to the eye position coordinates in the first image when the eye region in the first image is virtually projected onto the second image IMG_b. At this time, the movement control unit 13 may calculate the position coordinates of the center of gravity VC of the projection region VA based on the difference information of the eye position coordinates of the first image between the previous and current shooting timings and the second image at the previous shooting timing. Note that the difference information of the eye position coordinates is an example of the change information of the face position described above. Then, the movement control unit 13 calculates the rotation amount of the first movable mirror 50-1 and the second movable mirror 50-2 so that the center of gravity VC of the projection region VA is located at the center C of the second image. This allows the movement control unit 13 to move the field of view of the second imaging unit 40 in the X-axis and Z-axis directions so that the eye region A is always positioned at the center of the second image IMG_b. Note that in this figure, the projection region VA and eye region A are approximately elliptical regions symmetrical with respect to the major and minor axes, and the midpoint between the outer corner Ob and inner corner Ib of the eye, which are located at both ends of each of the projection region VA and eye region A, coincides with their center of gravity.

[0043] The number of pixels in the width direction of the eye region included in the second image IMG_b is set to be within a predetermined range with respect to the number of pixels in the width direction of the second image IMG_b. The number of pixels in the width direction of the second image IMG_b is xb, and the number of pixels in the height direction is zb. As an example, xb×zb=640×480. The number of pixels from the inner corner Ib of the eye to the outer corner Ob of the eye of the subject P captured in the second image IMG_b is x1. That is, x1 / xb is maintained within a predetermined range. For example, x1 / xb may be 0.5 or more and less than 1, and is preferably 0.8 or more and less than 1. This allows the information processing device 10a to accurately detect the pupil from the second image IMG_b.

[0044] Here, when the movement control unit 13 controls the movement of the first and second movable mirrors 50-1, 50-2 so that the eye region A is located at the center of the second image IMG_b, the amount of rotation differs depending on the distance D between the camera and the subject P in the Y-axis direction. Therefore, the movement control unit 13 needs to adjust the focus of the second image IMG_b.

[0045] 6 is a diagram for explaining the focus position control process by the movement control unit 13 according to the second embodiment. In this diagram, a schematic top view of the subject P, the first imaging unit 30a, and the second imaging unit 40 is shown on the upper side, and a first image IMG_a of the first imaging unit 30a and a second image IMG_b of the second imaging unit 40 are shown on the lower side.

[0046] First, the movement control unit 13 calculates the position coordinates (xai, zai) of the inner corner Ia of the subject P in the first image IMG_a. Then, the movement control unit 13 acquires the second image IMG_b by controlling the rotation of the movable mirror 50. The angle of the movable mirror 50 at this time is represented as G (xg, zg). The movement control unit 13 calculates the position coordinates (xbi, zbi) of the inner corner Ib of the second image IMG_b. The movement control unit 13 calculates the distance D by the stereo imaging method using the position coordinates (xai, zai) of the inner corner Ia of the eye IMG_a in the first image, the position coordinates (xbi, zbi) of the eye Ib of the second image IMG_b, and the angle G (xg, zg) of the movable mirror 50. Then, the movement control unit 13 determines the focal position of the second imaging unit 40 based on the distance D and generates control information for moving the focus to the focal position. That is, the movement control unit 13 treats the first imaging unit 30a and the second imaging unit 40 as if they were stereo cameras, and adjusts the focus of the second imaging unit 40, which has a narrow depth of field, to be focused on the subject P's eye position. In the above example, the movement control unit 13 uses the position coordinates of the inner corners Ia and Ib of the eye, but may use the position coordinates of the outer corners Oa and Ob of the eye instead.

[0047] Next, details of the condition evaluation process will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining the condition evaluation process by the condition evaluation unit 15 according to the second embodiment. The second image IMG_b is also shown in this figure.

[0048] First, the state evaluation unit 15 calculates the relative position of the center of gravity G of the pupil, which is a moving point, from the reference point for the second image IMG_b at each shooting timing. The reference point is preferably a fixed point whose position does not change substantially depending on the degree of opening of the eyelids or the movement of the line of sight. Here, the state evaluation unit 15 can use the position of the corneal reflection image as the reference point, but in the second embodiment, a point based on the position of at least one of the outer corner Ob and the inner corner Ib of the eye is used. That is, the state evaluation unit 15 calculates the relative position based on the position information of at least one of the outer corner Ob and the inner corner Ib of the eye and the position information of the center of gravity G of the pupil. For example, the state evaluation unit 15 calculates the relative position between the straight line L connecting the positions of the outer corner Ob and the inner corner Ib of the eye and the center of gravity G of the pupil. The state evaluation unit 15 calculates the distances Δx and Δz in the X-axis and Z-axis directions from the midpoint of the outer corner Ob and the inner corner Ib of the eye (which may coincide with the center C of the second image IMG_b) to the center of gravity G as the relative position of the center of gravity G of the pupil. The state evaluation unit 15 calculates the distances Δx and Δz in the X-axis and Z-axis directions as 1 The normalized value may be calculated as the relative position of the center of gravity G of the pupil in the second image IMG_b. The state evaluation unit 15 may use only one of the positions of the outer corner Ob and the inner corner Ib of the eye as a reference point. The state evaluation unit 15 then evaluates the vibration state of the pupil based on difference information between adjacent shooting timings of the relative position of the center of gravity G of the pupil.

[0049] In this way, the condition evaluation unit 15 uses at least one of the positions of the outer corner Ob and the inner corner Ib of the eye as the reference point. Therefore, the eye condition measurement system 1a does not need to include an infrared light source for forming a corneal reflection image, compared to a case where a corneal reflection image is used, and the system configuration is simplified. In addition, the condition evaluation unit 15 is capable of high-speed processing and miniaturization of the device because the calculation for calculating the relative position is simple and the amount of calculation is reduced. Therefore, although the information processing device 10a has been described as an independent computer device in the second embodiment, the information processing device 10a can also be implemented in the first imaging unit 30a or the second imaging unit 40.

[0050] FIG. 8 is a flowchart showing the procedure of information processing by the information processing device 10a according to the second embodiment.

[0051] First, the first acquisition unit 11 acquires the image from the first imaging unit 30a at t=t i The first image data is acquired at a predetermined frame rate (step S10).

[0052] Next, the movement control unit 13 detects the face of the subject P from the first image (step S11). For example, the movement control unit 13 may detect the face of the subject P using a trained convolutional neural network (CNN) that receives the first image as an input. Then, the movement control unit 13 may generate a normalized image by cutting out an image area corresponding to the face of the subject P.

[0053] Next, the movement control unit 13 calculates eye position coordinates in the first image (step S12). For example, the movement control unit 13 may detect the eye region of the subject P from the generated normalized image by template matching, and calculate the position coordinates of the center of gravity of the eye region in the first image as the eye position coordinates.

[0054] Then, the movement control unit 13 calculates the time t=t i The eye position coordinates of the first image in are t=t i-1 The movement control unit 13 compares the eye position coordinates of the first image in the image captured by the camera 1 with the eye position coordinates of the first image captured by the camera 1 in the ... i+1 ) (step S14). At this time, the movement control unit 13 calculates the amount of rotation of the movable mirror 50 at the current or previous shooting timing from the second imaging unit 40 (for example, i-1The second image data (second image data of the second imaging unit 40) may be acquired, and the amount of rotation of the movable mirror 50 may be calculated based on the eye position coordinates and the second image. The movement control unit 13 then generates a control signal for the field of view range in the X-axis and Z-axis directions based on the calculated amount of rotation. In addition, the movement control unit 13 calculates the amount of movement of the focal position of the second imaging unit 40 by the method shown in FIG. 6, and generates a control signal for the focal position based on the amount of movement.

[0055] Next, the movement control unit 13 controls the movement of the field of view and the focal position of the second imaging unit 40 (step S15). Specifically, the movement control unit 13 transmits a control signal for the field of view in the X-axis and Z-axis directions to the driving unit 51. The movement control unit 13 also transmits a control signal for the focal position to the liquid lens.

[0056] Next, the movement control unit 13 determines whether or not to end the series of photographing (step S16). If the series of photographing is not to be ended (No in step S16), the movement control unit 13 returns the process to step S10, and if the series of photographing is to be ended (Yes in step S16), the movement control unit 13 advances the process to step S17.

[0057] In step S17, the second acquisition unit 12 acquires the second image data from the second imaging unit .

[0058] Here, the information processing device 10a repeats the processes of steps S18 to S19 the same number of times as the number of frames of the acquired second image.

[0059] In step S18, the element position estimation unit 14 detects the eye region from the second image, for example, by template matching, and estimates the position coordinates of the eye elements (the outer corner of the eye, the inner corner of the eye, and the center of gravity of the pupil). The element position estimation unit 14 may estimate the position coordinates of the center of gravity of the pupil by detecting the image area of ​​the pupil, for example, by binarization, edge detection, and Hough transform, and calculating the position coordinates of the center of gravity of the image area.

[0060] Then, in step S19, the state evaluation unit 15 calculates the relative position of the center of gravity of the pupil by the method shown in FIG. 7 described above, and generates difference information of the relative position, thereby evaluating the vibration state.

[0061] Next, in step S20, the output unit 16 outputs information on the evaluated vibration state.

[0062] The processes of steps S17 to S19 may be executed in parallel with the processes of steps S10 to 16. Furthermore, the specific processes of each step are not limited to those described above.

[0063] In this way, the information processing device 10a of the second embodiment can achieve the same effect as that of the first embodiment. In particular, in order to measure the microvibration of the eyeball, it is necessary to strictly suppress the vibration of the head, so the head needs to be firmly fixed. Therefore, it is difficult to fix the head immediately after the subject has undergone head surgery due to a brain disease, so it is not possible to examine the eye condition such as the saccade phenomenon. Furthermore, with the current technology, it is only possible to examine the reaction of the brain in a tense state by fixing the head. This is true not only in the examination of the recovery state of the brain disease, but also in the case of judging the degree of interest in an image when the subject watches an image.

[0064] However, the information processing system 10a of the second embodiment can solve such problems. As a result, for example, the evaluation value evaluated by the information processing system 10a can be used as a criterion for determining the transition stage of brain disease rehabilitation, and medical professionals can easily diagnose the degree of recovery from the condition while reducing the burden on the patient subject. In addition, by collecting the evaluation value evaluated by the information processing system 10a as the interest level of the consumer subject in the advertising image, companies can quantitatively measure the effectiveness of their advertising.

[0065] <Embodiment 3> Next, a third embodiment of the present disclosure will be described with reference to Fig. 9. The third embodiment is characterized in that the field of view of the first imaging unit moves in response to the movement of the subject's head, and the field of view of the second imaging unit also moves accordingly.

[0066] FIG. 9 is a system configuration diagram of an eye condition measuring system 1b according to the third embodiment. The eye condition measuring system 1b according to the embodiment 3 has basically the same configuration and function as the eye condition measuring system 1a according to the embodiment 2. However, the eye condition measuring system 1b differs from the eye condition measuring system 1a in that the eye condition measuring system 1b has a first imaging unit 30b and an information processing device 10b instead of the first imaging unit 30a and the information processing device 10a, and further has a light source 63, a long-pass filter 70, and a half mirror 71.

[0067] The first imaging unit 30b has a tracking camera 32 whose field of view moves according to the movement of the head of the subject P. In the third embodiment, the field of view of the tracking camera 32 moves according to the movement of the eye region of the subject P so that the eye region of the subject P is imaged in the center of the first image. Specifically, the field of view of the tracking camera 32 moves by the rotation of the movable mirror 50 by the drive unit 51 according to the control signal of the field of view of the information processing device 10b, similar to the second imaging unit 40. That is, the movable mirror 50 moves the optical axis of the first imaging unit 30b in addition to the optical axis of the second imaging unit 40. As a result, the field of view of the second imaging unit 40 moves in conjunction with the movement of the field of view of the tracking camera 32. Here, if the detection wavelength of the light receiving element of the tracking camera 32 is called the tracking wavelength, the tracking wavelength is smaller than the pupil detection wavelength, which is the detection wavelength of the light receiving element of the second imaging unit 40. As an example, the tracking wavelength is 850 [nm]. The angle of view, focal length, and frame rate of the tracking camera 32 are the same as those of the wide-angle camera 31 in the second embodiment.

[0068] The light source 63 is a light source that irradiates the eye region of the subject P. The light source 63 is a light source having a wavelength region corresponding to the tracking wavelength, and is a near-infrared light source of 850 [nm] in the third embodiment.

[0069] The half mirror 71 is a half mirror that reflects a portion of the 850 [nm] light incident from the light source 63 toward the long pass filter 70. The half mirror 71 also transmits a portion of the 850 [nm] light incident from the long pass filter 70 toward the tracking camera 32. Note that the half mirror 71 may be replaced by a beam splitter with an arbitrary ratio of transmission and reflection.

[0070] The long-pass filter 70 is an optical filter that transmits light of the wavelength for pupil detection and reflects light of the wavelength for tracking. The long-pass filter 70 is disposed between the second imaging unit 40 and the movable mirror 50, and transmits light of the wavelength for pupil detection incident from the movable mirror 50 toward the second imaging unit 40. The long-pass filter 70 also reflects light of the wavelength for tracking incident from the movable mirror 50 toward the half mirror 71.

[0071] That is, part of the light from the light source 63 reaches the eye region of the subject P through the path of the half mirror 71 → the long-pass filter 70 → the second movable mirror 50-2 → the first movable mirror 50-1. The tracking camera 32 of the first imaging unit 30b captures the eye area (or face) of the subject P at a first angle of view by receiving light from the path of the eye area (or face) of the subject P → first movable mirror 50-1 → second movable mirror 50-2 → long-pass filter 70 → half mirror 71 → tracking camera 32.

[0072] In addition, the second imaging unit 40 images the eye area of ​​the subject P at a second angle of view by receiving light from the path of the eye area of ​​the subject P → the first movable mirror 50-1 → the second movable mirror 50-2 → the long-pass filter 70 → the second imaging unit 40.

[0073] The information processing device 10b has basically the same configuration and functions as the information processing device 10a, but the movement control process in the X-axis and Z-axis directions by the movement control unit 13 is different from that in the second embodiment (the process shown in step S14 in FIG. 5 and FIG. 8).

[0074] In the third embodiment, the movement control unit 13 first calculates the eye position coordinates in the first image. Then, the movement control unit 13 calculates the rotation amount of the first movable mirror 50-1 and the second movable mirror 50-2 so that the center of gravity of the eye area in the first image is located at the center of the first image at the next shooting timing. Here, since the tracking camera 32 of the first imaging unit 30b has a frame rate equal to or higher than the frame rate of the second imaging unit 40, the eye area of ​​the subject P will be captured at the center of the image in the second image captured by the second imaging unit 40.

[0075] As described above, according to the third embodiment, the information processing device 10b can move the field of view of the second imaging unit 40 with higher accuracy, following the movement of the head of the subject P. This allows the information processing device 10b to appropriately evaluate the change in the eye condition of the subject P in a relaxed state without fixing the head.

[0076] <Embodiment 4> Next, a fourth embodiment of the present disclosure will be described with reference to Figures 10 and 11. The fourth embodiment is characterized in that the position of the narrowest angle camera's field of view is roughly adjusted using the widest angle image, and the position of the field of view is finely adjusted using an image with a medium angle of view.

[0077] 10 is a system configuration diagram of an eye condition measuring system 1c according to embodiment 4. The eye condition measuring system 1c according to embodiment 4 basically has the same configuration and functions as the eye condition measuring system 1b according to embodiment 3. However, the eye condition measuring system 1c differs from the eye condition measuring system 1b in that it has a third imaging unit 33 and an information processing device 10c instead of the information processing device 10b.

[0078] The third imaging unit 33 is a camera that captures at least the face of the subject P at a third angle of view and generates third image data related to the third image. The third angle of view is larger than the first angle of view of the tracking camera 32 and larger than the second angle of view of the second imaging unit 40. In response to generating the third image data, the third imaging unit 33 transmits the third image data to the information processing device 10c.

[0079] The information processing device 10c has basically the same functions as the information processing device 10b, but differs from the information processing device 10b in that it roughly adjusts the movement control of the field of view of the tracking camera 32 and the second imaging unit 40 based on the third image data.

[0080] 11 is a block diagram showing a functional configuration of an information processing device 10c according to embodiment 4. The information processing device 10c has a third acquisition unit 17 in addition to the configuration of the information processing device 10b. The third acquisition unit 17 receives and acquires third image data from the third imaging unit 33, and supplies the third image data to the movement control unit 13.

[0081] The movement control unit 13 detects the face of the subject P from the third image, calculates the eye position coordinates in the third image, and calculates the amount of rotation of the movable mirror 50 so that the eye area fits within the first image based on the eye position coordinates. The movement control unit 13 then transmits a control signal for the field of view range based on the amount of rotation to the drive unit 51, and roughly adjusts the tilt angle of the movable mirror 50.

[0082] Such a coarse adjustment process may be executed before step S10 in Fig. 8. Furthermore, in order to prevent an increase in processing time, the coarse adjustment process may be executed, for example, only when the face of subject P is not detected in the first image in step S11 in Fig. 8 or when the eye region of subject P is not detected in the first image in step S12.

[0083] The movement control process using the first image having a first angle of view narrower than the third image corresponds to steps S10 to S15 in Fig. 8, but can also be called a fine adjustment process. The first angle of view in the fourth embodiment may be an angle of view necessary for capturing an image of at least the eye region of the subject P, and may be narrower than the first angle of view in the second and third embodiments. In this case, in the fine adjustment process, the face detection process corresponding to step S11 in Fig. 8 may be omitted. This can prevent an increase in the series of processing times due to the addition of the coarse adjustment process.

[0084] In this way, even if the head of the subject P moves significantly, the information processing device 10c of the fourth embodiment can move the field of view of the second imaging unit 40 so that the eye area of ​​the subject P does not go out of the field of view of the second imaging unit 40. This allows the subject P to move freely even during imaging, so that the information processing device 10c can evaluate changes in the eye condition in a more relaxed state.

[0085] Although the present disclosure has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the invention. For example, the eye condition to be evaluated is the amount of pupil contraction, and the condition evaluation unit 15 may evaluate the amount of pupil contraction of the subject P based on the time series data of the second image. The condition evaluation unit 15 may then evaluate the degree of interest in viewing an image such as an advertisement image based on the amount of pupil contraction.

[0086] In the above embodiment, the present disclosure has been described as a hardware configuration, but the present disclosure is not limited to this. The present disclosure can also be realized by having a processor execute a computer program to perform various processes related to the state evaluation method.

[0087] In the above example, the program can be stored and provided to the computer using various types of non-transitory computer readable media. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can provide the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

[0088] In the above-mentioned embodiment, the computer is configured as a computer system including a personal computer, a word processor, etc. However, the present invention is not limited to this, and the computer can also be configured as a server of a LAN (local area network), a host for computer (personal computer) communication, a computer system connected to the Internet, etc. It is also possible to distribute functions to each device on the network and configure a computer as a whole network.

[0089] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) a first acquisition unit that acquires image data relating to a first image obtained by capturing an image of the subject's head at a first angle of view from a first imaging unit; a second acquisition unit that acquires image data relating to a second image obtained by capturing an eye region of the subject at a second angle of view narrower than the first angle of view from a second imaging unit; a movement control unit that moves a field of view of the second imaging unit based on position information of the head of the subject acquired based on the first image; a condition evaluation unit that evaluates a change in the condition of the subject's eye based on the time-series data of the second image; An information processing system comprising: (Appendix 2) The movement control unit generates change information of a position of the head of the subject based on time-series data of the first image, and moves a field of view of the second imaging unit based on the change information. 2. An information processing system according to claim 1. (Appendix 3) The first acquisition unit acquires image data related to the first image at a frame rate equal to or higher than a frame rate at which the second acquisition unit acquires image data related to the second image. 3. An information processing system according to claim 2. (Appendix 4) an element position estimation unit that estimates a center position of the pupil of the subject in the second image and a position of at least one of an outer corner and an inner corner of the eye, The state evaluation unit evaluates a vibration state of the subject's pupil based on position information of at least one of the outer corner and the inner corner of the eye and centroid position information of the pupil. 4. An information processing system according to any one of claims 1 to 3. (Appendix 5) The state evaluation unit calculates a relative position of a center of gravity of the pupil with respect to a line connecting the positions of the outer corner and the inner corner of the eye, and evaluates a vibration state of the pupil of the subject based on the relative position. 5. An information processing system according to claim 4. (Appendix 6) The state evaluation unit evaluates an amount of pupil contraction of the subject based on time-series data of the second image. 4. An information processing system according to any one of claims 1 to 3. (Appendix 7) a first imaging unit that images the subject's head at a first angle of view; a second imaging unit that images an eye region of the subject at a second angle of view narrower than the first angle of view; Information processing device An eye condition measuring system comprising: The information processing device includes: a first acquisition unit that acquires image data relating to a first image from the first imaging unit; a second acquisition unit that acquires image data relating to a second image from the second imaging unit; a movement control unit that moves a field of view of the second imaging unit based on position information of the head of the subject acquired based on the first image; a condition evaluation unit that evaluates a change in the condition of the subject's eye based on the time-series data of the second image; An eye condition measuring system having (Appendix 8) A movable mirror that moves the optical axis of the second imaging unit; A drive unit that rotates the movable mirror; Further equipped with The movement control unit generates change information of a position of the head of the subject based on time-series data of the first image, and calculates an amount of rotation of the movable mirror based on the change information. 8. An eye condition measuring system as described in appendix 7. (Appendix 9) The movable mirror moves the optical axes of the first imaging unit and the second imaging unit. 9. An eye condition measuring system as described in appended claim 8. (Appendix 10) The first imaging unit captures an image of a subject at a frame rate equal to or higher than a frame rate of the second imaging unit. An eye condition measuring system according to any one of claims 7 to 9. (Appendix 11) A first acquisition step of acquiring image data relating to a first image obtained by capturing an image of the subject's head at a first angle of view from a first imaging unit; a second acquisition step of acquiring image data relating to a second image obtained by capturing an eye region of the subject at a second angle of view narrower than the first angle of view from a second imaging unit; a movement control step of moving a field of view of the second imaging unit based on position information of the head of the subject acquired based on the first image; a condition evaluation step of evaluating a change in the condition of the subject's eye based on the time-series data of the second image; An information processing method comprising: (Appendix 12) a first acquisition process for acquiring image data relating to a first image obtained by capturing an image of the subject's head at a first angle of view from a first imaging unit; a second acquisition process for acquiring image data relating to a second image obtained by capturing an eye region of the subject at a second angle of view narrower than the first angle of view from a second imaging unit; a movement control process for moving a field of view of the second imaging unit based on position information of the head of the subject acquired based on the first image; A condition evaluation process for evaluating a change in the condition of the subject's eye based on the time-series data of the second image. A program for causing a computer to execute the following.

[0090] This application claims priority based on Japanese Patent Application No. 2020-176066, filed on October 20, 2020, the disclosure of which is incorporated herein in its entirety. [Industrial Applicability]

[0091] The information processing system according to the present disclosure can be used to evaluate changes in the condition of a subject's eyes. [Explanation of symbols]

[0092] 1a, 1b, 1c Eye condition measurement system 10, 10a, 10b, 10c Information processing system (information processing device) 11 First acquisition part 12 Second acquisition part 13 Movement control section 14 Element position estimation part 15 Condition Assessment Section 16 Output section 17 Third acquisition part 30, 30a, 30b First imaging unit 31 Wide-angle camera 32 Tracking Camera 33 Third Imaging Unit 40 Second imaging unit 50 Movable mirror 51 Drive unit 61 Light source 63 Light source 70 Longpass Filter 71 Half Mirror 100 processors 101 ROM 102 RAM 103 Interface section (IF) P Subject IMG_a 1st image IMG_b 2nd image

Claims

1. a first acquisition means for acquiring image data relating to a first image obtained by capturing an image of the subject's head at a first angle of view from a first imaging means; a second acquiring means for acquiring image data relating to a second image obtained by capturing an eye region of the subject at a second angle of view narrower than the first angle of view from a second imaging means; a movement control means for moving a field of view of the second imaging means based on position information of the head of the subject acquired based on the first image; a condition evaluation means for evaluating a change in the condition of the subject's eye based on the time-series data of the second image; Equipped with the first acquisition means acquires image data related to the first image at a frame rate equal to or higher than a frame rate at which the second acquisition means acquires image data related to the second image; The frame rate of the first acquisition means is equal to or greater than 480 [fps] and equal to or less than 1000 [fps], An information processing system, wherein the frame rate of the second acquisition means is equal to or greater than 120 [fps] and equal to or less than 1000 [fps].

2. The movement control means generates change information of the position of the head of the subject based on the time-series data of the first image, and moves the field of view of the second imaging means based on the change information. The information processing system according to claim 1 .

3. an element position estimating means for estimating a position of a center of gravity of the pupil of the subject in the second image and a position of at least one of an outer corner and an inner corner of the eye, The state evaluation means evaluates a vibration state of the subject's pupil based on position information of at least one of the outer corner and the inner corner of the eye and position information of the center of gravity of the pupil.

3. The information processing system according to claim 1 or 2.

4. The state evaluation means calculates a relative position of the center of gravity of the pupil with respect to a line connecting the positions of the outer corner and the inner corner of the eye, and evaluates a vibration state of the pupil of the subject based on the relative position. The information processing system according to claim 3 .

5. The condition evaluation means evaluates an amount of pupil contraction of the subject based on the time-series data of the second image.

3. The information processing system according to claim 1 or 2.

6. a first imaging means for imaging a head of a subject at a first angle of view; a second imaging means for imaging an eye region of the subject at a second angle of view narrower than the first angle of view; Information processing device An eye condition measuring system comprising: The information processing device includes: a first acquisition means for acquiring image data relating to a first image from the first imaging means; a second acquisition means for acquiring image data relating to a second image from the second imaging means; a movement control means for moving a field of view of the second imaging means based on position information of the head of the subject acquired based on the first image; a condition evaluation means for evaluating a change in the condition of the subject's eye based on the time-series data of the second image; having the first imaging means images a subject at a frame rate equal to or higher than a frame rate of the second imaging means; The frame rate of the first imaging means is 480 [fps] or more and 1000 [fps] or less, An eye condition measuring system, wherein the frame rate of the second imaging means is 120 [fps] or more and 1000 [fps] or less.

7. a movable mirror that moves the optical axis of the second imaging means; A driving means for rotating the movable mirror; Further equipped with The movement control means generates change information of a position of the head of the subject based on the time-series data of the first image, and calculates an amount of rotation of the movable mirror based on the change information. The eye condition measuring system according to claim 6.

8. The movable mirror moves the optical axes of the first imaging means and the second imaging means. The eye condition measuring system according to claim 7.

9. acquiring image data relating to a first image obtained by capturing an image of the subject's head at a first angle of view from a first imaging means; acquiring image data relating to a second image obtained by capturing an eye region of the subject at a second angle of view narrower than the first angle of view from a second imaging means; moving a field of view of the second imaging means based on position information of the head of the subject acquired based on the first image; An information processing method for evaluating a change in an eye condition of the subject based on time-series data of the second image, comprising: the first imaging means images a subject at a frame rate equal to or higher than a frame rate of the second imaging means; The frame rate of the first imaging means is 480 [fps] or more and 1000 [fps] or less, An information processing method, wherein a frame rate of the second imaging means is 120 [fps] or more and 1000 [fps] or less.

10. a first acquisition process for acquiring image data relating to a first image obtained by capturing an image of the subject's head at a first angle of view from a first imaging means; a second acquisition process for acquiring image data relating to a second image obtained by capturing an eye region of the subject at a second angle of view narrower than the first angle of view from a second imaging means; a movement control process for moving a field of view of the second imaging means based on position information of the head of the subject acquired based on the first image; A condition evaluation process for evaluating a change in the condition of the subject's eye based on the time-series data of the second image. A program for causing a computer to execute the following: the first imaging means images a subject at a frame rate equal to or higher than a frame rate of the second imaging means; The frame rate of the first imaging means is 480 [fps] or more and 1000 [fps] or less, A program, wherein the frame rate of the second imaging means is 120 [fps] or more and 1000 [fps] or less.

Images