Information processing device, information processing method, and program

Abstract

This information processing device includes a head pose-rotation correlation unit, a head pose estimation unit, and an optical axis-visual axis correlation unit. The head pose-rotation correlation unit acquires a correlation between the eyeball rotation and the head pose as a head pose-rotation correlation. The head pose estimation unit acquires the head pose. The optical axis-visual axis correlation unit applies the head pose to the head pose-rotation correlation so as to estimate the eyeball rotation. The optical axis-visual axis correlation unit corrects the visual axis of the eyeball on the basis of the estimated eyeball rotation.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] In head-mounted displays (HMDs), gaze estimation technology is applied to reduce the rendering processing load (Foveated Rendering). However, further improvements in the accuracy of gaze estimation are required to realize gaze-based user interfaces (UIs), depth-of-field (DoF) representations, and display distortion correction due to eye movements (swim correction).

[0003] US Patent Application Publication No. 2016 / 0085299 JP 2022-095879 A

[0004] The optical axis and visual axis of the eye generally do not coincide, and the eyeball rotates when the head is tilted. When the eyeball rotates, the relationship between the optical axis and the visual axis changes accordingly. Patent Document 1 discloses a technology that corrects the positional relationship between the optical axis and the visual axis by estimating the direction of gravity using an acceleration sensor and estimating the rotation of the eyeball from the direction of gravity. However, since the rotation of the eyeball does not perfectly coincide with the direction of gravity, highly accurate correction is not possible. Patent Document 2 discloses a technology that estimates the rotation of the eyeball from the iris pattern. However, detecting the iris pattern places a heavy processing load, making it difficult to operate continuously in an HMD.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that are capable of estimating the gaze with high accuracy.

[0006] According to the present disclosure, there is provided an information processing device having a headpose-rotation correlation unit that acquires the correlation between eye rotation and head pose as a headpose-rotation correlation, a head pose estimation unit that acquires the head pose, and an optical axis-visual axis correlation unit that applies the head pose to the headpose-rotation correlation to estimate the eye rotation and corrects the visual axis of the eye based on the estimated eye rotation.The present disclosure also provides an information processing method in which information processing of the information processing device is executed by a computer, and a program that causes a computer to realize the information processing of the information processing device.

[0007] FIG. 1 is an explanatory diagram of gaze estimation taking eye rotation into consideration. FIG. 1 is an explanatory diagram of gaze estimation taking eye rotation into consideration. FIG. 2 is a diagram showing an example of the configuration of a display system. FIG. 3 is a diagram showing a conventional display system serving as a comparative example. FIG. 4 is a diagram showing an example of a calibration image for investigating the correlation between eye rotation and head pose. FIG. 5 is a diagram showing an example of a calibration image for investigating the correlation between the optical axis and the visual axis. FIG. 6 is a diagram showing an example of a processing flow for detecting head pose-rotation correlation by a calibration operation before content viewing. FIG. 7 is a diagram showing an example of a processing flow for gaze estimation when content is viewed. FIG. 8 is a diagram showing another example of the configuration of a display system. FIG. 9 is a diagram showing an example of the hardware configuration of an information processing device.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. Gaze estimation taking into account eye rotation] [2. System configuration example] [3. Display example of calibration image] [4. Calibration processing flow] [5. Gaze estimation processing flow] [6. Sequential update of head pose-rotation correlation] [7. Hardware configuration example] [8. Effects]

[0010] 1. Gaze Estimation Taking Eyeball Rotation into Account FIGS. 1 and 2 are explanatory diagrams of gaze estimation taking eyeball rotation into account.

[0011] Eye tracking is used in a variety of fields. Eye tracking is a technology that tracks what a user (US) is looking at in real time based on eye movement. The eye tracker detects the optical axis (LA) of the eyeball (EB) based on an image of the eye (eye image). The optical axis (LA) refers to the axis passing through the center of the cornea (CR) and the center of the pupil (PU), but the line of sight does not necessarily coincide with the optical axis (LA) of the eyeball (EB). The distribution of photoreceptors on the retina is not uniform, and the line of sight is the line connecting the area with a high density of photoreceptors (the depression in the center of the macula (MC): the macular fovea centralis)) and the nodal point (the central posterior surface of the lens).

[0012] The axis along which the line of sight passes is called the visual axis VA. The visual axis VA cannot be directly determined from the eye image. The visual axis VA is tilted by about 5° with respect to the optical axis LA. The offset between the visual axis VA and the optical axis LA varies from person to person, and this individual difference must be adjusted by calibration. The visual axis VA (line of sight) is estimated from the optical axis LA based on the calibration information.

[0013] When the head is tilted, the eyeball EB rotates. When the eyeball EB rotates, the three-dimensional positional relationship between the optical axis LA and the visual axis VA changes. To perform accurate gaze estimation, it is necessary to correct the visual axis VA taking into account the line of the eyeball EB. In this disclosure, the correlation between eyeball rotation and head pose is obtained in advance, and the visual axis VA is corrected based on the eyeball rotation estimated from the head pose. A display system for realizing this method will be specifically described below.

[0014] 2. System Configuration Example FIG. 3 is a diagram showing an example of the configuration of the display system 1. As shown in FIG.

[0015] The display system 1 includes an information processing device 10 and an HMD 20. The HMD 20 presents 3D images to a user US who is wearing the HMD. The information processing device 10 detects the user US's line of sight based on the user US's eye movements and performs display control according to the line of sight. Figure 3 selectively illustrates configurations related to line of sight estimation. Configurations typically used only for processing content (such as movies and games) are omitted from the illustration.

[0016] The HMD 20 has an IMU (Inertial Measurement Unit) 21, an external camera 22, a display 23, and an eye camera 24. The external camera 22 captures images of the surroundings of the HMD 20. Sensing information from the IMU 21 and the external camera 22 is used to estimate the head pose of the user US. The display 23 displays an image for calibration. The eye camera 24 captures an image of the eye of the user US and acquires the eye image.

[0017] The information processing device 10 includes a SLAM signal processing unit 11, a head pose estimation unit 12, a head pose-rotation correlation unit 13, a target display unit 14, an iris detection unit 15, a rotation detection unit 16, an optical axis estimation unit 17, an optical axis-visual axis correlation unit 18, and a visual axis estimation unit 19.

[0018] The SLAM signal processing unit 11 estimates the user's position based on sensing information from the IMU 21 and the external camera 22. The self-position estimation can be performed using SLAM (Simultaneous Localization and Mapping) technology. The head pose estimation unit 12 acquires the head pose of the user US based on the estimated self-position information.

[0019] The headpose-rotation correlation unit 13 acquires the correlation between eye rotation and head pose as the headpose-rotation correlation. In the example of FIG. 3 , the headpose-rotation correlation is determined by a prior calibration performed by the user US. The headpose-rotation correlation unit 13 acquires the eye rotation (direction and angle of rotation) corresponding to the head pose by applying the head pose estimated by the head pose estimation unit 12 to the head pose-rotation correlation.

[0020] For example, the headpose-rotation correlation unit 13 stores the correlation between eye rotation and head pose acquired for the user of the calibration information (the user US whose head pose is to be acquired when viewing content). However, the headpose-rotation correlation may be obtained by aggregating data on head pose and eye rotation collected from multiple people.

[0021] The iris detection unit 15 extracts the iris from the eye image acquired by the eye camera 24. The rotation detection unit 16 detects the rotation of the eyeball EB based on the iris image. Information on eyeball rotation is output to the head pose-rotation correlation unit 13. The head pose-rotation correlation unit 13 links the information on eyeball rotation acquired from the rotation detection unit 16 with the head pose information acquired from the head pose estimation unit 12. In this way, the head pose-rotation correlation is acquired.

[0022] The optical axis estimation unit 17 acquires the optical axis LA of the eyeball EB based on the eye image acquired from the eye camera 24. The optical axis-visual axis correlation unit 18 acquires the correlation between the optical axis LA of the eyeball EB and the visual axis VA as the optical axis-visual axis correlation. In the example of FIG. 3 , the optical axis-visual axis correlation is determined by a prior calibration operation performed by the user US. The optical axis-visual axis correlation unit 18 stores the correlation between the optical axis LA and the visual axis VA acquired for the user of the calibration information (the user US from whom the optical axis LA is acquired when viewing content). However, the optical axis-visual axis correlation may also be determined by aggregating data on the optical axis LA and the visual axis VA collected from multiple people.

[0023] The target display unit 14 generates an image (calibration image) to be used in the calibration work and displays it on the display 23. The calibration image includes a target TG (see FIGS. 5 and 6 ) to be gazed at. The target display unit 14 displays the target TG that the user US is paying attention to on the display 23. The target display unit 14 moves the position of the target TG to achieve natural line of sight and head movements required during calibration.

[0024] The target display unit 14 generates, as calibration images, an image for examining the correlation between eye rotation and head pose (see Figure 5), and an image for examining the correlation between the optical axis LA and the visual axis VA (see Figure 6).

[0025] The calibration image prompts the user US to follow the target TG with the eyes, thereby changing the eyeballs EB and head pose. For example, the rotation detection unit 16 detects the eyeball rotation of the user US accompanying the movement of the target TG. The head pose estimation unit 12 acquires the head pose of the user US, which changes in accordance with the movement of the target TG. The head pose-rotation correlation unit 13 acquires the correlation between the eyeball rotation accompanying the movement of the target TG and the head pose as the head pose-rotation correlation.

[0026] The correlation between the optical axis LA and the visual axis VA can be obtained based on the position of the target TG and information about the optical axis LA. For example, the optical axis-visual axis correlator 18 estimates the visual axis VA of the eyeball EB based on the position of the target TG obtained from the target display unit 14. The optical axis-visual axis correlator 18 associates the estimated visual axis VA with the optical axis LA obtained from the optical axis estimation unit 17, thereby obtaining the optical axis-visual axis correlation.

[0027] The optical axis-visual axis correlator 18 acquires the eye rotation corresponding to the headpose from the headpose-rotation correlator 13. The eye rotation is obtained by applying the headpose estimated by the headpose estimator 12 to the headpose-rotation correlation. The optical axis-visual axis correlator 18 estimates the visual axis VA by applying the optical axis LA acquired from the optical axis estimator 17 to the optical axis-visual axis correlation. The optical axis-visual axis correlator 18 corrects the visual axis VA of the eyeball EB based on the eye rotation acquired from the headpose-rotation correlator 13. The visual axis estimator 19 outputs the visual axis VA whose orientation has been corrected as the visual axis VA corresponding to the headpose.

[0028] FIG. 4 shows a conventional display system 1 as a comparative example. Ｃ 4 is a diagram showing the following: In FIG. 4, the configuration related to gaze estimation is selectively depicted, and the illustration of the configuration used only for processing normal content (movies, games, etc.) is omitted.

[0029] Display system 1 of the comparative example Ｃ In the target display unit 14, the line of sight is estimated based only on the correlation between the optical axis LA and the visual axis VA. Ｃdisplays only an image for examining the correlation between the optical axis LA and the visual axis VA. Since the correlation between head pose and eye rotation and the effect of eye rotation on the visual axis VA are not taken into consideration, proper gaze estimation is not performed when the head moves. In the method of the present disclosure shown in FIG. 3, the relationship between head pose and the visual axis VA is taken into consideration, so proper gaze estimation is performed even when the head moves.

[0030] 3. Display Examples of Calibration Images FIGS. 5 and 6 are diagrams showing display examples of calibration images.

[0031] Fig. 5 shows an example of a calibration image for investigating the correlation between eye rotation and head pose. In the example of Fig. 5, a spherical object floating in virtual space is presented as a target TG. The head pose estimation unit 12 tracks the head movement of the user US who is following the target TG. The target display unit 14 guides the head movement by varying the display position of the target TG.

[0032] For example, the target display unit 14 displays the target TG at a position that induces a change in head pose in order to realize a natural movement of the head. In the example of Fig. 5, the target display unit 14 displays an obstruction OB in front of the target TG, and encourages the user to move their head pose so as to peer into the target TG behind the obstruction OB. At this time, the target display unit 14 can also notify the user by voice or text of a message encouraging the user to change their head pose.

[0033] The calibration work is performed as a preliminary work before viewing regular content (movies, games, etc.). The subject of the calibration work is preferably the same user US as the viewer of the regular content (the user US from whom the head pose is acquired when viewing the regular content). This allows for appropriate correction of the visual axis VA taking into account individual differences between users US.

[0034] However, if there is little individual variation among users US, the visual axis VA can be corrected based on standard correlation data obtained from a large number of subjects. For example, the headpose-rotation correlation unit 13 can store the correlation between the average eye rotation and headpose based on the data of multiple subjects as the headpose-rotation correlation. In this case, a generally satisfactory gaze estimation can be performed without prior calibration.

[0035] FIG. 6 shows an example of a calibration image for examining the correlation between the optical axis LA and the visual axis VA. In the example of FIG. 6, one point is selected from multiple radially arranged points, and the selected point is presented as a target TG by, for example, lighting up. The calibration image is presented as an image (headlock image HI) that allows all points to be viewed with the head fixed. The iris detection unit 15 tracks the movement (eye rotation) of the eyeball EB of the user US as they follow the target TG. The target display unit 14 guides the movement of the eyeball EB by varying the display position of the target TG.

[0036] The iris detection unit 15 estimates eye rotation based on the iris pattern. Detecting the iris pattern requires high processing power, making it difficult to perform this process constantly when viewing normal content. However, if the iris pattern detection process is limited to the calibration process, the processing load is less likely to be a problem. When viewing normal content, the processing load is reduced by performing gaze estimation based on the process of estimating the optical axis LA and the process of converting the optical axis LA to the visual axis VA.

[0037] 4. Calibration Processing Flow FIG. 7 is a diagram showing an example of a processing flow for detecting head pose-rotation correlation by a calibration operation before content viewing.

[0038] The target display unit 14 displays a calibration image including the target TG on the display 23 (step S1). The SLAM signal processing unit 11 estimates the self-position of the user US during the calibration operation based on sensor information acquired from the IMU 21 and the external camera 22. The head pose estimation unit 12 acquires the head pose of the user US during the calibration operation based on the estimated self-position (step S2).

[0039] The eye camera 24 acquires an eye image of the user US following the target TG. The iris detection unit 15 detects the iris from the eye image. The rotation detection unit 16 detects eye rotation based on the tilt of the iris pattern (step S3). The head pose-rotation correlation unit 13 links the head pose acquired from the head pose estimation unit 12 with the eye rotation acquired from the rotation detection unit 16, and records the correspondence between the two as a head pose-rotation correlation (step S4). Note that steps S2 and S3 are performed in parallel, and either one may be performed first.

[0040] 5. Processing Flow of Gaze Estimation FIG. 8 is a diagram showing an example of a processing flow of gaze estimation during content viewing.

[0041] The SLAM signal processing unit 11 estimates the self-position of the user US while viewing content based on sensor information acquired from the IMU 21 and the external camera 22. The head pose estimation unit 12 acquires the head pose of the user US while viewing content based on the estimated self-position (step S11). The optical axis-visual axis correlation unit 18 applies the acquired head pose to the head pose-rotation correlation to estimate the eye rotation of the user US (step S12).

[0042] The eye camera 24 acquires an eye image of the user US while viewing content. The optical axis estimation unit 17 estimates the optical axis LA of the eyeball EB based on the eye image (step S13). The optical axis-visual axis correlation unit 18 applies the estimated optical axis LA to the optical axis-visual axis correlation to estimate the visual axis VA of the user US while viewing content (step S14). The optical axis-visual axis correlation unit 18 corrects the direction of the visual axis VA based on the eye rotation estimated from the head pose (step S15). Note that steps S11 to S12 and steps S13 to S14 are performed in parallel, and either one may be performed first.

[0043] 6. Sequential Update of Head Pose-Rotation Correlation Fig. 9 is a diagram showing another example of the configuration of a display system. The following description will focus on the differences from the display system 1 shown in Fig. 3.

[0044] In the display system of this example, the target display unit 14 shown in Fig. 3 is omitted. In this example, a dedicated image for the calibration work is not generated. In this example, an image of the content being viewed is extracted based on some kind of trigger, and the extracted image is used as the calibration image.

[0045] The trigger can be set arbitrarily by the system developer. For example, a trigger may be set when a scene including a clear target of gaze (attractive area), such as a scene of a shining object flying in the dark, is detected. The headpose-rotation correlation unit 13 sequentially updates the headpose-rotation correlation based on data of the eye rotation and headpose of the user US intermittently acquired based on a preset trigger.

[0046] When scene detection is used as a trigger as described above, the information processing device 30 may include a scene detection unit. The scene detection unit detects a video scene including an attention region as a unique scene. For example, an attention region is a display area of ​​an object that is likely to attract the user US's attention and that is likely to induce head movement of the user US by moving over a wide area of ​​the screen. In a scene in which a glowing object is flying in the dark, the glowing flying object is the attention region.

[0047] The rotation detection unit 16 detects eye rotation of the user US accompanying movement of the interest region when a peculiar scene is detected. The head pose estimation unit 12 acquires the head pose of the user US that changes with the movement of the interest region. The head pose-rotation correlation unit 13 updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying movement of the interest region and the head pose.

[0048] 7. Example of Hardware Configuration FIG. 10 is a diagram showing an example of the hardware configuration of the information processing device 10. As shown in FIG.

[0049] The information processing of the information processing device 10 is realized by, for example, a computer 1000. The computer 1000 has a central processing unit (CPU) 1100, a random access memory (RAM) 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0050] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0051] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0052] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0053] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0054] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0055] For example, when the computer 1000 functions as the information processing device 10 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0056] [8. Effects] The information processing device 10 has a head pose-rotation correlation unit 13, a head pose estimation unit 12, and an optical axis-visual axis correlation unit 18. The head pose-rotation correlation unit 13 obtains the correlation between eye rotation and head pose as a head pose-rotation correlation. The head pose estimation unit 12 obtains the head pose. The optical axis-visual axis correlation unit 18 corrects the visual axis VA of the eye EB based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation. In the information processing method disclosed herein, the processing of the information processing device 10 is executed by a computer 1000. A program disclosed herein causes the computer 1000 to realize the processing of the information processing device 10.

[0057] According to this configuration, the visual axis VA is corrected in consideration of the rotation of the eyeball EB, thereby enabling highly accurate line of sight estimation.

[0058] The head pose-rotation correlator 13 stores the correlation between eye rotation and head pose acquired for the user US whose head pose is to be acquired.

[0059] According to this configuration, the visual axis VA is appropriately corrected taking into account individual differences of the user US.

[0060] The information processing device 10 has a target display unit 14 and a rotation detection unit 16. The target display unit 14 displays a target TG that the user US is paying attention to on a display 23. The rotation detection unit 16 detects eye rotation of the user US accompanying movement of the target TG. The head pose estimation unit 12 acquires the head pose of the user US that changes accompanying movement of the target TG. The head pose-rotation correlation unit 13 acquires the correlation between the eye rotation accompanying movement of the target TG and the head pose as a head pose-rotation correlation.

[0061] According to this configuration, the user US is encouraged to change the head pose while following the target TG with his / her eyes. The head pose-rotation correlation is naturally acquired during the movement of following the target TG with his / her eyes.

[0062] The target display unit 14 displays a message prompting the user to change the head pose.

[0063] This configuration ensures that the head pose is varied.

[0064] The target display unit 14 displays a target TG at a position that induces a change in head pose.

[0065] This configuration ensures that the head pose is varied.

[0066] The target display unit 14 displays an obstruction OB in front of the target TG, thereby encouraging the head pose movement as if peering at the target TG behind the obstruction OB.

[0067] This configuration ensures that the head pose is varied.

[0068] The headpose-rotation correlation unit 13 sequentially updates the headpose-rotation correlation based on data on eye rotation and headpose of the user US that is intermittently acquired based on a preset trigger.

[0069] According to this configuration, an accurate head pose-rotation correlation that reflects the individual differences of the user US can be obtained.

[0070] The information processing device 10 has a scene detection unit. The scene detection unit detects a video scene including an interest region as a peculiar scene. The rotation detection unit 16 detects eye rotation of the user US accompanying movement of the interest region, triggered by the detection of the peculiar scene. The head pose estimation unit 12 acquires the head pose of the user US that changes with the movement of the interest region. The head pose-rotation correlation unit 13 updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying movement of the interest region and the head pose.

[0071] According to this configuration, the head pose-rotation correlation is acquired naturally during viewing of normal content (movies, games, etc.).

[0072] The head pose-rotation correlation unit 13 stores the correlation between the average eye rotation and head pose based on the data of a plurality of subjects as the head pose-rotation correlation.

[0073] According to this configuration, generally good gaze estimation can be performed while omitting prior calibration.

[0074] The information processing device 10 has an optical axis estimation unit 17. The optical axis estimation unit 17 acquires the optical axis LA of the eyeball EB. The optical axis-visual axis correlation unit 18 acquires the correlation between the optical axis LA and the visual axis VA as the optical axis-visual axis correlation. The optical axis-visual axis correlation unit 18 applies the optical axis LA to the optical axis-visual axis correlation to estimate the visual axis VA. The optical axis-visual axis correlation unit 18 corrects the estimated visual axis VA based on eye rotation.

[0075] According to this configuration, the visual axis VA can be estimated from the optical axis LA with high accuracy, which makes it possible to estimate the line of sight with high accuracy.

[0076] The optical axis-visual axis correlator 18 stores the correlation between the optical axis LA and the visual axis VA acquired for the user US for whom the optical axis LA is to be acquired.

[0077] This configuration enables accurate gaze estimation that reflects individual differences among users US.

[0078] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0079] [Additional Notes] The present technology can also be configured as follows: (1) An information processing device having: a headpose-rotation correlation unit that acquires a correlation between eye rotation and head pose as a headpose-rotation correlation; a headpose estimation unit that acquires the head pose; and an optical axis-visual axis correlation unit that corrects a visual axis of an eye based on the eye rotation obtained by applying the head pose to the headpose-rotation correlation. (2) The information processing device described in (1) above, wherein the headpose-rotation correlation unit stores the correlation between the eye rotation and the head pose acquired for a user for whom the head pose is to be acquired. (3) The information processing device according to (2) above, comprising: a target display unit that displays a target that the user is paying attention to on a display; and a rotation detection unit that detects the eye rotation of the user associated with movement of the target, wherein the head pose estimation unit acquires the head pose of the user that changes with movement of the target, and the head pose-rotation correlation unit acquires the correlation between the eye rotation associated with movement of the target and the head pose as the head pose-rotation correlation. (4) The information processing device according to (3) above, wherein the target display unit notifies a message that prompts the user to change the head pose. (5) The information processing device according to (3) or (4) above, wherein the target display unit displays the target at a position that causes the head pose to change. (6) The information processing device according to (5) above, wherein the target display unit displays an obstruction in front of the target and prompts the user to move the head pose as if peering at the target behind the obstruction. (7) The information processing device according to any one of (2) to (6), wherein the head pose-rotation correlation unit sequentially updates the head pose-rotation correlation based on data of the user's eye rotation and the head pose intermittently acquired based on a preset trigger.(8) The information processing device according to (7), comprising: a scene detection unit that detects a video scene including an attention region as a peculiar scene; and a rotation detection unit that detects the eye rotation of the user accompanying movement of the attention region, using the detection of the peculiar scene as the trigger, wherein the head pose estimation unit acquires the head pose of the user that varies accompanying the movement of the attention region, and the head pose-rotation correlation unit updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying the movement of the attention region and the head pose. (9) The information processing device according to (1), wherein the head pose-rotation correlation unit stores an average correlation between the eye rotation and the head pose based on data of a plurality of subjects as the head pose-rotation correlation. (10) The information processing device according to any one of (1) to (9), further comprising an optical axis estimation unit that acquires the optical axis of the eyeball, wherein the optical axis-visual axis correlation unit acquires the correlation between the optical axis and the visual axis as an optical axis-visual axis correlation, estimates the visual axis by applying the optical axis to the optical axis-visual axis correlation, and corrects the estimated visual axis based on the eyeball rotation. (11) The information processing device according to (10), further comprising: the optical axis-visual axis correlation unit stores the correlation between the optical axis and the visual axis acquired for a user from whom the optical axis is to be acquired. (12) An information processing method executed by a computer, comprising: acquiring a correlation between eyeball rotation and a headpose as a headpose-rotation correlation, acquiring the headpose, and correcting the visual axis of the eyeball based on the eyeball rotation obtained by applying the headpose to the headpose-rotation correlation. (13) The information processing method according to (12), further comprising storing the correlation between the eye rotation and the head pose acquired for the user from whom the head pose is to be acquired.(14) The information processing method according to (13) above, comprising: displaying a target that the user is paying attention to on a display; and detecting the eye rotation of the user accompanying movement of the target, wherein the head pose acquisition process acquires the head pose of the user that changes accompanying movement of the target, and the head pose-rotation correlation acquisition process acquires the correlation between the eye rotation accompanying movement of the target and the head pose as the head pose-rotation correlation. (15) The information processing method according to (14) above, comprising notifying a message that prompts the user to change the head pose. (16) The information processing method according to (14) or (15) above, wherein the target display process displays the target at a position that causes the head pose to change. (17) The information processing method according to (16) above, wherein the target display process displays an obstruction in front of the target and prompts the user to move the head pose as if peering at the target behind the obstruction. (18) The information processing method according to any one of (13) to (17), wherein the process of acquiring the headpose-rotation correlation successively updates the headpose-rotation correlation based on data of the user's eye rotation and the headpose acquired intermittently based on a preset trigger. (19) The information processing method according to (12), wherein the process of acquiring the headpose-rotation correlation includes storing, as the headpose-rotation correlation, an average correlation between the eye rotation and the headpose based on data of a plurality of subjects. (20) A program causing a computer to perform the following steps: acquiring a correlation between eye rotation and headpose as a headpose-rotation correlation; acquiring the headpose; and correcting the visual axis of the eye based on the eye rotation obtained by applying the headpose to the headpose-rotation correlation.

[0080] 10, 30 Information processing device 12 Head pose estimation unit 13 Head pose-rotation correlation unit 14 Target display unit 16 Rotation detection unit 17 Optical axis estimation unit 18 Optical axis-visual axis correlation unit 23 Display EB Eyeball LA Optical axis OB Obstruction TG Target US User VA Visual axis

Claims

1. An information processing device having: a headpose-rotation correlation unit that obtains the correlation between eye rotation and head pose as a headpose-rotation correlation; a headpose estimation unit that obtains the head pose; and an optical axis-visual axis correlation unit that corrects the visual axis of the eye based on the eye rotation obtained by applying the head pose to the headpose-rotation correlation.

2. The information processing device according to claim 1, wherein the head pose-rotation correlation unit stores the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired.

3. An information processing device as described in claim 2, comprising: a target display unit that displays a target that the user is focusing on on a display; and a rotation detection unit that detects the eye rotation of the user that accompanies movement of the target, wherein the head pose estimation unit acquires the head pose of the user that changes with movement of the target, and the head pose-rotation correlation unit acquires the correlation between the eye rotation that accompanies movement of the target and the head pose as the head pose-rotation correlation.

4. The information processing device according to claim 3, wherein the target display unit notifies a message prompting the user to change the head pose.

5. The information processing device according to claim 3, wherein the target display unit displays the target at a position that causes a change in the head pose.

6. The information processing device according to claim 5, wherein the target display unit displays an obstruction in front of the target and encourages the user to move the head pose in a way that peers at the target behind the obstruction.

7. The information processing device according to claim 2, wherein the head pose-rotation correlation unit sequentially updates the head pose-rotation correlation based on data of the user's eye rotation and head pose acquired intermittently based on a preset trigger.

8. An information processing device as described in claim 7, comprising: a scene detection unit that detects a video scene including an attention region as a peculiar scene; and a rotation detection unit that detects the eye rotation of the user accompanying movement of the attention region using the detection of the peculiar scene as the trigger, wherein the head pose estimation unit acquires the head pose of the user that changes as the attention region moves; and the head pose-rotation correlation unit updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying movement of the attention region and the head pose.

9. The information processing device according to claim 1, wherein the headpose-rotation correlation unit stores, as the headpose-rotation correlation, an average correlation between the eye rotation and the headpose based on data from a plurality of subjects.

10. An information processing device according to claim 1, further comprising an optical axis estimation unit that acquires the optical axis of the eyeball, wherein the optical axis-visual axis correlation unit acquires the correlation between the optical axis and the visual axis as an optical axis-visual axis correlation, estimates the visual axis by applying the optical axis to the optical axis-visual axis correlation, and corrects the estimated visual axis based on the eyeball rotation.

11. The information processing device according to claim 10, wherein the optical axis-visual axis correlation unit stores the correlation between the optical axis and the visual axis acquired for the user for whom the optical axis is to be acquired.

12. An information processing method executed by a computer, comprising: obtaining a correlation between eye rotation and head pose as a head pose-rotation correlation; obtaining the head pose; and correcting the visual axis of the eye based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation.

13. The information processing method according to claim 12, further comprising storing the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired.

14. An information processing method according to claim 13, comprising: displaying a target that the user is focusing on on a display; and detecting the eye rotation of the user accompanying movement of the target; wherein the head pose acquisition process acquires the head pose of the user that changes in accordance with movement of the target; and the head pose-rotation correlation acquisition process acquires the correlation between the eye rotation accompanying movement of the target and the head pose as the head pose-rotation correlation.

15. The information processing method according to claim 14, further comprising: notifying a message prompting the user to change the head pose.

16. The information processing method according to claim 14, wherein the target display process displays the target at a position that causes a change in the head pose.

17. The information processing method according to claim 16, wherein the target display process displays an obstruction in front of the target and prompts the user to move the head pose in a way that looks at the target behind the obstruction.

18. The information processing method according to claim 13, wherein the process of acquiring the head pose-rotation correlation successively updates the head pose-rotation correlation based on data of the user's eye rotation and head pose acquired intermittently based on a preset trigger.

19. The information processing method according to claim 12, wherein the process of acquiring the head pose-rotation correlation stores the correlation between the average eye rotation and the head pose based on data from multiple subjects as the head pose-rotation correlation.

20. A program that causes a computer to perform the following steps: obtain a correlation between eye rotation and head pose as a head pose-rotation correlation; obtain the head pose; and correct the visual axis of the eye based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation.

Images