Sight line detector, sight line detection method, and program

Abstract

To provide a sight line detector, a sight line detection method, and a program which can reduce a frequency of sight line calibration, in a human interface accompanied by sight line detection.SOLUTION: A sight line detector comprises: an input object position information acquisition unit which acquires input object position information indicating a position of an input object for performing operation input; an input time eyeball attitude information acquisition unit which acquires input time eyeball attitude information which is the information on an eyeball attitude at the time of the operation input; an input time sight line position information acquisition unit which acquires input time sight line position information indicating a position of the sight line with respect to the input object at the time of the operation input, on the basis of the input time eyeball attitude information; a correction value acquisition unit which acquires a correction value on the basis of the input time sight line position information and the input object position information; and a sight line detection position acquisition unit which acquires a sight line detection position on the basis of the correction value and the input time eyeball attitude information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention provides a gaze detection device, a gaze detection method, and a program that can reduce the frequency of gaze calibration in a human interface involving gaze detection. [Background technology]

[0002] Human interfaces that use gaze detection, such as HMDs (Head Mounted Displays) used during VR (Virtual Reality) experiences, can adjust the gaze detection position by performing gaze calibration before the experience. However, even if gaze calibration is performed as the initial setting, there may be cases where the HMD's position shifts during the experience, making it necessary to interrupt the experience and perform gaze calibration again.

[0003] Patent Document 1 discloses the following invention. The invention states that "the gaze detection adjustment unit 24-1a performs gaze detection calibration based on the gaze position P continuously sampled from the gaze detection unit 23-1a while the pointer 50 is stopped within the display area of ​​the start icon 40 and the display position of the start icon 40. Specifically, a correction parameter is obtained from the deviation between the detected gaze position P and the center position of the display area of ​​the start icon 40, and gaze detection errors due to individual differences are corrected," and that "when the user selects an icon 70, it is estimated that the user is looking at the icon 70 (an example of a gaze target), and therefore the information processing device 20-4 can perform gaze detection calibration based on the display position of the selected icon 70 and the detected gaze position P." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2016 / 139850 Summary of the Invention [Problem to be solved by the invention]

[0005] The disclosures of the above prior art documents are incorporated herein by reference. The following analysis has been carried out by the present inventors.

[0006] As described above, the invention disclosed in Patent Document 1 enables gaze calibration during operation based on the display position of an icon, which is the operation object, and the gaze position detected by the gaze detection unit. However, the embodiment of Patent Document 1 does not disclose specific processing by the gaze detection adjustment unit. For example, it does not disclose a method for calculating the adjustment amount of the gaze detection position (referred to as a correction amount in the present invention) or a method for adjusting gaze detection based on the adjustment amount.

[0007] Therefore, in one aspect of the present invention, it is an object to specifically disclose and provide a gaze detection device, a gaze detection method, and a program that can reduce the frequency of gaze calibration in a human interface that involves gaze detection. [Means for solving the problem]

[0008] According to a first aspect of the present invention, there is provided a gaze detection device having an input target position information acquisition unit that acquires input target position information, which is information indicating the position of an input target for performing operation input; an input time eyeball posture information acquisition unit that acquires input time eyeball posture information, which is information about the eyeball posture at the time of the operation input; an input time gaze position information acquisition unit that acquires input time gaze position information, which indicates the position of the gaze relative to the input target at the time of the operation input, based on the input time eyeball posture information; a correction value acquisition unit that acquires a correction value based on the input time gaze position information and the input target position information; and a gaze detection position acquisition unit that acquires a gaze detection position based on the correction value and the input time eyeball posture information.

[0009] According to a second aspect of the present invention, there is provided a gaze detection method for executing the following steps on a computer, the method including: a step of acquiring input target position information, which is information indicating the position of an input target for performing an operation input; a step of acquiring eyeball posture information at the time of input, which is information about the eyeball posture at the time of the operation input; a step of acquiring gaze position information at the time of input, which indicates the position of the gaze relative to the input target at the time of the operation input, based on the eyeball posture information at the time of input; a step of acquiring a correction value based on the gaze position information at the time of input and the input target position information; and a step of acquiring a gaze detection position based on the correction value and the eyeball posture information at the time of input.

[0010] According to a third aspect of the present invention, there is provided a program for causing a computer to execute the following processes: a process for acquiring input target position information, which is information indicating the position of an input target for performing an operation input; a process for acquiring eyeball posture information at the time of input, which is information about the eyeball posture at the time of the operation input; a process for acquiring gaze position information at the time of input, which indicates the position of the gaze relative to the input target at the time of the operation input, based on the eyeball posture information at the time of input; a process for acquiring a correction value based on the gaze position information at the time of input and the input target position information; and a process for acquiring a gaze detection position based on the correction value and the eyeball posture information at the time of input.

[0011] The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present invention can also be embodied as a computer program product. [Effects of the Invention]

[0012] According to each aspect of the present invention, it is an object to specifically disclose and provide a gaze detection device, a gaze detection method, and a program that can reduce the frequency of gaze calibration in a human interface that involves gaze detection. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of the configuration of a gaze detection device according to the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an overview of processing performed by a gaze detection device according to the present disclosure. [Figure 3] 10A and 10B are diagrams for explaining correction values ​​in the gaze detection device of the present disclosure. [Figure 4] 10 is a flowchart illustrating an example of the operation of the gaze detection device of the present disclosure. [Figure 5] 1 is a block diagram illustrating an example of a hardware configuration of a gaze detection device according to the present disclosure. [Figure 6] FIG. 1 is a diagram illustrating an overview of processing performed by the gaze detection device of the present disclosure. [Figure 7] 10 is a flowchart illustrating an example of an operation of the gaze detection device of the present disclosure. [Figure 8] 1 is a diagram illustrating an overview of the configuration of an automatic gaze correction system according to an embodiment of the present disclosure. [Figure 9] 1 is a diagram showing the configuration of an HMD-equipped device according to an embodiment of the present disclosure. [Figure 10] 1 is a diagram illustrating the configuration of a hand controller input device according to an embodiment of the present disclosure. [Figure 11] 1 is a flowchart illustrating the operation of an embodiment of the present disclosure. [Figure 12] FIG. 10 is a diagram for explaining eyeball posture in an embodiment of the present disclosure. [Figure 13] 1 is a flowchart illustrating the operation of an embodiment of the present disclosure. [Figure 14] 1 is a flowchart illustrating the operation of an embodiment of the present disclosure. [Figure 15] 10A and 10B are diagrams for explaining calculation of a correction amount in an embodiment of the present disclosure. [Figure 16] 10 is a flowchart illustrating the operation of another embodiment 1 of the present disclosure. [Figure 17]10 is a flowchart illustrating the operation of another embodiment 2 of the present disclosure. [Figure 18] FIG. 10 is a block diagram showing a configuration of another embodiment 3 of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] First, an overview of one embodiment will be described. Note that the reference numerals in the drawings attached to this overview are attached to each element as an example for convenience to facilitate understanding, and the description of this overview is not intended to be limiting in any way. In this disclosure, the drawings relate to one or more embodiments.

[0015] [composition] 1 is a block diagram showing an example of the configuration of an eye gaze detection device 10 according to the present disclosure. The eye gaze detection device 10 according to the present disclosure includes an input target position information acquisition unit 11, an input eye posture information acquisition unit 12, an input gaze position information acquisition unit 13, a correction value acquisition unit 14, and an eye gaze detection position acquisition unit 15.

[0016] The input target position information acquisition unit 11 acquires input target position information, which is information indicating the position of an input target for performing operation input. The eyeball posture information acquisition unit 12 acquires eyeball posture information at input, which is information on the eyeball posture at the time of the operation input. The gaze position information acquisition unit 13 acquires gaze position information at input, which indicates the position of the gaze relative to the input target at the time of the operation input, based on the eyeball posture information at input. The correction value acquisition unit 14 acquires a correction value based on the gaze position information at input and the input target position information. The gaze detection position acquisition unit 15 acquires the gaze detection position based on the correction value and the eyeball posture information at input.

[0017] One of the features of the gaze detection device 10 in one embodiment is that, at the time of operation input, the correction value acquisition unit 14 acquires a correction value based on gaze position information at the time of input and input target position information.

[0018] In this way, the gaze detection device 10 of the present disclosure can acquire a correction value for correcting gaze position information when an operation is input during a VR experience using an HMD, and acquire a gaze detection position, which is the gaze position after correction. This can significantly reduce the frequency of having to stop the experience and perform gaze calibration again due to misalignment of the HMD, etc., and can provide a higher quality VR experience. A more detailed configuration and other details are described below.

[0019] [First embodiment] [Processing Overview] FIG. 2 is a diagram illustrating an overview of processing by the gaze detection device 10 according to the first embodiment of the present disclosure. This diagram illustrates a screen 22 used by a user 21, viewed from behind the user 21 toward the screen 22. As shown in this diagram, the user 21 is operating a cursor on the screen 22 through gaze detection. A solid line segment 24 and a dotted line segment 25 extending from both eyes 23 of the user 21 represent the line of sight (visual axis) of the user 21. A confirmation dialog 26 is displayed on the screen 22, and the user 21 presses an OK button 27 in response. At this time, a correction value is calculated to correct the cursor position, as indicated by the dotted line 25. In this way, the gaze detection device 10 according to this embodiment executes processing to correct the cursor position, i.e., the gaze detection position, by utilizing the fact that the user may gaze at an object on the screen when performing an operation input on the object.

[0020] [composition] A gaze detection device 10 according to the first embodiment of the present disclosure is similar to the first embodiment and has the configuration shown in Fig. 1. That is, the gaze detection device 10 has an input target position information acquisition unit 11, an input-time eyeball posture information acquisition unit 12, an input-time gaze position information acquisition unit 13, a correction value acquisition unit 14, and a gaze detection position acquisition unit 15.

[0021] A feature of the gaze detection device 10 of this embodiment is that the input object position information acquisition unit 11 acquires the position information of the input object displayed as an image, such as the OK button 27 in Figure 2, and acquires a correction value for the gaze detection position based on the position information.

[0022] The input target position information acquisition unit 11 acquires input target position information, which indicates the position of an input target for performing an operation input. The "operation input" refers to a command or the like input using an input / output interface of the gaze detection device 10. The "input target" refers to an object that is the target of an operation input, such as a button in a dialog box displayed on an image projected by an HMD. The "information indicating a position" refers to a coordinate value, where the coordinate axes may have an X-axis and a Z-axis in the horizontal direction (the direction of gaze) with the head of the person performing the operation input as the origin, and a Y-axis in the vertical direction. The "acquisition" refers to the result of information processing performed by a CPU (Central Processing Unit) being stored in a memory such as RAM and made available for use (the same applies below). For example, this unit corresponds to the process in which the CPU installed in the gaze detection device 10 processes information for rendering an image displayed on the HMD and temporarily stores information indicating the position of the input target, such as a button, in a memory such as RAM (Random Access Memory).

[0023] The input target position information of an object displayed on the screen is a coordinate value indicating the position of the object, and it is desirable that the coordinate value indicates the center of the object, because when gazing at an object, humans tend to direct their gaze to the center of the object.

[0024] The screen on which the object to be input is displayed may be any screen whose position on the screen can be specified by coordinate values, such as a screen provided by a computer that displays an operating system that provides a GUI (Graphical User Interface) on a liquid crystal display, the above-mentioned HMD, a touch panel display, etc.

[0025] The eyeball posture information acquisition unit 12 acquires eyeball posture information (φ raw ,θ raw) is acquired. "Eye posture" is the direction in which the line of sight (visual axis) points, and is expressed as an elevation angle and an azimuth angle. Various methods can be used to acquire these angles, and there is no limit to one method. For example, a method of irradiating the eye with near-infrared light and acquiring the light reflected from the cornea (black part of the eye) and sclera (white part of the eye) using a light-receiving element or the like to acquire the direction of the eye, or a method of irradiating the cornea with a light source and acquiring the direction based on the position of the reflected image can be applied.

[0026] The gaze position information acquisition unit 13 at the time of input acquires gaze position information at the time of input (coordinate values) that indicates the gaze position relative to the input target at the time of operation input, based on the eye posture information at the time of input (angle). The "gaze position" refers to the position (gazing point) at which the user is gazing within the image, and is calculated as a point on the same coordinate system as the coordinates included in the input target position information. The gaze point is calculated using the eye posture information at the time of input described above. Basically, it can be calculated by finding the intersection of the gazes of both eyes, but if the eye posture information at the time of input is the angle of the optical axis, there will be a difference angle (κ angle) with the visual axis, so it is desirable to calculate the gaze point by performing calibration in advance or calculating an approximate value using a method of existing technology.

[0027] The gaze position information at the time of input may be acquired multiple times and a representative value of the multiple values ​​may be calculated. For example, the gaze position information acquisition unit at the time of input may be configured to acquire multiple pieces of gaze position information within a predetermined time before and after the input, and to use the center of gravity value of the multiple pieces of gaze position information as the gaze position information at the time of input.

[0028] Furthermore, among the gaze position information at the time of input acquired multiple times, gaze position information whose distance from the cursor position is smaller than a predetermined value may be removed, and a representative value may be calculated using only the gaze position information at the time of input after removal, and may be set as the gaze position information at the time of input. For example, the gaze position information acquisition unit at the time of input may further include a cursor position information acquisition unit (not shown) that acquires cursor position information that is information indicating the cursor position, and the gaze position information acquisition unit at the time of input may remove, from the plurality of pieces of gaze position information, gaze position information whose difference between each acquired piece of gaze position information and the cursor position information at the time the gaze position information was acquired is smaller than a predetermined value, and a center of gravity value of the remaining plurality of pieces of gaze position information after removal may be set as the gaze position information at the time of input.

[0029] As described above, the process of acquiring multiple pieces of gaze position information and calculating a representative value of these improves the reliability of the calculation of gaze position information at the time of input. Furthermore, by adopting gaze position information in which the distance between the gaze position and the cursor position is equal to or greater than a predetermined value, it is possible to efficiently calculate gaze position information at the time of input with higher validity.

[0030] The correction value acquisition unit 14 acquires a correction value based on the acquired information on the gaze position at the time of input and the acquired information on the position of the input target. Figures 3(a) and 3(b) are diagrams for explaining the correction value in the gaze detection device 10 of the present disclosure. H in Figure 3(a) φ and H in Fig. 3(b). θ is the correction value (angle) of the eye posture information at the time of input. On the other hand, the error (= correction value) between the gaze position information at the time of input and the input target position information is e (see Figure 2), and x head The axial error is e x , y head The axial error is e y Then, tanH φ =e x / L, tanH θ =e y / L, where L is the distance from the eyeball to the input target position information, and is calculated using the coordinate values ​​of the input target position information. φ , H θ is a small value, so tanH φ ≒H φand tanH θ ≒H θ It is possible to set H φ ≒e x / L, H θ ≒e y / L.

[0031] The gaze detection position acquisition unit 15 acquires the acquired correction value (H φ ,H θ ) and the acquired eye posture information at the time of input (φ raw ,θ raw ) and obtain the gaze detection position (φ, θ) based on φ = φ raw +H φ ,θ=θ raw +H θ Next, the gaze detection position is calculated using (φ, θ). The gaze detection position can be calculated by finding the intersection of the visual axes (gazing point), but as mentioned above, the optical axis measured by an eye camera or the like does not necessarily coincide with the visual axis, so it is desirable to calculate the gaze point after correcting the error between the optical axis and the visual axis by prior calibration or the like.

[0032] [Explanation of operation] 4 is a flowchart illustrating an example of the operation of the gaze detection device 10 of the present disclosure. As shown in this figure, when the gaze detection device 10 starts operation, it first acquires input target position information (step S41). Next, it acquires eyeball posture information at the time of input (step S42). Next, it acquires gaze position information at the time of input (step S43). Next, it acquires a correction value based on the gaze position information at the time of input and the input target position information (step S44). After that, it acquires a gaze detection position based on the correction value and the eyeball posture information at the time of input (step S45), and the process ends.

[0033] [Hardware configuration] Fig. 5 is a block diagram showing an example of a hardware configuration of the gaze detection device 10 according to the present disclosure. The gaze detection device 10 can be configured by an information processing device (computer) and has the configuration shown in Fig. 5. For example, the gaze detection device 10 includes a CPU (Central Processing Unit) 51, a memory 52, an input / output interface 53, and a NIC (Network Interface Card) 54 as a communication means, which are interconnected by an internal bus 55.

[0034] However, the configuration shown in Fig. 5 is not intended to limit the hardware configuration of the devices that make up the gaze detection device 10. Each gaze detection device 10 may include hardware that is not shown. Furthermore, the number of CPUs and the like included in the gaze detection device 10 is not intended to be limited to the example shown in Fig. 5, and for example, each device may include multiple CPUs.

[0035] The memory 52 is a RAM (Random Access Memory), a ROM (Read Only Memory), or an auxiliary storage device (such as a hard disk).

[0036] The input / output interface 53 is a means for interfacing with a display device or input device (not shown). The display device is, for example, a touch panel display, an HMD, etc. The input device includes, for example, a device that accepts user operations such as a keyboard or a mouse, as well as a camera or an image sensor that captures images of the eyeballs to obtain eyeball posture information.

[0037] The functions of the gaze detection device 10 are realized by processing modules, namely, an input target position information acquisition program, an input eyeball posture information acquisition program, an input gaze position information acquisition program, a correction value acquisition program, and a gaze detection position acquisition program.

[0038] The processing modules are implemented, for example, by the CPU 51 executing a program stored in the memory 52. ​​The program can be updated by downloading it via a network or by using a storage medium storing the program. The processing modules may also be implemented by semiconductor chips. That is, it is sufficient if there is some means for executing the functions performed by the processing modules using some kind of hardware and / or software.

[0039] [Hardware operation] When the gaze detection device 10 starts operating, the input object position information acquisition program is first called from the memory 52 to the CPU 51 and placed into an execution state. This program acquires coordinate values, which are position information on the screen of the input object, such as a user interface button, from the currently active program and temporarily stores them in the memory 52.

[0040] Next, the program for acquiring eyeball posture information at the time of input is called from the memory 52 to the CPU 51 and placed into an execution state. This program acquires an image of the eyeball using a camera or the like connected to the input / output interface 53, and calculates the eyeball posture based on this image. Note that various methods can be applied to acquire eyeball posture information, and the method is not limited to one. The calculated eyeball posture information is temporarily stored in the memory 52 as eyeball posture information at the time of input.

[0041] Next, the gaze position information acquisition program at the time of input is called from the memory 52 to the CPU 51 and put into an execution state. This program calls the eyeball posture information at the time of input temporarily stored in the memory 52, and calculates the gaze position information at the time of input as coordinate values ​​on the screen based on the eyeball posture information (as described above).

[0042] Next, the correction value acquisition program is called from memory 52 and is put into execution state by CPU 51. This program acquires the difference (distance) between the input target position information and the gaze position information at the time of input, which are temporarily stored in memory 52. ​​Then, the angle formed by the line connecting the eyeball and the input target position and the line connecting the eyeball and the gaze position at the time of input is calculated and used as the correction value (angle).

[0043] Next, the gaze detection position acquisition program is called from memory 52 and executed by CPU 51. This program calls the eyeball posture information at the time of input stored in memory 52, and calculates the corrected eyeball posture (angle) at the time of input by adding the calculated correction value. The coordinates of the intersection of the line from the eyeball to the display screen, which is the angle forming this corrected eyeball posture at the time of input, with the screen are calculated and set as the gaze detection position (coordinate value).

[0044] [Effect description] As described above, the gaze detection device 10 of this embodiment is capable of acquiring the gaze detection position during operation input, and therefore can provide a gaze detection device, gaze detection method, and program that can reduce the number of calibrations.

[0045] [Second embodiment] [Processing Overview] In the second embodiment, an example of the gaze detection device 10 in which the input target used for calculating the correction value is a tangible object will be described. Here, a hand controller used as an input device in a VR device or the like will be described as an example.

[0046] FIG. 6 is a diagram illustrating an overview of the processing of the gaze detection device 10 according to the second embodiment of the present disclosure. This diagram illustrates a hand controller, which is a device for inputting operations to the gaze detection device 10 of this embodiment. As shown in the diagram, a user can hold the hand controller and operate buttons to operate an input target on the screen. As shown in the diagram, the hand controller has a decision button for selecting and confirming an object. In addition, the hand controller may have a cross button for controlling the cursor position.

[0047] The hand controller has a motion sensor and measures the coordinates (x controller ,y controller ,z controller ) on the display. Suppose a user holds the hand controller and places a finger on the confirmation button to input an operation. At this time, the hand controller detects the placement of the finger using a touch sensor or the like, and identifies the position of the placed button on the above-mentioned coordinate system, thereby obtaining input target position information. At the same time, eye posture information is acquired using an eye camera or the like, and gaze position information on the above-mentioned coordinate system is obtained based on the eye posture information. Next, the gaze detection device 10 calculates a correction value for gaze detection from the acquired gaze detection position information and input target position information.

[0048] The correction value is then applied to the acquired eyeball posture information, and a position on the screen based on the corrected eyeball posture information is determined as the gaze detection position.

[0049] In this way, the gaze detection device 10 of this embodiment is able to calculate the correction value for gaze detection using a tangible object that is the target of gaze, such as a physical button on the hand controller, rather than on the screen, and detect the gaze.

[0050] [composition] An example of the configuration of a gaze detection device 10 according to a second embodiment of the present disclosure is based on the above embodiment and has the configuration shown in Fig. 1. That is, the gaze detection device 10 has an input target position information acquisition unit 11, an eyeball posture information acquisition unit at input 12, an gaze position information acquisition unit at input 13, a correction value acquisition unit 14, and a gaze detection position acquisition unit 15. These components have already been described in the above embodiment, so only new features will be described below.

[0051] The gaze detection device 10 of this embodiment is characterized in that, first, the input target for acquiring input target position information is not an object displayed as an image on the screen, but a tangible object such as a physical button. Second, the input target position information acquisition unit 11 acquires position information of the input target for performing an operation input by distinguishing between no input, an operation input in which the input target is pressed with a finger, and an operation input in which the finger is placed on the input target, and the input-time eyeball posture information acquisition unit 12 acquires input-time eyeball posture information when the operation input is an operation input in which the finger is placed on the input target.

[0052] [Explanation of operation] 7 is a flowchart illustrating an example of the operation of the gaze detection device 10 of the present disclosure. As shown in this figure, first, it is determined whether a finger is placed on a button (step S71). When a finger is placed on the button (step S71, Y), position information of the button is acquired (step S72). Next, eyeball posture information at the time of finger placement is acquired (step S73). Gaze position information at the time of finger placement is acquired based on the acquired eyeball posture information (step S74). After that, a correction value is acquired based on the gaze position information at the time of finger placement and the position information of the button (step S75), and a gaze detection position is acquired based on the correction value and the eyeball posture information at the time of finger placement (step S76).

[0053] [Effect description] As described above, the gaze detection device 10 of this embodiment utilizes the fact that the gaze is focused on a button when a finger is placed on the button during operation input, and can acquire a correction value for a tangible object such as a physical button and acquire the gaze detection position based on the correction value. This reduces the need for calibration during operation input.

[0054] [Example] [Processing Overview] In the following examples, we will explain the processes of an automatic gaze correction system that is based on the gaze detection device 10 of the above embodiment and is composed of an HMD-equipped device and a hand controller input device. In this example, we will explain automatic gaze correction when clicking a button on a screen UI displayed in a virtual space, which is an operation that focuses the gaze on one point.

[0055] 8, the automatic gaze correction system is composed of an HMD-equipped device 100 and a hand controller input device 200. The HMD-equipped device 100 and the hand controller input device 200 are connected so as to be able to communicate with each other via radio waves.

[0056] As shown in FIG. 9, the HMD-equipped device 100 includes a display device 101, a position detection device 102, a posture detection device 103, a gaze detection device 104, a calculation device 105, a control device 106, and a communication device 107.

[0057] The display device 101 displays a screen output from the control device 106. The position detection device 102 detects the amount of change in position of the HMD-equipped device 100 at a specified frame period, and outputs the amount to the calculation device 105.

[0058] The posture detection device 103 detects the amount of posture change of the HMD-equipped device 100 at a specified frame period and outputs the result to the calculation device 105. The gaze detection device 104 calculates the eye postures of the left and right eyes from a photograph of the operator's eyes taken at a specified frame period and outputs the result to the calculation device 105.

[0059] The arithmetic device 105 updates the position and orientation of the HMD-equipped device 100 in the virtual space and the position and orientation of the hand controller input device 200 using the position change amount and orientation change amount output from the position detection device 102 and the orientation detection device 103, and the position change amount and orientation change amount of the hand controller input device 200 output from the communication device 107, performs rendering of the screen to be displayed on the display device 101, and outputs it to the control device 106. The arithmetic device 105 also has a correction amount calculation unit 151, calculates the correction amount of the eyeball orientation on two axes (left and right), corrects the eyeball orientation on two axes (left and right) output from the gaze detection device 104, and calculates the gaze detection position from the corrected eyeball orientation.

[0060] The control device 106 outputs the screen output from the arithmetic device 105 to the display device 101. The communication device 107 receives the position change amount, posture change amount, and button state (whether a finger is placed on a button or not, and whether a button is pressed or not) of the hand controller input device 200 from the communication device 205 of the hand controller input device 200, and outputs them to the arithmetic device 105.

[0061] As shown in Fig. 10, hand controller input device 200 has a button detection device 201, a position detection device 202, an attitude detection device 203, a control device 204, and a communication device 205. As shown in Fig. 6, hand controller input device 200 is shaped to be held in one hand with the index finger, middle finger, ring finger, and little finger, and the buttons are pressed with the thumb. The buttons on hand controller input device 200 include a cross button and an enter button.

[0062] The button detection device 201 outputs the button states on the hand controller input device 200 to the control device 204. The position detection device 202 detects the amount of change in position of the hand controller input device 200 at a specified frame period and outputs it to the control device 204. The orientation detection device 203 detects the amount of change in orientation of the hand controller input device 200 at a specified frame period and outputs it to the control device 204. The control device 204 outputs the button states output from the button detection device 201 and the amount of change in position and orientation output from the position detection device 202 and orientation detection device 203 to the communication device 205. The communication device 205 transmits the button states output from the button detection device 201 and the amount of change in position and orientation output from the position detection device 202 and orientation detection device 203 to the communication device 107 of the HMD-equipped device 100.

[0063] [Explanation of the operation of the embodiment] First, the operation of acquiring the gaze detection position will be described. As shown in Fig. 11, the HMD-equipped device 100 acquires the left and right eyeball postures using the gaze detection device 104 (step S101, YES). As shown in Fig. 12(a) and Fig. 12(b), each eyeball posture is defined using φ and θ. When the left and right eyeball postures have been acquired, the gaze detection device 104 outputs φ and θ to the calculation device 105. The calculation device 105 calculates the eyeball posture φ acquired in S101. raw and θ raw , which represents the correction amount of eye posture, H φ and H θ Using the above formula, the postures of the left and right eyes are corrected as shown in formula 1 (step S102).

[0064]

number

[0065] After correcting the eyeball posture, if the left and right lines of sight intersect (step S103, YES), the line of sight detection position, which is the intersection of the left and right lines of sight, is obtained (step S104).

[0066] Next, the operation of calculating the correction amount for eyeball posture will be described. As shown in FIG. 13, if there is a screen UI in the forward direction (shown in FIG. 2) of the hand controller input device 200 (step S201, YES), a cursor is displayed at the intersection of the forward direction of the hand controller input device 200 and the screen UI (step S202). If there is a button on the screen UI that the user wants to click, the cursor is moved so that it is displayed on the button. When the enter button on the hand controller input device 200 is pressed while the cursor is on the button, a button click on the screen UI is detected (step S203, YES). At this time, the arithmetic unit 105 calculates the correction amount H by the correction amount calculation unit 151 based on the coordinates of the center of the clicked button on the screen UI and the gaze detection position. φ and H θ (step S204). Finally, the current correction amount is updated with the calculated correction amount (step S205).

[0067] Next, a detailed description will be given of the processing of the correction amount calculation unit 151. As shown in Fig. 14, if the time when the button is clicked is t, the gaze detection positions acquired from time t-Δ to time t+Δ are TIFF2025161039000003.tif6150 is referenced (step S301). However, Δ is specified by the designer. At this time, in addition to the gaze detection position, the cursor position TIFF2025161039000004.tif6150 is also referenced. Of the referenced gaze detection positions, those that satisfy the following conditional expression are excluded (step S302).

[0068]

number

[0069] Here, a is a constant parameter that is specified by the designer. Using TIFF2025161039000006.tif6150, the center of gravity S of the gaze detection position is calculated using the following formula (step S303).

[0070]

number

[0071] The calculated center of gravity S and the center position P of the button on the screen UI button is converted into a position vector in the head coordinate system (see Fig. 12(a) and Fig. 12(b)), and x head and y head S and P in the axial direction button The error in e x and e y Next, the correction amount H for each of the left and right eyes is calculated using the following formula (step S304). φ and H θ is calculated (step S305).

[0072]

number

[0073] However, as shown in Figure 15(a) and Figure 15(b), L is the center of gravity of the head coordinate system. Calculate using the following formula using the components of each axis of TIFF2025161039000009.tif6150.

[0074]

number

[0075] [Effect description] According to the present invention, in a VR experience using gaze detection, the gaze detection position acquired during an operation in which the operator's gaze is focused on one point and the correction amount for the eyeball posture are calculated from the position of the focused point, and the correction amount is updated as needed, thereby preventing deviation of the gaze detection position due to deviation of the wearing position of the HMD-equipped device during the VR experience. Conventionally, when the deviation of the gaze detection position becomes large, the operator has to perform gaze calibration, but because it is corrected automatically, the operator can concentrate more on the VR experience.

[0076] [Another embodiment of the invention] [Another Embodiment 1] In another embodiment shown in FIG. 16, when checking the button position on the hand controller input device 200 and placing a finger on it, the operator utilizes visually recognizing the button position to calculate the correction amount of the eyeball posture.

[0077] As shown in FIG. 16, first, it is detected whether a finger is placed on a button of the hand controller input device 200 (step S401). At this time, by referring to the button input force amount Δu detected by the button detection device 201, it is possible to detect whether a finger is placed on the button as follows. Here, a and b (0 < a << b) represent constant parameters and are specified by the designer.

[0078] [Number]

[0079] When it is detected that a finger has been placed (step S401, YES), the button position on the screen UI in step S204 is replaced with the button position on the hand controller input device 200 where the finger is placed, and the correction amount of the eyeball posture is calculated in the same manner as in step S204 (step S402). Finally, the current correction amount is updated with the calculated correction amount (step S403).

[0080] In this way, in another embodiment, by utilizing the operation of placing a finger on the button of the hand controller input device 200, a stable effect can be obtained without depending on the specifications of the screen UI for each application.

[0081] [Another Embodiment 2] In another embodiment shown in FIG. 17, based on the width and height of the button on the screen UI, the correction amount of the eyeball posture is calculated.

[0082] As shown in FIG. 17, after calculating the error between the center of the button on the screen UI and the center of gravity of the line-of-sight detection position (step S504), the button width Q on the screen UI w and the button height Q h are obtained (S505), Qw and Q h are the reference values ​​of the button width, Q w, ref and the reference value Q of the button height h, ref If it is greater than Q (step S506, YES), the correction amount is calculated as follows using the expressions (8) and (9) of Equation 7 (step S507): w and Q h are the reference values ​​of the button width, Q w, ref and the reference value Q of the button height h, ref In the following cases (step S506, NO), the correction amount is calculated using the formulas (5) and (6) of Equation 4 as follows (step S508): w, ref and Q h, ref is specified by the designer.

[0083]

number

[0084] In this way, in another embodiment, when the button width and button height on the screen UI exceed the reference value, the correction amount is reduced, thereby preventing an incorrect correction amount from being set when dealing with large buttons where the user is unlikely to focus on the center of the button.

[0085] [Another Example 3] In another embodiment shown in FIG. 18, instead of an HMD-equipped device, a touch panel display 300 equipped with a gaze detection camera is used to correct the eye posture of the operator while operating the display.

[0086] The touch panel display 300 equipped with a gaze detection camera includes a display device 301 , a gaze detection device 302 , a screen input detection device 303 , a control device 304 , and a calculation device 305 . The display device 301 displays a screen output from the control device 304 .

[0087] The gaze detection device 302 calculates the eye postures of the left and right eyes from photographs of the operator's eyes taken at a specified frame cycle, and outputs the calculated results to the calculation device 305. The screen input detection device 303 detects the operator's screen input, and outputs the calculated results to the control device 304. The calculation device 305 has a correction amount calculation unit 351, calculates the amount of correction for the eye posture on two axes (left and right), corrects the eye posture on two axes (left and right) output from the gaze detection device 302, and calculates the gaze detection position from the corrected eye posture. In this way, in another embodiment, a touch panel display equipped with a gaze detection camera is used instead of an HMD-equipped device.

[0088] A part or all of the above disclosure may also be described as follows, but is not limited to the following. [Appendix 1] This is the same as the gaze detection device according to the first aspect described above. [Appendix 2] Preferably, the gaze detection device according to claim 1, wherein the input object position information acquisition unit further acquires position information of the input object displayed as an image. [Appendix 3] The input target is displayed on a touch panel display, preferably the gaze detection device of appendix 1 or 2. [Appendix 4] Preferably, the gaze detection device according to claim 1, wherein the input target is configured by a tangible object. [Appendix 5] The input target position information acquisition unit acquires position information of the input target for performing the operation input by distinguishing between no input, an operation input in which the input target is pressed with a finger, and an operation input in which the finger is placed on the input target, and when an operation input in which the finger is placed on the input target is performed as the operation input, the eyeball posture information acquisition unit at the time of input acquires the eyeball posture information at the time of input, preferably the gaze detection device of Appendix 4. [Appendix 6] Preferably, the gaze detection device according to claim 1, wherein the correction value acquisition unit further acquires a correction value based on a width and a height of the input object. [Appendix 7] Preferably, the gaze position information acquisition unit at the time of input acquires a plurality of pieces of gaze position information within a predetermined time before and after the time of input, and sets a center of gravity value of the plurality of pieces of gaze position information as the gaze position information at the time of input. [Appendix 8] Preferably, the gaze detection device according to Appendix 7 further includes a cursor position information acquisition unit that acquires cursor position information that is information indicating a cursor position, and the gaze position information acquisition unit at the time of input removes, from the plurality of gaze position information, gaze position information for which the difference between each acquired piece of gaze position information and the cursor position information at the time of acquisition of that gaze position information is smaller than a predetermined value, and sets the center of gravity value of the remaining plurality of gaze position information thus removed as the gaze position information at the time of input. [Appendix 9] This is the same as the gaze detection method relating to the second viewpoint described above. [Appendix 10] This is the same as the program related to the third perspective mentioned above. Note that Supplements 9 and 10 can be expanded into Supplements 2 to 8 in the same manner as Supplement 1.

[0089] The disclosures of the above-cited patent documents and other documents are incorporated herein by reference. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the scope of the claims), and further based on the basic technical concept thereof. Furthermore, various combinations and selections of the various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concept, including the scope of the claims. In particular, with regard to the numerical ranges set forth herein, any numerical value or subrange included within the range should be construed as being specifically set forth, even if not otherwise specified. [Explanation of symbols]

[0090] 10: Gaze detection device 11: Input target position information acquisition unit 12: Eye posture information acquisition unit at the time of input 13: Gaze position information acquisition unit during input 14: Correction value acquisition unit 15: Gaze detection position acquisition unit 21: User 22: Screen 23: Both eyes 24: Solid line segment 25:Dotted line 26:Confirmation dialog 27: OK button 28: Cursor 40: Start icon 50: Pointer 51:CPU 52: Memory 53: Input / output interface 54:NIC 55: Internal bus 70: Icon 100: HMD-equipped device 101:Display device 102: Position detection device 103: Attitude detection device 104: Line of sight detection device 105: Arithmetic device 106: Control device 107:Communication equipment 151: Correction amount calculation section 200: Hand controller input device 201: Button detection device 202: Position detection device 203: Attitude detection device 204: Control device 205: Communication equipment 300:Touch panel display 301:Display device 302: Line of sight detection device 303: Screen input detection device 304: Control device 305: Arithmetic device 351: Correction amount calculation unit

Claims

1. an input target position information acquisition unit that acquires input target position information that is information indicating the position of an input target for performing an operation input; an eyeball posture information acquisition unit at the time of input that acquires eyeball posture information at the time of input, which is information about the eyeball posture at the time of the operation input; an input-time gaze position information acquisition unit that acquires input-time gaze position information indicating a gaze position with respect to the input target at the time of the operation input based on the input-time eyeball posture information; a correction value acquisition unit that acquires a correction value based on the gaze position information at the time of input and the input target position information; a gaze detection position acquisition unit that acquires a gaze detection position based on the correction value and the eyeball posture information at the time of input; A gaze detection device having:

2. the input target position information acquisition unit further acquires position information of the input target displayed as an image. The gaze detection device according to claim 1.

3. 3. The gaze detection device according to claim 2, wherein the input object is displayed on a touch panel display.

4. The input object is composed of a tangible object. The gaze detection device according to claim 1.

5. the input target position information acquisition unit acquires position information of the input target for performing an operation input by distinguishing between no input, an operation input in which the input target is pressed with a finger, and an operation input in which the finger is placed on the input target; When the operation input is performed by placing a finger on the input target, the eyeball posture information acquisition unit at the time of input acquires eyeball posture information at the time of input; 5. The gaze detection device according to claim 4.

6. the correction value acquisition unit further acquires a correction value based on the width and height of the input object. The gaze detection device according to claim 1.

7. the input time gaze position information acquisition unit acquires a plurality of gaze position information within a predetermined time before and after the input time, and defines a center of gravity value of the plurality of gaze position information as the input time gaze position information.

7. The gaze detection device according to claim 1.

8. a cursor position information acquisition unit that acquires cursor position information that is information indicating a cursor position; and the gaze position information acquisition unit at the time of input removes gaze position information, from the plurality of gaze position information, for which a difference between each of the acquired gaze position information and cursor position information at the time of acquisition of the gaze position information is smaller than a predetermined value, and sets a center of gravity value of the remaining plurality of gaze position information after removal as the gaze position information at the time of input; The gaze detection device according to claim 7.

9. A method for executing the following steps on a computer: acquiring input target position information which is information indicating the position of an input target for performing an operation input; acquiring eyeball posture information at the time of input, which is information about the eyeball posture at the time of the operation input; acquiring gaze position information at the time of input, which indicates a position of a gaze with respect to the input target at the time of the operation input, based on the eyeball posture information at the time of input; acquiring a correction value based on the gaze position information at the time of input and the input target position information; acquiring a gaze detection position based on the correction value and the eyeball posture information at the time of input; A gaze detection method including:

10. A process of acquiring input target position information which is information indicating the position of an input target for performing an operation input; A process of acquiring eyeball posture information at the time of input, which is information on the eyeball posture at the time of the operation input; a process of acquiring gaze position information at the time of input, which indicates a position of a gaze with respect to the input target at the time of the operation input, based on the eyeball posture information at the time of input; A process of acquiring a correction value based on the gaze position information at the time of input and the input target position information; a process of acquiring a gaze detection position based on the correction value and the eyeball posture information at the time of input; A program that causes a computer to execute the following.

Images