Eye gaze detection device

The line-of-sight detection device enhances user experience by detecting both eyes' gaze points and adjusting processing based on non-dominant eye variation, addressing the issue of unintended processing in conventional technologies.

JP7581067B2Active Publication Date: 2024-11-12CANON KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2021013827
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-01-29
Publication Date
2024-11-12
Estimated Expiration
2041-01-29

AI Technical Summary

Technical Problem

Conventional line-of-sight detection technologies perform processing based on gaze points without considering user intention, leading to undesirable outcomes for users.

Method used

A line-of-sight detection device that detects the gaze points of both eyes, differentiates the dominant and non-dominant eye, and adjusts processing based on the variation in the non-dominant eye's line of sight to enhance user experience.

Benefits of technology

Enables processing that suits user intent, improving immersion and convenience by adjusting depth of field and movement resolution based on concentration levels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581067000001
    Figure 0007581067000001
  • Figure 0007581067000002
    Figure 0007581067000002
  • Figure 0007581067000003
    Figure 0007581067000003
Patent Text Reader

Abstract

To provide a technique which enables appropriate processing based on a point of gaze.SOLUTION: A line-of-sight detection device of the present invention comprises: line-of-sight detection means configured to detect respective lines of sight of the left and right eyes of a user; point-of-gaze detection means configured to detect a point of gaze based on the line of sight of at least dominant one of the left and right eyes; and control means configured to control point of gaze-based processing according to the amount of change in line of sight of at least non-dominant one of the left and right eyes.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a gaze detection device. [Background technology]

[0002] Various techniques have been proposed to detect (estimate) the line of sight (line of sight direction; line of sight position) and perform processing based on the gaze point (gaze position). For example, Patent Document 1 proposes a technique to identify a subject (recognize the subject intended by the photographer) based on information on the gaze position of the photographer looking through a viewfinder, and to control the focus on the subject.

[0003] Technologies for performing processing based on the gaze point are used not only for camera control, but also for xR (virtual reality (VR), augmented reality (AR), mixed reality (MR), substitute reality (SR), etc.). For example, Patent Document 2 proposes a technology for a head mounted display (HMD) that determines where the user is gazing at in an image, and makes the image clearer near the gaze point and blurs the image in areas away from the gaze point. This processing is a process for virtually controlling (changing) the depth of field based on the gaze point, and this processing can enhance the user's sense of immersion and provide a more comfortable VR experience.

[0004] As gaze detection in HMDs and the like, gaze detection using both eyes has been proposed. For example, a method of detecting the gaze of each of the right and left eyes, a method of distinguishing between the dominant eye and the non-dominant eye and detecting the gaze point with emphasis on the dominant eye, etc. have been proposed. Patent Document 3 proposes a technology that determines the dominant eye of a user and adjusts the position of an image based on information about the dominant eye. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2004-8323 A [Patent Document 2] Special Publication No. 2020-504959 [Patent Document 3] JP 2007-34628 A Summary of the Invention [Problem to be solved by the invention]

[0006] A method has been proposed for determining the gaze point using gaze information from the dominant eye, but in the conventional technology, the gaze point-based processing is not dependent on the user's intention (such as the way of looking at the gaze point), and the processing may not be desirable for the user. For example, the technology disclosed in Patent Document 2 (control of the depth of field based on the gaze point) performs control that is desirable for users who want to achieve a high sense of immersion, but this control is not desirable for users who want to capture the entire image without immersion.

[0007] An object of the present invention is to provide a technique that can suitably perform processing based on a gaze point. [Means for solving the problem]

[0008] A first aspect of the present invention includes a gaze detection means for detecting the gaze of each of a user's right eye and left eye, a gaze point detection means for detecting a gaze point based on the gaze of a dominant eye of the right eye or the left eye, and a control means for controlling a process to be executed based on the gaze point to be different depending on an amount of change in the gaze of the non-dominant eye of the right eye or the left eye. a display means for displaying an image to be shown to the user; Have The control means is configured to make the depth of field of the image based on the gaze point shallower as the amount of variation is smaller. The present invention relates to a gaze detection device. A second aspect of the present invention is a gaze detection device comprising: gaze detection means for detecting the gaze of a user's right eye and left eye, respectively; gaze point detection means for detecting a gaze point based on the gaze of a dominant eye of the right eye or the left eye; and control means for controlling the processing executed based on the gaze point to be different depending on the amount of fluctuation in the gaze of the non-dominant eye of the right eye or the left eye, wherein the control means reduces the effect of changes in the user's gaze on the detected position of the gaze point as the amount of fluctuation becomes smaller. A third aspect of the present invention is a gaze detection device comprising: gaze detection means for detecting the gaze of a user's right eye and left eye, respectively; gaze point detection means for detecting a gaze point based on the gaze of a dominant eye of the right eye or the left eye; control means for controlling the processing executed based on the gaze point to be different depending on the amount of fluctuation in the gaze of the non-dominant eye of the right eye or the left eye; and display means for displaying items that can be moved in response to instructions from the user, wherein the control means is configured to increase the movement resolution of the items in the vicinity of the gaze point the smaller the amount of fluctuation. A fourth aspect of the present invention is a gaze detection means for detecting the gazes of the right and left eyes of a user, The gaze detection device comprises a gaze point detection means for detecting a gaze point based on the line of sight of the dominant eye between the right eye and the left eye, a control means for controlling the processing executed based on the gaze point to be different depending on the amount of fluctuation in the line of sight of the non-dominant eye between the right eye and the left eye, and a display means for displaying items that can be moved in response to instructions from the user, wherein the control means makes it easier for the item to be attracted to the gaze point the smaller the amount of fluctuation.

[0009] The present invention 5 The aspect detects the gaze of the user's right eye and left eye, respectively. Gaze detection and detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye. Gaze detectionand controlling the processing to be executed based on the gaze point to be different depending on the amount of change in the line of sight of the non-dominant eye out of the right eye and the left eye. control Steps and a display step of displaying an image to be shown to the user; Have In the control step, the smaller the amount of variation, the shallower the depth of field of the image based on the gaze point. The control method is characterized by: A sixth aspect of the present invention is a control method comprising: a gaze detection step of detecting the gaze of a user's right eye and left eye, respectively; a gaze point detection step of detecting a gaze point based on the gaze of a dominant eye of the right eye or the left eye; and a control step of controlling so that the processing executed based on the gaze point is different depending on the amount of fluctuation in the gaze of the non-dominant eye of the right eye or the left eye, wherein in the control step, the smaller the amount of fluctuation, the smaller the effect of a change in the user's gaze on the detected position of the gaze point is. A seventh aspect of the present invention is a control method comprising a gaze detection step of detecting the gaze of a user's right eye and left eye, respectively; a gaze point detection step of detecting a gaze point based on the gaze of a dominant eye of the right eye or the left eye; a control step of controlling the processing executed based on the gaze point to be different depending on the amount of fluctuation in the gaze of the non-dominant eye of the right eye or the left eye; and a display step of displaying items that can be moved in response to instructions from the user, wherein in the control step, the smaller the amount of fluctuation, the higher the movement resolution of the items in the vicinity of the gaze point is. An eighth aspect of the present invention is a control method comprising a gaze detection step of detecting the gaze of a user's right eye and left eye, respectively; a gaze point detection step of detecting a gaze point based on the gaze of a dominant eye of the right eye or the left eye; a control step of controlling a process executed based on the gaze point to be different depending on an amount of fluctuation in the gaze of the non-dominant eye of the right eye or the left eye; and a display step of displaying items that can be moved in response to instructions from the user, wherein in the control step, the smaller the amount of fluctuation, the easier it is for the item to be attracted to the gaze point.

[0010] The present invention 9 The present invention is also directed to a program for causing a computer to function as each of the means of the above-mentioned gaze detection device. 10 The aspect is a computer readable program that causes a computer to function as each of the means of the above-mentioned gaze detection device. possible storage media. Effect of the Invention

[0011] According to the present invention, processing based on the gaze point can be suitably performed. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram illustrating a configuration of an HMD according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of an HMD according to the first embodiment. [Diagram 3] 1 is a diagram for explaining the principle of a gaze detection method according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an eye image according to the first embodiment. [Diagram 5] 4 is a flowchart of a gaze detection operation according to the first embodiment. [Figure 6] 11 is a flowchart of a control operation of a gaze point processing according to the first embodiment. [Figure 7] FIG. 4 is a diagram for explaining a gaze point process according to the first embodiment. [Figure 8] FIG. 11 is a diagram for explaining a gaze point process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] <<Example 1>> A first embodiment of the present invention will be described below with reference to FIGS.

[0014] <Configuration Description> FIG. 1 is a schematic diagram of a head mounted display (HMD) 100 as an example of a display device having a gaze detection device according to a first embodiment. The HMD 100 is mounted on the head of a user. The HMD 100 is a display device that is worn when used. On the left side of FIG. 1, the HMD 100 is shown as seen from the top of the head of a user wearing the HMD 100. of The right side of the figure includes a block diagram showing the functional configuration of the gaze detection device. The HMD 100 can also be regarded as an example of a gaze detection device having a display unit.

[0015] A user wears the housing 103 of the HMD 100, and can view an image displayed on a left display 106 with a left eye 101 through a left eyepiece 104, and can view an image displayed on a right display 107 with a right eye 102 through a right eyepiece 105. The HMD 100 can be worn in two ways, for example, non-transparent wearing and transparent wearing. When the HMD 100 is worn in a non-transparent manner, images stored inside the HMD 100 or images acquired by the HMD 100 from the outside, specifically images captured in the past, game images, etc., are displayed on the display (left display 106 and right display 107). When the HMD 100 is worn in a transparent manner, a live image (almost real-time image) captured by the left camera 108 is displayed on the left display 106, and a live image captured by the right camera 109 is displayed on the right display 107. An image that combines the live image with another image may be displayed on the display, or the live image and another image may be appropriately switched and displayed on the display. The left gaze detector 110 detects (estimates) the gaze (gaze direction; gaze position) of the left eye 101 of a user wearing the HMD 100, and the right gaze detector 111 detects the gaze of the right eye 102 of the user. A method of detecting the gaze will be described later.

[0016] Next, each functional unit of the gaze detection device will be described. The gaze detection unit 112 detects the gazes of the left eye 101 and right eye 102 of the user. Specifically, the gaze detection unit 112 acquires gaze detection results (gaze information) from the left gaze detector 110 and right gaze detector 111, respectively. The dominant eye determination unit 113 determines which of the left eye 101 and the right eye 102 is the dominant eye and which is the non-dominant eye (which of the gaze information of the left eye 101 and the gaze information of the right eye 102 is the gaze information of the dominant eye and which is the gaze information of the non-dominant eye).

[0017] The gaze point detection unit 114 detects the gaze point (gaze position) based on gaze information of at least the dominant eye of the left eye 101 and the right eye 102. The concentration level detection unit 115 detects (determines) the amount of fluctuation in the user's gaze based on gaze information of at least the non-dominant eye of the left eye 101 and the right eye 102, and detects (determines) the degree of concentration of the user according to the amount of fluctuation.

[0018] The process control unit 116 performs processing based on the gaze point detected by the gaze point detection unit 114. At that time, the process control unit 116 controls the processing based on the gaze point based on the concentration level (amount of change in line of sight) detected by the concentration level detection unit 115. For example, the process control unit 116 adjusts the control amount of a specific function acting on the displays (left display 106 and right display 107) based on the concentration level.

[0019] There are various ways in which users look at the gaze point. In some cases, the user looks at the gaze point absentmindedly, and in other cases, the user looks at the gaze point with some intention and concentrates on it. If such differences in views can be automatically determined and processing (processing based on the gaze point) can be automatically controlled according to the determination result, the processing based on the gaze point can be performed suitably. For example, since a user who wants to have a high sense of immersion tends to concentrate on looking at the gaze point, the sense of immersion can be suitably increased by performing processing for increasing the sense of immersion only when the user's level of concentration is high. A user who wants to perform an operation on the GUI (such as pressing a button icon) also tends to concentrate on looking at the gaze point. Therefore, if an assistance function that assists an operation on the GUI is enabled only when the user's level of concentration is high, convenience can be improved (a decrease in convenience due to the assistance function being provided to a user who does not need assistance can be suppressed). In addition, since the user does not need to set (such as switching processing based on the gaze point), the sense of immersion can be further increased and convenience can be further improved.

[0020] It has been reported that when a user is looking at an object absentmindedly, only the gaze of the dominant eye is accurately directed toward the object, while the gaze of the non-dominant eye fluctuates (shakes) relatively greatly. It has also been reported that when a user is concentrating on looking at an object, such as when performing a task that requires depth information (e.g., pressing a button icon), the amount of gaze fluctuation of the non-dominant eye also becomes smaller and the eye is directed toward the object.

[0021] Therefore, in the first embodiment, the gaze point is detected based on at least the gaze information of the dominant eye, and the amount of change (degree of concentration) of the user's gaze is detected based on at least the gaze information of the non-dominant eye. Then, the process based on the gaze point is controlled based on the detected amount of change (degree of concentration). In this way, the process based on the gaze point can be suitably performed as described above.

[0022] 2 is a cross-sectional view of the HMD 100 cut along the YZ plane defined by the Y-axis and Z-axis shown in FIG. 1, and is a schematic diagram of the configuration of the HMD 100. In FIG. 2, the configuration of the HMD 100 is shown in more detail than in FIG. 1. In FIG. 2, the right eyepiece 105, the right display 107, the right camera 109, and the right gaze detector 111 are omitted in order to show a cross section seen from the left eye side of a user wearing the HMD 100. The right eyepiece 105 has the same configuration as the left eyepiece 104, the right display 107 has the same configuration as the left display 106, the right camera 109 has the same configuration as the left camera 108, and the right gaze detector 111 has the same configuration as the left gaze detector 110.

[0023] In the HMD 100, the CPU 128 controls the entire HMD 100 (each unit). The memory 129 stores images (image information) to be displayed on the displays (left display 106 and right display 107), programs executed by the CPU 128, and the like. For example, the CPU 128 executes the programs stored in the memory 129 to realize the functions of the gaze detection unit 112, dominant eye determination unit 113, gaze point detection unit 114, concentration detection unit 115, and process control unit 116.

[0024] The left display 106 is a display unit that displays an image, and is, for example, a combination of a liquid crystal panel and a backlight, an organic EL panel, etc. The display drive circuit 124 drives the left display 106 (and the right display 107). The left eyepiece 104 is an eyepiece for observing the image displayed on the left display 106.

[0025] The left camera 108 is a camera that captures an image of the outside when the HMD 100 is worn through the lens, and has a camera imaging element 125, an aperture mechanism 126, and a focus mechanism 127. The memory 129 has a function of storing an imaging signal (captured image) from the camera imaging element 125.

[0026] The left gaze detector 110 includes a light source 120, a light splitter 121, a light receiving lens 122, and an eye image sensor 123. The light source 120 is a light source that illuminates the left eye 101 for gaze detection. For example, the light source 120 is an infrared light emitting diode that emits infrared light that is insensitive to the user, and is arranged around the left eyepiece 104. The number of light sources 120 is not particularly limited. An optical image of the illuminated left eye 101 (eye image; an image formed by light emitted from the light source 120 and reflected by the left eye 101) passes through the left eyepiece 104 and is reflected by the light splitter 121. The eye image is then formed by the light receiving lens 122 on the eye image sensor 123, which is a two-dimensional array of photoelectric elements such as CMOS. The eye image includes a reflection image (corneal reflection image; Purkinje image) of the light from the light source 120 reflected by the cornea. The light receiving lens 122 positions the pupil of the left eye 101 and the ocular imaging element 123 in a conjugate imaging relationship. Using a predetermined algorithm described later, the line of sight of the left eye 101 is detected from the positional relationship between the pupil (pupil image) and the corneal reflection image in the eyeball image formed on the ocular imaging element 123. The memory 129 has a function of storing an imaging signal from the ocular imaging element 123 and a function of storing line of sight correction data.

[0027] <Explanation of gaze detection operation> The gaze detection method will be described with reference to Figs. 3, 4(A), 4(B), and 5. The gaze of the left eye 101 and the gaze of the right eye 102 are detected by the following method. Fig. 3 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for performing gaze detection. As shown in Fig. 3, the light sources 120a and 120b are arranged approximately symmetrically with respect to the optical axis of the light receiving lens 122, and illuminate the user's eye 140 (left eye 101 or right eye 102). A part of the light emitted from the light sources 120a and 120b and reflected by the eye 140 is collected by the light receiving lens 122 on the eye imaging element 123. Fig. 4(A) is a schematic diagram of an eye image (eyeball image projected on the eye imaging element 123) captured by the eye imaging element 123, and Fig. 4(B) is a diagram showing the output intensity of the CMOS in the eye imaging element 123. Fig. 5 is a schematic flowchart of the gaze detection operation.

[0028] 5, when the gaze detection operation starts, the light sources 120a and 120b emit infrared light toward the user's eye 140. An image of the user's eyeball illuminated by the infrared light is formed on the eye imaging element 123 through the light receiving lens 122 and is photoelectrically converted by the eye imaging element 123. As a result, an electrical signal of the eye image that can be processed is obtained.

[0029] In step S002, the gaze detector (left gaze detector 110 or right gaze detector 111) sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging element 123 to the CPU 128.

[0030] In step S003, the CPU 128 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 120a and 120b and the pupil center c from the eye image obtained in step S002.

[0031] Infrared light emitted from light sources 120a and 120b illuminates cornea 142 of user's eye 140. At this time, corneal reflection images Pd and Pe formed by a part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 122 and focused on eye imaging element 123 to become corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from ends a and b of pupil 141 are also focused on eye imaging element 123 to become pupil end images a' and b' in the eye image.

[0032] FIG. 4B shows luminance information (luminance distribution) of the region α in the eye image of FIG. 4A. In FIG. 4B, the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe, and the coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb. As shown in FIG. 4B, an extremely high level of luminance is obtained at the X-coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from the X-coordinate Xa to the X-coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the ocular imaging element 123), an extremely low level of luminance is obtained except for the X-coordinates Xd, Xe. Then, a luminance intermediate between the above two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the above two types of luminance is obtained in a region where the X coordinate (coordinate in the X-axis direction) is smaller than X coordinate Xa, and a region where the X coordinate is larger than X coordinate Xb.

[0033] From the luminance distribution as shown in FIG. 4B, the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of the corneal reflection images, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of the pupil edge images a' and b'. In addition, when the rotation angle θx of the optical axis of the eye 140 relative to the optical axis of the light receiving lens 122 is small, the X coordinate Xc of the pupil center image c' (center of the pupil image) obtained by forming an image of the light beam from the pupil center c on the eye imaging element 123 can be expressed as Xc≒(Xa+Xb) / 2. In other words, the X coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil edge images a' and b'. In this manner, the coordinates of the corneal reflection images Pd', Pe' and the coordinates of the pupil center image c' can be estimated.

[0034] In step S004, CPU 128 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eye 140 relative to light receiving lens 122, and can be found using a function of the distance (Xd-Xe) between corneal reflection images Pd', Pe'.

[0035] In step S005, CPU 128 calculates the rotation angle of the optical axis of eye 140 relative to the optical axis of light receiving lens 122. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 approximately coincide with each other. For this reason, if the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, then the rotation angle θx of eye 140 in the ZX plane (plane perpendicular to the Y axis) can be calculated by the following formula 1. The rotation angle θy of eye 140 in the ZY plane (plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0036] In step S006, the CPU 128 uses the rotation angles θx, θy calculated in step S005 to determine (estimate) the user's gaze position (the position where the gaze is fixed; the position where the user is looking) on ​​the display surface of the display (left display 106 or right display 107). If the coordinates (Hx, Hy) of the gaze position are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the gaze position can be calculated by the following formulas 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0037] Parameter m in Equations 2 and 3 is a constant determined by the configuration of the optical system (light receiving lens 122, etc.), and is a conversion coefficient that converts the rotation angles θx, θy into coordinates corresponding to the pupil center c on the display surface of the display. Parameter m is determined in advance and stored in memory 129. Parameters Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in gaze, and are acquired by performing a calibration operation. Parameters Ax, Bx, Ay, and By are stored in memory 129 before the gaze detection operation starts.

[0038] In step S007, the CPU 128 stores the coordinates (Hx, Hy) of the gaze position in the memory 129, and ends the gaze detection operation.

[0039] Although the gaze detection method using the corneal reflection images of light sources 120a and 120b has been described, the present invention is not limited to this, and any method may be used to detect the gaze as long as it uses an eye image, such as detecting the eye rotation angle from the eye image.

[0040] <Explanation of control operation of gaze processing> The control operation for controlling the gaze point processing (processing based on the gaze point) will be described with reference to Figs. 6, 7(A) and 7(B). Fig. 6 is a schematic flow chart of the control operation for the gaze point processing. Figs. 7(A) and 7(B) are schematic diagrams of the display on the displays (left display 106 and right display 107).

[0041] When the control operation starts, in step S101, the gaze detection unit 112 acquires gaze information (gaze detection result) of the left eye 101 from the left gaze detector 110, and acquires gaze information of the right eye 102 from the right gaze detector 111. The gaze detection unit 112 outputs (transmits) the acquired gaze information to the dominant eye determination unit 113. The method of gaze detection by the left gaze detector 110 and the right gaze detector 111 is as described above.

[0042] In step S102, the dominant eye discrimination unit 113 discriminates which of the left eye 101 and the right eye 102 is the dominant eye and which is the non-dominant eye (which of the gaze information of the left eye 101 and the gaze information of the right eye 102 is the gaze information of the dominant eye and which is the gaze information of the non-dominant eye). The dominant eye discrimination unit 113 outputs the dominant eye / non-dominant eye discrimination result and the gaze information (gaze information of the left eye 101 and the right eye 102) acquired by the gaze detection unit 112 to the gaze point detection unit 114 and the concentration detection unit 115. The dominant eye discrimination unit 113 may output the gaze information based on the discrimination result without outputting the dominant eye / non-dominant eye discrimination result. For example, the gaze information of the dominant eye may be output to the gaze point detection unit 114, and the gaze information of the non-dominant eye may be output to the concentration detection unit 115.

[0043] Most people have a dominant eye, and the distance between the gaze position of the dominant eye and the position of the object the user wants to see tends to be shorter than the distance between the gaze position of the non-dominant eye and the position of the object. In general, in order to sense perspective and depth with high accuracy, the non-dominant eye directs its gaze to the vicinity of the object (the gaze of the non-dominant eye fluctuates near the object). The amount of fluctuation (range of fluctuation) of the gaze of the non-dominant eye is relatively large, and it is not possible to accurately determine where the user is looking just by looking at the gaze of the non-dominant eye. As described above, the gaze detected for the non-dominant eye often does not necessarily match the gaze intended by the user, and it is known that the accuracy of gaze detection for the non-dominant eye is not high.

[0044] For example, the dominant eye discrimination unit 113 can discriminate the left eye 101 or the right eye 102, whichever has a smaller amount of deviation (amount of deviation in time series) between an object prepared for the user to gaze at and the detected gaze position, as the dominant eye, and discriminate the eye with a larger amount of deviation as the non-dominant eye. The dominant eye discrimination unit 113 may acquire the distance from the eye head to the pupil of the user who is looking at the object for each of the left eye 101 and the right eye 102, and discriminate the dominant eye and the non-dominant eye based on the acquired distance. A known technique can be used to discriminate the dominant eye / non-dominant eye. The user may register in advance which of the left eye 101 and the right eye 102 is the dominant eye and which is the non-dominant eye. In that case, the dominant eye discrimination unit 113 may discriminate which of the left eye 101 and the right eye 102 is the dominant eye and which is the non-dominant eye based on the registration information of the dominant eye / non-dominant eye.

[0045] In step S103, the gaze point detection unit 114 detects the gaze point based on the gaze information of at least the dominant eye of the left eye 101 and the right eye 102. As shown in FIG. 7(A), the gaze point is detected within the gaze detection area of ​​the display (left display 106 and right display 107). The gaze point detection unit 114 notifies the processing control unit 116 of the detected gaze point. As described above, it is known that the accuracy of gaze detection for the non-dominant eye is not high. Therefore, the gaze information of the dominant eye is used to detect the gaze point. In the processing based on the gaze point, the gaze point of the dominant eye may be used as the gaze point of the non-dominant eye. Note that the HMD 100 may have a function of correcting (reducing) the blur of the gaze of the dominant eye (detection result; gaze information) and the blur of the gaze of the non-dominant eye (detection result; gaze information). In that case, the gaze point detection unit 114 may detect the gaze point of the left eye 101 from the gaze information of the left eye 101, and may detect the gaze point of the right eye 102 from the gaze information of the right eye 102. The gaze point of the left eye 101 is a position on the left display 106 that the left eye 101 is gazed at, and the gaze point of the right eye 102 is a position on the right display 107 that the right eye 102 is gazed at.

[0046] In step S104, the concentration detection unit 115 detects (determines) the amount of change in the user's line of sight based on the line of sight information of at least the non-dominant eye of the left eye 101 and the right eye 102, and detects (determines) the user's concentration level according to the amount of change. The concentration detection unit 115 notifies the process control unit 116 of the detected concentration level. As described above, when the user is looking at an object absentmindedly, the line of sight of the non-dominant eye changes (shakes) relatively greatly. Also, when the user is performing an operation requiring depth information (e.g., pressing a button icon), when the user is concentrating on looking at the object, the amount of change in the line of sight of the non-dominant eye also decreases, and the non-dominant eye faces the direction of the object. Therefore, the concentration detection unit 115 detects the amount of change in the line of sight (amount of change over time) based on the line of sight information of the non-dominant eye, and detects a lower concentration level as the amount of change increases (the lower the amount of change, the higher the concentration level). The concentration level detection unit 115 may detect a value that changes continuously with respect to the change in the amount of variation as the concentration level, or may detect a value that changes stepwise with respect to the change in the amount of variation using one or more thresholds. For example, in order to detect either of two states of the concentration level, "low" or "high," the concentration level detection unit 115 may detect a "low" concentration level when the amount of variation is equal to or greater than a threshold, and may detect a "high" concentration level when the amount of variation is less than the threshold. The amount of variation may be the magnitude of variation from the gaze point detected by the gaze point detection unit 114, or may not be the magnitude of variation. For example, the amount of variation may be the magnitude of variation (such as a standard deviation) from an average position of gaze positions that change over time.

[0047] It is preferable to use the functions of estimating the gaze point from the gaze information of the dominant eye in the gaze point detection unit 114 and estimating the concentration level from the gaze information of the non-dominant eye in the concentration level detection unit 115 in order to reduce the load of calculation processing. When it is difficult to distinguish between the dominant eye and the non-dominant eye, such as when the eyes are substantially the same, the concentration level detection unit 115 may estimate the concentration level based on both the amount of change in the line of sight of the left eye 101 and the amount of change in the line of sight of the right eye 102. For example, the concentration level may be estimated according to the average of the amount of change in the line of sight of the left eye 101 and the amount of change in the line of sight of the right eye 102, or the concentration level may be estimated according to the larger of the amount of change in the line of sight of the left eye 101 and the amount of change in the line of sight of the right eye 102.

[0048] In step S105, the process control unit 116 performs processing based on the gaze point detected by the gaze point detection unit 114. At that time, the process control unit 116 controls the processing based on the gaze point based on the concentration level (amount of change in line of sight) detected by the concentration level detection unit 115.

[0049] In the first embodiment, the process based on the gaze point is a process of pseudo-controlling (changing) the depth of field of the image displayed on the displays (left display 106 and right display 107) by applying a blurring effect to the image, based on the gaze point. The blurring effect is applied by image processing such as filtering. The process control unit 116 controls the area to which the blurring effect is applied, the strength of the blurring effect, and the like, based on the concentration level (amount of change in line of sight) detected by the concentration level detection unit 115.

[0050] Here, consider a case where the degree of concentration detected by the concentration detection unit 115 is high, such as when the user's gaze is directed in the direction of a certain target and the user is intentionally concentrating on looking at the target. In this case, the process control unit 116 imparts a blurring effect to the image so that a display such as that shown in FIG. 7(A) is performed. The display of FIG. 7(A) will be explained. A clear display is maintained in the area including the gaze point and its surroundings, while the other areas are imparted with a blurring effect and are displayed unclearly. The area to which the blurring effect is applied and the area to which it is not applied are separated by a blurring effect line.

[0051] The size (diameter) of the blur effect line, the number of blur effect lines, the strength of the blur effect, etc. are not particularly limited, and may be changed according to the degree of concentration. For example, a plurality of blur effect lines may be set around the gaze point, and a stronger blur effect may be applied to a plurality of regions separated by the plurality of blur effect lines, the farther the region is from the gaze point. In addition, when the gaze point captures the face of a subject, the detection position of the gaze point may be changed to the center of the region including the face or body of the subject, and a blur effect may be applied to the image so that the depth of field based on the changed gaze point becomes shallow.

[0052] As described above, when the HMD 100 is worn in a see-through manner, a live image is displayed on the display. In this case, the aperture mechanism 126 and the focus mechanism 127 may be controlled so that a live image with a blurring effect is captured (a live image with a shallow depth of field based on the gaze point). By focusing on the gaze point with the focus mechanism 127 and reducing the aperture value with the aperture mechanism 126, a blurring effect can be applied to the live image (a live image with a shallow depth of field based on the gaze point can be captured). Of course, regardless of whether the HMD 100 is worn in a see-through manner or a non-see-through manner, a blurring effect may be applied to the image by image processing.

[0053] It is known that when humans concentrate, the peripheral visual field (also known as the effective visual field), which contributes greatly to cognition, narrows. Therefore, applying a blurring effect based on the gaze point to an image when the user is highly concentrated does not hinder the user's concentration. Rather, applying a blurring effect based on the gaze point to an image when the user is highly concentrated can increase the user's sense of immersion.

[0054] Consider a case where the concentration level detected by the concentration level detection unit 115 is low, such as when the user wants to check the entire image. In this case, the user may select the image with a blur effect applied as shown in FIG. Therefore, in this case, the process control unit 116 stops applying the blurring effect so that the display shown in Fig. 7(B) (a clear display overall) is performed. The display shown in Fig. 7(B) meets the needs of the user who wants to check the entire image, and is a display suitable for the user.

[0055] As described above, according to the first embodiment, the gaze point is detected based on the line of sight of the dominant eye of the right eye and the left eye, and the process based on the gaze point is controlled based on the amount of change in the line of sight of at least the non-dominant eye of the right eye and the left eye. This allows the process based on the gaze point to be performed suitably. Specifically, the process control unit 116 makes the depth of field of the image based on the gaze point shallower as the concentration level detected by the concentration level detection unit 115 increases (as the amount of change in the line of sight detected by the concentration level detection unit 115 decreases). This allows the user's immersive feeling and convenience to be improved.

[0056] The present invention is also applicable to HMD To Although application examples have been described, the present invention is applicable to devices in general that realize xR (virtual reality (VR), augmented reality (AR), mixed reality (MR), substitute reality (SR), etc.). The present invention is also applicable to devices that have an imaging unit (camera) and a display unit (display), such as an in-vehicle camera, a personal computer, a tablet terminal, or a smartphone. The gaze detection device to which the present invention is applied may be a device separate from the camera, the display device, etc.

[0057] <<Example 2>> A second embodiment of the present invention will be described below with reference to Fig. 8. In the following, the same points (configuration, processing, etc.) as those in the first embodiment will be omitted, and only the points different from the first embodiment will be described. The basic configuration and processing flow of the HMD according to the second embodiment are the same as those in the first embodiment, but the function of the processing control unit 116 is different.

[0058] In the second embodiment, as shown in FIG. 8, it is assumed that a button icon, which is a GUI displayed in a VR space, is selected and clicked with a pointing controller. For example, when a user moves a pointing controller with his / her hand, a virtual pointing controller displayed in the VR space also moves in conjunction with the movement. The pointing controller in the VR space emits a laser from its tip, and a pointer is displayed at the position where the laser hits the button icon. A click on the pointing controller is reflected in the button icon (the button icon indicated by the pointer) that the laser hits. The pointer is a type of item that can be moved according to an instruction from the user.

[0059] When using a pointing controller to move a pointer to a button icon in a VR space, there are problems such as the pointer not moving to the targeted button icon, passing over the button icon, pointing to a different button icon nearby, etc. In order to solve these problems, in the second embodiment, the controllability of the pointer movement is changed based on the user's concentration level.

[0060] The process control unit 116 of the second embodiment increases the movement resolution of the pointer near the point of gaze as the concentration level detected by the concentration level detection unit 115 increases (as the amount of change in the line of sight detected by the concentration level detection unit 115 decreases).

[0061] Specifically, when the degree of concentration detected by concentration detection unit 115 is high, process control unit 116 sets a control line based on the point of gaze, as shown in Fig. 8. Then, process control unit 116 controls the movement resolution of the pointer so that the pointer moves roughly outside the control line as viewed from the point of gaze, and finely inside the control line.

[0062] The size (diameter) of the control line, the number of control lines, the value of the movement resolution in each area divided by the control lines, etc. are not particularly limited, and may be changed according to the concentration level. In addition, when the gaze point captures the edge of the button icon, the detection position of the gaze point may be changed to the center of the button icon, etc. Then, the control line may be set based on the changed gaze point so that the movement resolution of the pointer becomes higher the closer it is to the changed gaze point.

[0063] For example, when a user wants to select or click a button icon, and wants to accurately focus the pointer on the button icon, the gaze point will focus on the desired button icon, and a high level of concentration will be detected. With the above control, in such a case, the farther the pointer is from the desired button icon, the more roughly the pointer will move, and the closer the pointer is to the desired button icon, the more finely the pointer will move. As a result, the pointer can be quickly and accurately aligned with the button icon.

[0064] When the concentration level detected by the concentration level detection unit 115 is low, the process control unit 116 does not set a control line (cancels the setting of the control line) and sets the movement resolution of the pointer uniformly over the entire image. At this time, the process control unit 116 sets the movement resolution lower than the movement resolution in the vicinity of the gaze point when the concentration level is high. This makes it possible to prevent the pointer movement speed from slowing down against the user's intention. Then, it becomes possible to move the pointer quickly over the entire image. In addition, by canceling the function of changing the movement resolution according to the pointer position, the processing load of the CPU 128 can also be reduced. Note that if the movement resolution in the vicinity of the gaze point is lower when the concentration level is low than when the concentration level is high, the control line may be set so that the movement resolution is higher closer to the gaze point.

[0065] The process control unit 116 may make it easier to make the pointer stick to the gaze point as the degree of concentration detected by the concentration detection unit 115 increases (as the amount of change in the line of sight detected by the concentration detection unit 115 decreases). For example, the process control unit 116 sets a longer distance (adhesion distance) as the degree of concentration increases, and when the pointer approaches within the range of the adhesion distance from the gaze point, the pointer is made to stick to the gaze point. When the user is concentrating on looking at a button icon, if the pointer approaches a desired button icon, the pointer moves as if it is attracted to the desired button icon, improving the speed and accuracy of the alignment. Also, the higher the concentration level, the easier the alignment. Furthermore, when the concentration level is high, even if the hand operating the pointer shakes, the pointer does not follow the hand shaking and continues to stick to the desired button icon, making it easier to maintain the aligned state, and thus making it possible to suppress erroneous operations such as selection errors. On the other hand, when the concentration level is low, it is possible to suppress the pointer from sticking to the button icon against the user's intention. Note that when the concentration level is low, the pointer may not be stuck to the gaze point.

[0066] The process control unit 116 may reduce the influence of the change in the user's gaze on the detection position of the gaze point as the concentration level detected by the concentration level detection unit 115 increases (as the amount of change in the gaze detected by the concentration level detection unit 115 decreases). In other words, the process control unit 116 may make it more difficult to change the detection position of the gaze point as the concentration level increases. For example, when the concentration level is high, the process control unit 116 treats eye shaking (fixational eye movement) against the user's intention and momentary changes in the gaze as errors and does not reflect them in the detection position of the gaze point. This makes it possible to maintain the detection position of the gaze point and execute the desired processing (processing as intended by the user) as processing based on the gaze point even if there is fixational eye movement or momentary changes in the gaze. On the other hand, when the concentration level is low, the process control unit 116 reflects all gaze changes, including fixational eye movement and momentary changes in the gaze, on the detection position of the gaze point. This makes it possible to quickly change the detection position of the gaze point. Note that the higher the concentration level is, the smaller the weight is applied to the current gaze position as the detection position of the gaze point. andThe detected position of the gaze point may be updated by combining the two images.

[0067] The above-mentioned three types of control (control of the movement resolution of the pointer, control of the adhesion of the pointer, and control of the detection position of the gaze point) may be appropriately combined and executed. The above-mentioned controls may be combined with other controls for aligning the pointer with the gaze point.

[0068] As described above, in the second embodiment as well, the process based on the gaze point can be suitably performed.

[0069] The present invention is also applicable to HMD To Although application examples have been described, the present invention is applicable to devices in general that realize xR (virtual reality (VR), augmented reality (AR), mixed reality (MR), substitute reality (SR), etc.). The present invention is also applicable to devices that have an imaging unit (camera) and a display unit (display), such as an in-vehicle camera, a personal computer, a tablet terminal, or a smartphone. The gaze detection device to which the present invention is applied may be a device separate from the camera, the display device, etc.

[0070] <Other Examples> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0071] 100: HMD 112: Gaze detection unit 113: Dominant eye determination unit 114: gaze point detection unit 115: concentration detection unit 116: processing control unit

Claims

1. A gaze detection means for detecting the gaze of each of a user's right eye and left eye; a gaze point detection means for detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control means for controlling a process to be executed based on the gaze point to be different depending on a variation in a line of sight of a non-dominant eye of the right eye and the left eye; a display means for displaying an image to be shown to the user; having The control means is configured to make the depth of field of the image based on the gaze point shallower as the amount of variation becomes smaller. A gaze detection device comprising:

2. A gaze detection means for detecting the gaze of a user's right eye and left eye, a gaze point detection means for detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control means for controlling so as to vary a process to be executed based on the gaze point according to a variation in a line of sight of a non-dominant eye out of the right eye and the left eye; having The control means reduces an influence of a change in the line of sight of the user on a detected position of the gaze point as the amount of change is smaller. A gaze detection device comprising:

3. A gaze detection means for detecting the gaze of a user's right eye and left eye, a gaze point detection means for detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control means for controlling a process to be executed based on the gaze point to be different depending on a variation in a line of sight of a non-dominant eye of the right eye and the left eye; a display means for displaying an item that can be moved in response to an instruction from the user, The control means increases the movement resolution of the item in the vicinity of the gaze point as the amount of fluctuation decreases. A gaze detection device comprising:

4. A gaze detection means for detecting the gaze of a user's right eye and left eye, a gaze point detection means for detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control means for controlling a process to be executed based on the gaze point to be different depending on a variation in a line of sight of a non-dominant eye of the right eye and the left eye; a display means for displaying an item that can be moved in response to an instruction from the user, The control means makes it easier for the item to be attracted to the gaze point as the amount of fluctuation is smaller. A gaze detection device comprising:

5. The amount of change in the line of sight is the magnitude of variation from the gaze point.

5. The gaze detection device according to claim 1, wherein the gaze detection device detects a gaze of a subject.

6. The control means controls so that, when an amount of change in the line of sight of the right eye and an amount of change in the line of sight of the left eye are substantially the same, the processing executed based on the gaze point is made different depending on both the amount of change in the line of sight of the right eye and the amount of change in the line of sight of the left eye.

6. The gaze detection device according to claim 1,

7. a gaze detection step of detecting the gazes of the right and left eyes of a user; a gaze point detection step of detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control step of controlling so that a process to be executed based on the gaze point is made different depending on a change amount of a line of sight of a non-dominant eye of the right eye and the left eye; a display step of displaying an image to be shown to the user; having In the control step, the depth of field of the image based on the gaze point is made shallower as the amount of fluctuation is smaller. A control method comprising:

8. A gaze detection step of detecting the gaze of a user's right eye and left eye, respectively; a gaze point detection step of detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control step of controlling so that a process to be executed based on the gaze point is made different depending on a change amount of a line of sight of a non-dominant eye of the right eye and the left eye; having In the control step, the influence of the change in the user's line of sight on the detected position of the gaze point is reduced as the amount of change is smaller. A control method comprising:

9. A gaze detection step of detecting the gaze of a user's right eye and left eye, respectively; a gaze point detection step of detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control step of controlling so that a process to be executed based on the gaze point is made different depending on a change amount of a line of sight of a non-dominant eye of the right eye and the left eye; a display step for displaying items that can be moved in response to an instruction from the user; P and having In the control step, the smaller the amount of fluctuation, the higher the movement resolution of the item in the vicinity of the gaze point. A control method comprising:

10. A gaze detection step of detecting the gazes of a user's right eye and left eye, respectively; a gaze point detection step of detecting a gaze point based on a line of sight of a dominant eye out of the right eye and the left eye; a control step of controlling so that a process to be executed based on the gaze point is made different depending on a change amount of a line of sight of a non-dominant eye of the right eye and the left eye; a display step of displaying items that can be moved in response to an instruction from the user; having In the control step, the smaller the amount of fluctuation, the easier it is for the item to be attracted to the gaze point. A control method comprising:

11. A program for causing a computer to function as each of the means of the gaze detection device according to any one of claims 1 to 6.

12. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the gaze detection device according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Optical device with visual axis function

    JP2004008323A

  • Method and system for image processing

    JP2007034628A

  • Foveated Video Link for VR, Low Latency, Wireless HMD Video Streaming with Eye Tracking

    JP2020504959A

  • System and Method for Dynamic Content Delivery Based on Gaze Analytics

    US20150130703A1

  • System and Method for Dynamic Content Delivery Based on Gaze Analytics

    US20200089317A1