Display control apparatus, control method, and storage medium

Abstract

To enable authentication using a line of sight with higher accuracy.SOLUTION: A display control unit configured to display a plurality of components constituting a preset password on a display surface of a display device, an estimation unit configured to acquire an estimated gaze point obtained by estimating a position at which a user is looking, an information acquisition unit configured to acquire calibration information used for correction of the estimated gaze point based on a positional relationship between the estimated gaze point and each component, and a correction unit configured to correct the estimated gaze point based on the calibration information; And an authentication unit configured to determine whether the user has input the password.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a display control device, a control method, and a program. [Background technology]

[0002] In recent years, XR (Extended Reality) technology has become well known, blending real and virtual spaces to create new experiences. XR technology is a collective term for VR (Virtual Reality), MR (Mixed Reality), AR (Augmented Reality), and SR (Substitutional Reality). VR technology allows users to experience virtual reality. MR technology allows users to experience mixed reality, which seamlessly blends reality and virtual reality. AR technology overlays and projects virtual space onto real space. SR technology overlays and projects past footage onto real space for users to see.

[0003] One type of XR technology is an XR system that uses a head-mounted display (hereinafter referred to as "HMD"). In an XR system, the actual outside scenery and virtual objects are displayed on the HMD and presented to the user. This allows the user to experience an XR space.

[0004] The application of XR technology is expanding beyond games to include training and simulations for jobs where serious incidents are likely to occur (surgery, construction, factories, finance, etc.), and demand for XR technology is on the rise. In addition, because HMDs are worn on the head, they are well suited to gaze detection functions that detect the user's gaze information.

[0005] However, safety against risks such as theft and impersonation cannot be ignored. Iris authentication, a form of biometric authentication, can be considered as a method of personal authentication in an HMD with the above-mentioned gaze input function. Biometric authentication involves a high risk of theft because it uses information that can identify an individual. Another authentication method involves inputting a password using a controller or the like. However, there are problems with the cost involved in attaching an operating unit such as a controller, and with users who are physically handicapped and cannot use this input method. To solve this problem, Patent Document 1 describes a personal authentication method that uses gaze input to input a password. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-162420 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, if the calibration work to obtain the amount of deviation that occurs between the user's intended viewing position (viewpoint position) and the user's viewpoint position recognized by the camera is not performed in advance, the accuracy of gaze input will decrease.

[0008] Therefore, the present invention provides a technology that enables more accurate authentication using the line of sight. [Means for solving the problem]

[0009] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. A display that displays a plurality of elements that make up a preset password on the display surface of a display device. display control means; an estimation means for acquiring an estimated gaze point that estimates the position at which the user is looking; an information acquisition means for acquiring calibration information used to correct the estimated gaze point based on a positional relationship between the estimated gaze point and each component; a correction means for correcting the estimated gaze point based on the calibration information; an authentication means for determining whether the user has entered the password by determining the component the user is looking at based on the estimated gaze point; The display control device is characterized by having:

[0010] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a display control step of displaying a plurality of elements constituting a preset password on a display surface of a display device; an estimation step of acquiring an estimated gaze point by estimating a position where the user is looking; an information acquisition step of acquiring calibration information used to correct the estimated gaze point based on a positional relationship between the estimated gaze point and each component; a correction step of correcting the estimated gaze point based on the calibration information; an authentication step of determining whether the user has entered the password by determining the component the user is looking at based on the estimated gaze point; The control method is characterized by having the following. [Effects of the Invention]

[0011] According to the present invention, it is possible to perform authentication using the line of sight with higher accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an HMD. [Figure 2] FIG. 1 is a diagram illustrating the placement of an HMD. [Figure 3] FIG. 10 is a diagram illustrating a gaze detection method. [Figure 4] FIG. 10 is a diagram illustrating a gaze detection method. [Figure 5]10 is a flowchart of gaze detection. [Figure 6] FIG. 10 is a diagram illustrating acquisition of calibration information. [Figure 7] 10 is a flowchart of an image display process. [Figure 8] FIG. 10 is a diagram illustrating image generation. [Figure 9] FIG. 10 is a diagram illustrating image generation. [Figure 10] 10 is a flowchart of a calibration information acquisition process. [Figure 11] 10 is a flowchart showing the process when a password entry fails. [Figure 12] FIG. 10 is a diagram illustrating password input. [Figure 13] 10A and 10B are diagrams illustrating an input determination area and an input auxiliary area. [Figure 14] FIG. 10 is a diagram illustrating an additional icon. [Figure 15] FIG. 10 is a diagram illustrating an eye-gaze input error. [Figure 16] 10A and 10B are diagrams illustrating an improvement in the accuracy of an estimated gaze point. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Although the configuration of the device will be described below using a configuration that realizes MR, the embodiment can also be realized with a configuration that realizes VR, XR, or SR.

[0014] (Device configuration) 1 is a block diagram showing an example of the configuration of an HMD 100, which is a head-mounted display device according to this embodiment. FIG. 2 is a diagram illustrating the layout of the HMD 100.

[0015] The HMD 100 includes an imaging unit 101, a system control unit 102, an image processing unit 103, an image generation processing unit 104, a nonvolatile memory 105, a memory 106, a head position detection unit 107, a gaze detection unit 108, a display unit 109, and a bus 110. The components of the HMD 100 other than the bus 110 are connected via the bus 110. The HMD 100 may include only some of the components such as the display unit 109, and the other components may be included in a display control device. In this case, the display control device controls the display on the display unit 109 of the HMD 100.

[0016] The imaging unit 101 is a camera unit that captures an image by capturing an image of a subject. The imaging unit 101 has an imaging element (such as a CCD sensor or a CMOS sensor), a lens, a shutter, an aperture, a distance measurement unit, and an A / D converter. The image captured by the imaging unit 101 is transmitted to the image processing unit 103. The image undergoes various processes (described later) in the image processing unit 103, and is then recorded in the memory 106 as a real space image. As shown in FIG. 2, the imaging unit 101 is disposed in front of the user's head so as to capture an image of the range in the user's line of sight.

[0017] The system control unit 102 controls each component of the HMD 100 according to a program stored in the nonvolatile memory 105, for example, using the memory 106 as a work memory.

[0018] The image processing unit 103 performs various types of image processing on images. Images to be processed include images stored in the memory 106 and images acquired by the imaging unit 101. The image processing performed by the image processing unit 103 includes image encoding, decoding, compression, decompression, enlargement / reduction (resizing), noise reduction, color conversion, and the like. The image processing unit 103 is, for example, a circuit block dedicated to performing specific image processing. Depending on the type of image processing, the system control unit 102, rather than the image processing unit 103, may also perform the image processing according to a program.

[0019] The image generation processing unit 104 is realized by a device such as a graphics card. Generally, the image generation processing unit 104 has a graphics memory (not shown). Image information generated by the system control unit 102 is written via the bus 110 to the graphics memory held by the image generation processing unit 104. The image generation processing unit 104 generates a virtual space image such as CG based on the image information written to the graphics memory. Note that the graphics memory does not necessarily have to be held by the image generation processing unit 104. The memory 106 may also realize the function of the graphics memory.

[0020] Furthermore, the image generation processing unit 104 synthesizes the generated virtual space image with a real space image (the real space image written in the memory 106) to generate a mixed reality image. Then, the image generation processing unit 104 converts the mixed reality image into an image signal suitable for the display unit 109 and sends it out. Furthermore, in response to an instruction from the system control unit 102, the image generation processing unit 104 generates an image related to the user's gaze or eyes based on gaze point data (information on the position where the user is looking) calculated by the gaze detection unit 108.

[0021] The nonvolatile memory 105 stores images, other data, various programs (various programs for operating the system control unit 102), etc. The nonvolatile memory 105 includes, for example, a hard disk (HD) or a ROM (Read Only Memory).

[0022] The memory 106 is a storage unit that stores (records) the real space image obtained from the imaging unit 101. The memory 106 stores the gaze point data of the user calculated by the gaze detection unit 108. The memory 106 may store gaze point data of one or more people, and may store gaze point data of a person other than the user wearing the HMD 100. The memory 106 stores the head position The memory 106 stores various data such as the user's head position data obtained by the detection unit 107, and mixed reality images to be displayed on the display unit 109. The memory 106 also stores password information entered by the user at the time of initial setup. The memory 106 has a storage capacity sufficient to store the above-mentioned data.

[0023] Head position detection unit 107 includes an internal sensor that measures data related to the position and orientation of HMD 100. Head position detection unit 107 calculates position data of the user's head (head position data) based on the data measured by the sensor. Head position detection unit 107 includes a gyro sensor, an acceleration sensor, a magnetic sensor, or the like. The calculated head position data is stored in memory 106.

[0024] The gaze detection unit 108 is an estimation unit that acquires an estimated gaze point, which is an estimated position where the user is looking on the display surface of the display unit 109. The gaze detection unit 108 includes a sensor for measuring information about the eyes of the user of the HMD 100 (information such as gaze). As will be described in detail later, the gaze detection unit 108 is also an information acquisition unit that calculates (acquires) the user's gaze point data (data indicating the estimated gaze point) and calibration information based on the measured data. Specifically, the gaze detection unit 108 performs a user's eye detection process based on an eye image (eyeball image) in which the eyeball is captured, and calculates the user's gaze point data and calibration information. The user's gaze point data and calibration information are stored in the memory 106. In the gaze detection unit 108, for example, as shown in FIG. 3, a light receiving lens 122, light sources 120a and 120b, and an eye image sensor 123 are arranged at positions where the user's eyes are reflected.

[0025] The display unit 109 displays a mixed reality image (display image) on a display surface. For example, as shown in Fig. 2, the display unit 109 is disposed in the HMD 100 at a position that covers both eyes of the user.

[0026] (Description of gaze detection operation) The gaze detection method will be described using Figures 3, 4A, 4B, and 5. Figure 3 is a diagram for explaining the principle of the gaze detection method and is a schematic diagram of an optical system for performing gaze detection. As shown in Figure 3, illumination light sources 120 (light sources 120a and 120b) are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 122 and illuminate user's eyeball 140. A portion of the light emitted from light sources 120a and 120b and reflected by eyeball 140 is collected by light-receiving lens 122 onto ocular imaging element 123. Figure 4A is a schematic diagram of an eye image captured by ocular imaging element 123 (eyeball image projected onto ocular imaging element 123), and Figure 4B is a diagram showing the output intensity of the CCD in ocular imaging element 123. Figure 5 is a schematic flowchart of the gaze detection operation.

[0027] 5, light sources 120a and 120b emit infrared light toward user's eyeball 140. An image of the user's eyeball illuminated by the infrared light is formed on eye image sensor 123 through light receiving lens 122 and is photoelectrically converted by eye image sensor 123. As a result, an electrical signal of the eye image that can be processed is obtained.

[0028] In step S502, the gaze detection unit 108 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 123 to the image processing unit 103.

[0029] In step S503, the image processing unit 103 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 120a and 120b and the pupil center c from the eye image obtained in step S502.

[0030] Infrared light emitted from light sources 120a and 120b illuminates cornea 142 of user's eyeball 140. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 122 and formed on ocular imaging element 123 as corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from edges a and b of pupil 141 are also formed on ocular imaging element 123 as pupil edge images a' and b' in the eye image.

[0031] FIG. 4B shows luminance information (luminance distribution) of region α' in the eye image of FIG. 4A. In FIG. 4B, the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the X-axis (horizontal) coordinates of the corneal reflection images Pd' and Pe' are set to Xd and Xe, and the X-axis coordinates of the pupil edge images a' and b' are set to Xa and Xb. As shown in FIG. 4B, an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 123), an extremely low level of luminance is obtained except for the coordinates Xd and Xe. A luminance intermediate between the two types of luminance described above is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the two types of luminance described above is obtained in a region where the X coordinate (coordinate in the X-axis direction) is smaller than coordinate Xa and a region where the X coordinate is larger than coordinate Xb.

[0032] From the luminance distribution shown in FIG. 4B, the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of extremely high luminance, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of extremely low luminance. Furthermore, when the rotation angle θx of the optical axis of the eyeball 140 relative to the optical axis of the light receiving lens 122 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the ocular imaging element 123 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X-coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0033] In step S504, the image processing unit 103 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of the eyeball 140 relative to the light receiving lens 122, and can be calculated using a function of the distance (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0034] In step S505, the image processing unit 103 calculates the rotation angle of the optical axis of the eyeball 140 relative to the optical axis of the light receiving lens 122. The X coordinate of the midpoint between the corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of the cornea 142 are approximately the same. Therefore, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, the rotation angle θ of the eyeball 140 in the ZX plane (plane perpendicular to the Y axis) is X can be calculated using the following equation 1. The rotation angle θy of the eyeball 140 in the ZY plane (plane perpendicular to the X axis) can also be calculated in the same way as the rotation angle θx. β×Oc×SINθ X ≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0035] In step S506, the image processing unit 103 uses the calculated rotation angles θx and θy to determine (estimate) the user's gaze point (the position where the gaze is fixed; the position where the user is looking) in the viewing image displayed on the display unit 109. If the coordinates (Hx, Hy) of the gaze point are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the gaze point can be calculated by the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0036] The parameter m in equations 2 and 3 is a constant determined by the configuration of the finder optical system (such as the light receiving lens 122), and is a conversion coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c in the visual image. The parameter m is determined in advance and stored in the memory unit 129. The coefficients Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in the gaze, and are acquired by performing a calibration operation and stored in the memory unit 129 before the gaze detection operation starts.

[0037] In step S507, the image processing unit 103 stores the coordinates (Hx, Hy) of the gaze point in the memory unit 129, and ends the gaze detection operation. Note that the gaze direction is the direction from the user's eyeball to the gaze point, and therefore can be calculated based on the coordinates of the user's eyeball and the coordinates of the gaze point.

[0038] In the above description, a method for acquiring the coordinates of the gaze point (viewpoint position) on the display unit using the corneal reflection images of the light sources 120a and 120b has been described. However, the present invention is not limited to this, and any method for acquiring the coordinates of the gaze point (eyeball rotation angle) from a captured eyeball image may be used.

[0039] (Calibration procedure explanation) As described above, the image processing unit 103 acquires the rotation angles θx and θy of the eyeball from the eye image in the gaze detection routine. The image processing unit 103 estimates the position of the gaze point by performing a calculation to convert the coordinates of the pupil center position into a corresponding position on the display unit 109.

[0040] Ideally, the estimated gaze point A and the position where the user is actually looking should match, as shown in Figure 6A. However, due to factors such as individual differences in the shape of the human eyeball, unless the user adjusts the gaze correction coefficients Ax, Ay, Bx, and By to appropriate values, a discrepancy will occur between the position B where the user is actually gazing and the calculated estimated gaze point C, as shown in Figure 6B. In the above example, although the user intended to gaze at the person at position B in the virtual space, it is mistakenly estimated that the user is gazing at the background. Therefore, a calibration procedure must be performed to obtain the appropriate correction coefficient values ​​for the user and store them in the camera. An example of the calibration procedure is shown below.

[0041] The first requirement in the calibration process is to identify the location on the display unit 109 where the user is actually gazing. As shown in FIG. 6C , multiple indices at different positions are highlighted and the user is asked to look at the indices, thereby identifying the location of gaze. A gaze point detection flow is performed when the user gazes at each target, and an appropriate coefficient value is calculated from the calculated coordinates of multiple estimated gaze points and the coordinate positions of each target. This section explains how to calculate the coefficients (i.e., Ax and Bx in the coordinate calculation formula) of the optical axis of the eyeball 117 in the ZX plane direction based on the detection results of the gaze points of the indices CL and CR used in the calibration process. The coordinate of the indices CL on the display unit 109 (display element) is defined as Hx1, and the coordinate of the indices CR is defined as Hx2. Furthermore, the rotation angles of the eyeball at the time of gaze, acquired by the gaze detection routine, are defined as θx1 and θx2, respectively. It is desirable that θx1 and θx2 are values ​​that take into account the effects of variability and noise, such as the average or median of multiple acquisition results. Combined with the coordinate calculation formula above, these values ​​have the following relationship: Hx1=m×(Ax×θx1+Bx) Hx2=m×(Ax×θx2+Bx)

[0042] By solving the above equation, the image processing unit 103 obtains the coefficient in the ZX plane direction (i.e., the above The image processing unit 103 also calculates the coefficients in the ZY plane direction (i.e., Ay and By in the coordinate calculation formula) from the indices CT and CB on the display unit 109. The image processing unit 103 then stores the calculated coefficients Ax, Bx, Ay, and By in the memory 106, and the calibration process is completed.

[0043] A process for displaying an image on the display unit 109 when the HMD 100 shown in Fig. 1 is used will be described with reference to the flowchart of Fig. 7, Figs. 8A to 8C, and Fig. 9. Figs. 8A to 8C and 9 are diagrams for explaining the generation of an image (data) to be displayed on the display unit 109.

[0044] First, the operation of the system control unit 102 will be described in detail with reference to the flowchart in Fig. 7. At the start of the flowchart in Fig. 7, the user is wearing the HMD 100, and the HMD 100 is activated or has been released from sleep mode.

[0045] Furthermore, in front of the user, there is a real space 801 as shown in Fig. 8A. The processing of each step on the time axis will be explained using timing T0 to timing T4 shown in Fig. 9 as an example. With respect to timing T0 to timing T4, the timing advances by one each time a series of processing steps from step S701 to step S705 (described later) is executed.

[0046] In step S700, the system control unit 102 acquires calibration information and also acquires a password input value entered using the user's line of sight. The system control unit 102 then determines whether or not "user's password information (preset password information) stored in advance in the memory 106" matches the "password input value." If it is determined that the password information matches the password input value, it is determined that authentication has been successful, and the process proceeds to step S701. In this case, the system control unit 102 stores in the memory 106 the calibration information acquired while the user was entering the password. Details of the process in step S700 will be described later.

[0047] In step S701, the system control unit 102 controls the imaging unit 101 to capture an image of the subject. Thereafter, the image generation processing unit 104 performs various image processing on the captured image of the subject (hereinafter referred to as the "captured image"), and then stores the captured image that has undergone various image processing in the memory 106 (temporary storage memory) as a real space image. Note that in this embodiment, the range indicated by the dashed frame 802 in FIG. 8A is the range that can be captured by the imaging unit 101, and an image within that range can be acquired as the captured image. Furthermore, the range indicated by the solid frame 803 is the range that can be displayed on the display unit 109.

[0048] In step S702, the system control unit 102 controls the gaze detection unit 108 to acquire (calculate) gaze point data of the user at each timing. Specifically, the gaze detection unit 108 corrects the "gazing point data calculated by the gaze detection operation (processing of the flowchart in FIG. 5)" based on the calibration information acquired in step S700. As a result, the gaze detection unit 108 calculates new gaze point data. The system control unit 102 stores the calculated gaze point data in the memory 106.

[0049] For example, in FIG. 9, at timing T0, the user's gaze direction is calculated to be leftward based on the movement of the user's head and eyes. At timing T1, the user's gaze direction is calculated to be forward. While FIG. 9 shows an example of the gaze direction in the horizontal direction, the gaze direction in the vertical (perpendicular) direction can also be calculated in a similar manner. In addition, in the gaze detection process, the gaze of only one eye may be detected, or the gaze of both eyes may be detected. If the lines of sight of both eyes are detected, it is possible to improve the accuracy of the line of sight information and calculate the focal position in the depth direction.

[0050] In step S703, the head position detection unit 107 acquires head position data of the user. In this embodiment, the head position detection unit 107 can acquire at least information on the orientation of the head (the direction and amount of rotation relative to a reference orientation) as head position data. For example, in FIG. 9, information indicating that the orientation of the user's head is directly above is acquired as head position data between timing T0 and timing T2. Then, information indicating that the orientation of the user's head is slightly to the left of directly above is acquired as head position data between timing T3 and timing T4. The head position data is stored in memory 106.

[0051] In step S704, the system control unit 102 reads the gaze area data and the user's gaze point data from the memory 106, and updates the user's gaze area data for the physical space image. The gaze area data is data on the center coordinates of an area in the physical space image to be displayed on the display unit 109. Therefore, the gaze area data can be updated by adding a difference (amount of deviation) from the coordinates of the previous gaze point, which is calculated from the user's gaze point data, to the current gaze area data (center coordinates of the area displayed on the display unit 109).

[0052] Specific processing will be described using timing T0 in Fig. 9 as an example. For ease of explanation, it will be assumed that the gaze point data handles only three directions in the horizontal direction (left, center, and right), and that a unique coordinate deviation amount is defined for each direction.

[0053] For example, at each timing in FIG. 9, the area surrounded by the dashed line in real space is the area of ​​the real space image. The area surrounded by the solid line is the area showing the image (display image) displayed on the display unit 109. At timing T0, the user's gaze point data is data showing that the user is facing leftward with respect to the display unit 109. Therefore, at timing T1, the coordinates shown by the gaze area data are updated so as to shift leftward.

[0054] In contrast, at timing T1, the user's gaze point data indicates that the user is facing forward with respect to display unit 109. Therefore, at timing T2, the coordinates indicated by the gaze area data are not updated. Furthermore, at timing T2, the user's gaze point data indicates that the user is facing right with respect to display unit 109. Therefore, at timing T3, the coordinates indicated by the gaze area data are updated so as to be shifted rightward.

[0055] The initial value of the fixation area data may be any value, and may be, for example, the center coordinate of the real space image. The updated fixation area data is stored in the memory 106.

[0056] In step S705, the image generation processing unit 104 generates a virtual space image based on the user's gaze area data and head position data. The virtual space image may be an image for generating a mixed reality image. In this embodiment, the image generation processing unit 104 acquires the coordinates and orientation of the head in real space from the head position data, and generates one or more CGs corresponding to the coordinates and orientation based on the image information written in the graphics memory. The image generation processing unit 104 then arranges the CGs to be arranged within the gaze area from the one or more generated CGs. In this way, the image generation processing unit 104 generates a virtual space image such as the image shown in FIG. 8B.

[0057] In step S706, the image generation processing unit 104 generates a mixed reality image, such as the image shown in FIG. 8C, based on the gaze area data, the real space image, and the virtual space image. The system control unit 102 displays the mixed reality image on the display unit 109.

[0058] In step S707, the system control unit 102 determines whether or not an instruction to end the display of the mixed reality image has been issued by the user. If it is determined that an instruction to end the display of the mixed reality image has been issued, the display process of the display unit 109 ends, and the process of this flowchart ends. If it is determined that an instruction to end the display of the mixed reality image has not been issued, the process proceeds to step S701.

[0059] (Process to obtain calibration information when logging in with password) The process (method) of simultaneously acquiring calibration information when logging in by entering a password using gaze will be described in detail with reference to the flowchart of Fig. 10. The flowchart of Fig. 10 shows detailed processing in step S700. In other words, the flowchart of Fig. 10 is a flowchart for simultaneously entering a password and acquiring calibration information.

[0060] First, as a prerequisite, gaze point data is calculated for each sampling period based on the gaze detection operation described using the flowchart of FIG. 5 and the like.

[0061] In step S1001, the system control unit 102 randomly arranges icons (hereinafter referred to as "password icons"), which are display items indicating each component of the password information stored in the memory 106, on the periphery or center of the display surface of the display unit 109. For example, FIG. 12 shows a display image 1100 that is displayed on the display surface of the display unit 109 when the user has previously set a password consisting of nine unique digits. The direction of the arrows in FIG. 12 indicates the input order of the components of the previously set password. Password icons A1 to A9 are display items that indicate each digit that makes up the password. In this embodiment, a password consisting of nine unique digits will be used for explanation. However, the number of digits in the password may be any number. Passwords are not limited to numbers, and may be composed of letters such as alphabets.

[0062] The process of step S1001 is performed to accurately acquire calibration information. To do so, the user needs to see indicators at the center and periphery of the display surface of display unit 109, and therefore password icons are randomly placed at the periphery or center of the display surface.

[0063] In step S1002, the system control unit 102 sets an input assistance area based on the coordinates of each password icon displayed in step S1001 and a preset distance. In this embodiment, the "input assistance area" is a determination area used for password input instead of the "input determination area" when calibration information is not applied (the estimated gaze point is not corrected). The "input determination area" is a determination area for password input when calibration information is applied.

[0064] For example, FIG. 13 shows an input determination area A (input determination area A including the display area of password icon A5) within a certain range centered on password icon A5, together with a display image 1100 including a plurality of password icons. FIG. 13 also shows an "input assistance area B", an "estimated fixation point P'" (estimated fixation point P') indicated by the fixation point data, and a "fixation point P" that the user is actually looking at. The size relationship between the input determination area A used after the calibration operation and the input assistance area B used before the calibration operation is A < B. When the estimated fixation point P' is located in the input determination area A, the system control unit 102 determines that the user is looking at the password icon A5 within the input determination area A (while inputting the number "5").

[0065] However, if the calibration operation has not been performed in advance, as shown in FIG. 13, a deviation may occur between the estimated fixation point P' and the fixation point P that the user is actually looking at, and there is a possibility that the password intended by the user cannot be input. Therefore, before the calibration operation is performed, the system control unit 102 expands the area (determination area) for determining whether the user is looking at the password icon from the input determination area A to the input assistance area B. By doing so, it is possible to match the password icon corresponding to the actually viewed point P with the password icon that the user wants to select.

[0066] In step S1003, the system control unit 102 displays at least one of the gaze pointer and the display item indicating the input assistance area B on the display unit 109 based on the gaze point data calculated for each sampling period. The gaze pointer is a display item indicating an estimated fixation point, which is a position estimated to be where the user is looking. However, the gaze pointer and the display item indicating the input assistance area B may not be displayed if they seem to obstruct the user's gaze.

[0067] In step S1004, the system control unit 102 determines whether the estimated point of gaze is located in the judgment area set for each password icon. At this time, when the second or subsequent component of the password is to be input, the estimated point of gaze may be corrected based on the calibration information stored in the memory 106. If it is determined that the estimated point of gaze is located in any of the judgment areas, it is determined that the user is inputting (is currently inputting) the component corresponding to the password icon (hereinafter referred to as the "selected icon") in the judgment area (the judgment area in which the estimated point of gaze is located). In this case, the process proceeds to step S1005. If it is determined that the estimated point of gaze is not located in any of the judgment areas, the process of step S1004 is repeated.

[0068] In step S1005, the system control unit 102 displays an additional icon (display item) that highlights the selected icon (component) that the user is looking at so that the user can focus on the selected icon. FIG. 14 shows an example of the additional icon A'. One cause of failure in the calibration process is that the user's gaze point becomes dispersed. For this reason, the additional icon A' is displayed in order to prevent the gaze point from becoming dispersed. The icon A' may be set to be hidden when the process proceeds to step S1015 (when the password input process and calibration for the selected icon are completed).

[0069] In step S1006, the system control unit 102 calculates an eye-gaze input error based on the difference between the coordinates of the estimated gaze point and the display coordinates of the password icon determined in step S1004. For example, Fig. 15 shows the coordinates of the password icon A5 and the coordinates of the user's estimated gaze point P' in a display image 1100 displaying the password icon. Here, the position coordinates of the estimated gaze point P' are assumed to be (Xp, Yp), and the display coordinates of the password icon A5 are assumed to be (0, 0). The eye-gaze input error corresponds to the distance r between the coordinates (Xp, Yp) of the estimated gaze point P' and the display coordinates (0, 0) of the password icon A5.

[0070] In step S1007, the system control unit 102 determines whether the first (initial) component of the password is currently being input based on the password information (password information input by the user) stored in the memory 106. If it is determined that the first component of the password is currently being input, the process proceeds to step S1008. If it is determined that the first component of the password is not currently being input, the process proceeds to step S1009.

[0071] In step S1008, the system control unit 102 controls the line-of-sight detection unit 108. Based on the positional relationship between the estimated gaze point and the selected icon (the display position of the component being input), the above-mentioned calibration work is performed. In this way, the gaze detection unit 108 acquires calibration information. In this embodiment, the calibration information has two major components (elements).

[0072] The first component is an offset component that occurs when the HMD 100 is worn or removed. The offset component is the amount of deviation in the eye position relative to the HMD 100, based on when the HMD 100 is worn normally. The offset component can be obtained by a calibration operation that is performed when the user looks at an index at the center or periphery of the display surface.

[0073] The second component is the pupil sensitivity component. The pupil sensitivity component is a correction coefficient for the amount of displacement of the detected eyeball rotation angle. The pupil sensitivity component is information that indicates the relationship between the user's viewing position and the eyeball rotation angle. The pupil sensitivity component is a value used when estimating the gaze point, and is a value that differs from person to person. The pupil sensitivity component can be obtained through a calibration process that is performed when the user looks at indices around the display surface.

[0074] In step S1008, if the offset component and pupil-sensitive component (both X-direction component and Y-direction component) in the calibration information, including the results of past calibration operations, have been acquired, the above-mentioned determination area is changed. That is, in such a case, the area for determining whether the user is looking at the password icon (determination area) is changed from the input auxiliary area B to the input determination area A. Note that the determination area may be maintained as the input auxiliary area B until the flowchart in FIG. 10 ends.

[0075] In step S1009, the system control unit 102 compares the "pupil sensitivity component of the user newly acquired by the processing of this flowchart" with the "pupil sensitivity component stored in the memory 106" to identify (specify) the individual user. Then, the system control unit 102 determines whether the user's calibration information has been registered (stored in the memory 106). If it is determined that the user's calibration information has been registered, the process proceeds to step S1010. If it is determined that the user's calibration information has not been registered, the process proceeds to step S1013.

[0076] Here, if the password icon selected by the user when entering the first password is the icon in the center of the display screen, only the offset component can be acquired (the pupil sensitivity component cannot be acquired). For this reason, since it is not possible to identify (specify) the individual, the system control unit 102 may determine that the user has not been registered. Furthermore, if the system control unit 102 has stored calibration information associated with the user currently entering passwords in the memory 106 in step S1013 described below, it may determine that the user's calibration information has been registered.

[0077] In step S1010, the system control unit 102 corrects the estimated gaze point based on the user's calibration information stored in the memory 106. That is, if it is determined that the calibration information of the user wearing the HMD 100 has been registered (stored in the memory 106), the estimated gaze point (gazing point data) is corrected based on the calibration information. This makes it possible to gradually improve the estimation accuracy of the estimated gaze point (estimated gaze point data) during the series of password input processes (during input of the second and subsequent components).

[0078] A method for gradually increasing the accuracy of the estimated gaze point will be described with reference to Fig. 16. Fig. 16 shows password icons A1 to A9 in a display image 1100. Also, the pupil sensitivity component that can be obtained from each password icon by a calibration operation is plotted along the x-axis. The arrows in Figure 16 indicate the order in which the passwords were set.

[0079] First, the user looks at the password icon A5 to input the number "5." At this time, since the password icon A5 is an index located in the center of the display surface, the calibration information that can be obtained includes only the offset component.

[0080] Next, the user looks at password icon A1 to input the number "1." Because password icon A1 is an index located on the periphery of the display surface, the calibration information that can be acquired includes a pupil sensitivity component and an offset component. At this time, input is possible in a state where the positional deviation of the gaze point caused by the offset component has been corrected by using the calibration information acquired when the user looked at password icon A5. Furthermore, when inputting the number "1," it is possible to acquire pupil sensitivity component Y1 and the offset component when the person looks at the upper part of the display surface.

[0081] Next, the user looks at password icon A8 to input the number "8." Because password icon A8 is an index located on the periphery of the display surface, it is an index from which the pupil sensitivity component Y1 when a person looks up and the pupil sensitivity component X1 and offset component when a person looks to the right can be obtained. At this time, by using the calibration information acquired by the user's gaze on password icons A5 and A1, input can be performed with the positional deviation of the gaze point caused by the pupil sensitivity component Y1 and the offset component corrected. In this way, when inputting password components, the accuracy of the estimated gaze point can be gradually improved by correcting the estimated gaze point (gaze point data) based on the calibration information acquired up to that point.

[0082] In step S1011, the system control unit 102 calculates the eye-gaze input error by the same process as in step S1006.

[0083] In step S1012, the system control unit 102 acquires calibration information through the same process as in step S1008.

[0084] In step S1013, the system control unit 102 associates the calibration information of the user acquired up to step S1013 with the user and stores (registers) it in the memory 106.

[0085] In step S1014, the system control unit 102 determines whether the gaze input error calculated up to step S1014 is within a preset threshold. If it is determined that the gaze input error is within the preset threshold, it is determined that the gaze point intended by the user has been estimated, and the process proceeds to step S1015. In this case, the system control unit 102 considers that the user's input of the component indicated by the selection icon has been completed. If it is determined that the gaze input error exceeds the threshold, the process proceeds to step S1010. In this case, correction of the estimated gaze point and acquisition of calibration information are repeated again. This makes it possible to acquire highly accurate calibration information.

[0086] In step S1015, the system control unit 102 determines whether the "number of characters in the password input value, which is the password information entered by the user up to that point" matches the "number of characters in the password information stored in memory 106." If it is determined that the number of characters in the two pieces of password information match, the process proceeds to step S1016. If it is determined that the number of characters in the two pieces of password information do not match, the process proceeds to step S1004.

[0087] In step S1016, the system control unit 102 determines whether the "number of characters of the password input value, which is the password information entered by the user up to that point" matches the "password information stored in memory 106." If it is determined that the two pieces of password information match, the process of this flowchart ends. If it is determined that the two pieces of password information do not match, the process proceeds to the process for when password entry fails (the process of the flowchart in FIG. 11).

[0088] (Processing when password entry fails) The process of re-entering a password when the user fails to enter it correctly will be described with reference to the flowchart in Fig. 11. The flowchart in Fig. 11 is a flowchart for re-entering a password when the user fails to enter it correctly. The basic process is the same as in the flowchart in Fig. 10, so only a brief explanation will be given. For steps that have the same numbers as in Fig. 10, the same process is basically performed in Fig. 10 and Fig. 11.

[0089] When the processes from step S1001 to step S1005 are completed, the process proceeds to step S1010. In step S1010, the system control unit 102 corrects the estimated gaze point (gazing point data) using the calibration information acquired when the input failed.

[0090] In step S1011, the system control unit 102 calculates the eye-gaze input error.

[0091] In step S1014, the system control unit 102 determines whether the gaze input error calculated in step S1011 is within a threshold value. If it is determined that the gaze input error is within the threshold value, the process proceeds to step S1015. If it is determined that the gaze input error exceeds the threshold value, the process proceeds to step S1012.

[0092] Once the calibration information is acquired and stored (registered) in memory 106 through the processes of steps S1012 and S1013, the process proceeds to step S1010. This series of processes performs calibration only for gaze points where the gaze input error exceeds the threshold value based on the calibration information acquired at the time of failure. This makes it possible to shorten the password input time.

[0093] If it is determined in step S1015 that the set number of passwords have been entered, the process proceeds to step S1016.

[0094] In step S1016, the system control unit 102 determines whether the "number of characters of the password input value, which is the password information entered by the user up to that point" matches the "password information stored in memory 106." If it is determined that the two pieces of password information do not match, the process returns to step S1001 of this flowchart. Therefore, using the calibration information acquired in this series of processes, the process for re-entering the password in the event of a failed password entry is performed again.

[0095] According to this embodiment, in a head-mounted display device having a gaze detection unit, it is possible to prevent the user from performing operations solely for calibration. This improves user convenience. Furthermore, since calibration is performed while entering a password using gaze, it is less likely to be bothersome to the user than when calibration is performed before entering a password using gaze. Therefore, authentication using gaze can be performed with higher accuracy without bothering the user.

[0096] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" should be changed to "If A is larger (higher) than B, proceed to step S3." "Proceed to step S1 if A is greater (higher) than B, and proceed to step S2 if A is less (lower) than B." Conversely, "Proceed to step S1 if A is greater (higher) than B, and proceed to step S2 if A is less (lower) than B" may be read as "Proceed to step S1 if A is greater (higher) than B, and proceed to step S2 if A is less (lower) than B." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater (higher; longer; more) than A," and "less than or equal to A" may be read as "less (lower; shorter; fewer) than A." Furthermore, "greater (higher; longer; more) than A" may be read as "greater than or equal to A," and "less (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0097] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0098] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0099] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0100] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0101] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) a display control means for displaying a plurality of elements constituting a preset password on a display surface of a display device; an estimation means for acquiring an estimated gaze point that estimates the position at which the user is looking; an information acquisition means for acquiring calibration information used to correct the estimated gaze point based on a positional relationship between the estimated gaze point and each component; a correction means for correcting the estimated gaze point based on the calibration information; an authentication means for determining whether the user has entered the password by determining the component the user is looking at based on the estimated gaze point; A display control device comprising: (Configuration 2) When the estimated gaze point is located in a first range including a range in which a first component of the plurality of components constituting the password is displayed, the authentication means determining that a first component is being input; 2. The display control device according to configuration 1, (Configuration 3) If the estimated point of gaze is not corrected by the correction means, the authentication means determines that the user is inputting the first component when the estimated point of gaze is located in a second range that includes the first range and is wider than the first range. 3. The display control device according to configuration 2. (Configuration 4) the display control means displays a display item indicating the second range on the display surface if the estimated gaze point has not been corrected by the correction means. 4. The display control device according to configuration 3. (Configuration 5) when it is determined by the authentication means that the user is inputting the first component, the display control means displays a display item that highlights the first component on the display surface. 5. The display control device according to any one of configurations 2 to 4. (Configuration 6) When the authentication means determines that the user is inputting the first component, the information acquisition means If a difference in distance between the first constituent element and the estimated gaze point is greater than a specific threshold, the calibration information is acquired based on a positional relationship between the estimated gaze point and the first constituent element. 6. The display control device according to any one of configurations 2 to 5. (Configuration 7) the display control means does not display the estimated gaze point when the plurality of components are displayed on the display surface. 7. The display control device according to any one of configurations 1 to 6. (Configuration 8) the calibration information includes at least one of an offset component relating to an eye position relative to the display device and a pupil sensitivity component indicating a relationship between a viewing position of the user and a rotation angle of the eyeball; 8. The display control device according to any one of configurations 1 to 7. (Configuration 9) the information acquisition means acquires the calibration information while the user is inputting a first component of the plurality of components. 9. The display control device according to any one of configurations 1 to 8. (Configuration 10) further comprising a storage means for storing said calibration information for at least one person; The correction means comparing the calibration information stored in the storage means with the calibration information acquired by the information acquisition means to determine whether or not the calibration information corresponding to the user is stored in the storage means; In a first case where it is determined that the calibration information corresponding to the user is stored in the storage means, the estimated gaze point is corrected based on the calibration information stored in the storage means. 10. The display control device according to any one of configurations 1 to 9. (Configuration 11) In the first case, the information acquisition means acquires the calibration information based on the positional relationship between the estimated gaze point corrected by the correction means and the component the user is looking at. and storing the acquired calibration information in the storage means in association with the user. 11. The display control device according to configuration 10. (Configuration 12) When the user inputs the second or subsequent component of the plurality of components, the correction means corrects the estimated gaze point based on the calibration information of the user stored in the storage means; the authentication means determines whether the user has input the second or subsequent component elements based on the corrected estimated gaze point. 12. The display control device according to configuration 10 or 11. (method) a display control step of displaying a plurality of elements constituting a preset password on a display surface of a display device; an estimation step of acquiring an estimated gaze point by estimating a position where the user is looking; an information acquisition step of acquiring calibration information used to correct the estimated gaze point based on a positional relationship between the estimated gaze point and each component; a correction step of correcting the estimated gaze point based on the calibration information; an authentication step of determining whether the user has entered the password by determining the component the user is looking at based on the estimated gaze point; A control method comprising: (program) 13. A program for causing a computer to function as each means of the display control device according to any one of configurations 1 to 12. [Explanation of symbols]

[0102] 100: HMD, 102: System control unit, 103: Image processing unit, 109: Display section

Claims

1. a display control means for displaying a plurality of elements constituting a preset password on a display surface of a display device; an estimation means for acquiring an estimated gaze point that estimates the position at which the user is looking; an information acquisition means for acquiring calibration information used to correct the estimated gaze point based on a positional relationship between the estimated gaze point and each component; a correction means for correcting the estimated gaze point based on the calibration information; an authentication means for determining whether the user has entered the password by determining the component the user is looking at based on the estimated gaze point; A display control device comprising:

2. the authentication means determines that the user is currently inputting a first component when the estimated gaze point is located in a first range including a range in which a first component of the plurality of components constituting the password is displayed; 2. The display control device according to claim 1.

3. If the estimated point of gaze is not corrected by the correction means, the authentication means determines that the user is inputting the first component when the estimated point of gaze is located in a second range that includes the first range and is wider than the first range.

3. The display control device according to claim 2.

4. the display control means displays a display item indicating the second range on the display surface if the estimated gaze point has not been corrected by the correction means.

4. The display control device according to claim 3.

5. When the authentication means determines that the user is inputting the first component, the display control means displays a display item that highlights the first component on the display surface.

3. The display control device according to claim 2.

6. When the authentication means determines that the user is inputting the first component, the information acquisition means If a difference in distance between the first component and the estimated gaze point is greater than a specific threshold, the calibration information is acquired based on a positional relationship between the estimated gaze point and the first component.

3. The display control device according to claim 2.

7. the display control means does not display the estimated gaze point when the plurality of components are displayed on the display surface.

2. The display control device according to claim 1.

8. the calibration information includes at least one of an offset component relating to an eye position relative to the display device and a pupil sensitivity component indicating a relationship between a viewing position of the user and a rotation angle of the eyeball; 2. The display control device according to claim 1.

9. the information acquisition means acquires the calibration information while the user is inputting a first component of the plurality of components.

2. The display control device according to claim 1.

10. further comprising storage means for storing said calibration information for at least one person; The correction means comparing the calibration information stored in the storage means with the calibration information acquired by the information acquisition means to determine whether or not the calibration information corresponding to the user is stored in the storage means; In a first case where it is determined that the calibration information corresponding to the user is stored in the storage means, the estimated gaze point is corrected based on the calibration information stored in the storage means.

2. The display control device according to claim 1.

11. In the first case, the information acquisition means acquires the calibration information based on a positional relationship between the estimated gaze point corrected by the correction means and a component that the user is looking at, and stores the acquired calibration information in the storage means in association with the user. The display control device according to claim 10 .

12. When the user inputs the second or subsequent component of the plurality of components, the correction means corrects the estimated gaze point based on the calibration information of the user stored in the storage means; the authentication means determines whether the user has input the second or subsequent component elements based on the corrected estimated gaze point. The display control device according to claim 10 .

13. a display control step of displaying a plurality of elements constituting a preset password on a display surface of a display device; an estimation step of acquiring an estimated gaze point by estimating a position where the user is looking; an information acquisition step of acquiring calibration information used to correct the estimated gaze point based on a positional relationship between the estimated gaze point and each component; a correction step of correcting the estimated gaze point based on the calibration information; an authentication step of determining whether the user has entered the password by determining the component the user is looking at based on the estimated gaze point; A control method comprising:

14. A program for causing a computer to function as each of the means of the display control device according to any one of claims 1 to 12.

Images