Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises: a display unit; a first acquisition means for acquiring an eye image of a subject; a line-of-sight estimation means for performing line-of-sight estimation for estimating the line of sight of the subject by using the eye image; and a control means for performing calibration related to the line-of-sight estimation. The control means controls the display unit so as to display an instruction image related to the calibration, and the first acquisition means acquires at least a part of the eye image of the subject, which is captured when the instruction image is displayed on the display unit, as a first eye image to be used also for biometric authentication of the subject.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] In this type of information processing device, gaze detection as described in, for example, Japanese Patent Laid-Open No. 2003-144999 may be performed.

[0003] Japanese Patent Application Laid-Open No. 2015-013031

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve the technology related to the prior art documents mentioned above.

[0005] One aspect of an information processing device includes a display unit, a first acquisition means for acquiring an eye image of a subject, a gaze estimation means for performing gaze estimation to estimate the gaze of the subject using the eye image, and a control means for performing calibration related to the gaze estimation, wherein the control means controls the display unit to display an instruction image related to the calibration, and the first acquisition means acquires at least a portion of the eye image of the subject, captured while the instruction image is displayed on the display unit, as a first eye image that is also used for biometric authentication of the subject.

[0006] One aspect of the information processing method is an information processing method in an information processing device having a display unit, which acquires an eye image of a subject, performs gaze estimation to estimate the subject's gaze using the eye image, performs calibration related to the gaze estimation, controls the display unit to display an instruction image related to the calibration, and acquires at least a portion of the eye image of the subject captured while the instruction image is displayed on the display unit as a first eye image that is also used for biometric authentication of the subject.

[0007] One aspect of the recording medium has a computer program recorded thereon for causing a computer having a display unit to execute an information processing method that acquires an eye image of a subject, performs gaze estimation to estimate the subject's gaze using the eye image, performs calibration related to the gaze estimation, controls the display unit to display an instruction image related to the calibration, and acquires at least a portion of the subject's eye image captured while the instruction image is being displayed on the display unit as a first eye image that is also used for biometric authentication of the subject.

[0008] 1 is a block diagram showing an example of an information processing device according to an embodiment. FIG. 2 is a flowchart showing an example of an operation related to calibration for gaze estimation. FIG. 3 is a block diagram showing another example of an information processing device according to an embodiment. FIG. 4 is a block diagram showing an example of a configuration of a calculation device according to an embodiment. FIG. 5 is a block diagram showing another example of an information processing device according to an embodiment. FIG. 6 is a block diagram showing another example of an information processing device according to an embodiment showing an example of an information processing system according to an embodiment. FIG. 7 is a diagram showing an example of a head-mounted display. FIG. 8 is a diagram showing an example of an instruction image for guiding the gaze of a user. FIG. 9 is a flowchart showing another example of an operation related to calibration for gaze estimation. FIG. 10 is a block diagram showing another example of an information processing device according to an embodiment. FIG. 11 is a diagram showing an example of a positional relationship between the user's eyes and a camera. FIG. 12 is a block diagram showing another example of an information processing device according to an embodiment. FIG. 12 is a diagram showing an example of a head-mounted display when a user is wearing glasses. FIG. 13 is a diagram showing an example of an operation of the information processing device when a user is wearing glasses.

[0009] First Embodiment A first embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 1 and Fig. 2. In the following, the first embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 10.

[0010] 1 , the information processing device 10 includes a display unit 11, an image acquisition unit 12, a gaze estimation unit 13, and a control unit 14. The display unit 11 may be disposed in a housing of the information processing device 10, or may have a housing different from the housing of the information processing device 10 (for example, a housing unique to the display unit 11).

[0011] The image acquisition unit 12 acquires an image of the eyes of the user of the information processing device 10. The image of the user's eyes may be generated by, for example, a camera arranged around the display unit 11 capturing an image of the user's eyes. The camera may be a device different from the information processing device 10, or may constitute a part of the information processing device 10. When a camera constitutes a part of the information processing device 10, the camera may constitute at least a part of the image acquisition unit 12. The number of cameras capturing an image of the user's eyes is not limited to one, and multiple cameras may be used.

[0012] The gaze estimation unit 13 performs gaze estimation to estimate the user's gaze (e.g., the user's gaze toward the display unit 11) using the eye image acquired by the image acquisition unit 12. The following method is an example of a gaze estimation method. It is assumed that the user's eyes are illuminated by a point light source emitting near-infrared light. The gaze estimation unit 13 may estimate the gaze based on the relative positional relationship between the position of near-infrared light reflected by the cornea (i.e., Purkinje image) included in the eye image and the position of the pupil included in the eye image. Alternatively, the gaze estimation unit 13 may estimate the gaze based on the relative positional relationship between the position of the inner corner or outer corner of the eye included in the eye image and the position of the pupil or iris included in the eye image. Alternatively, the gaze estimation unit 13 may estimate the gaze using a machine learning model that outputs the gaze direction when an eye image is input. The gaze estimation method is not limited to the above-described method, and various existing methods can be applied. The gaze estimation may also be referred to as "gaze measurement" or "eye tracking." The user is the target of gaze estimation. For this reason, a user may also be referred to as a "subject" or a "subject."

[0013] The control unit 14 performs calibration related to gaze estimation. Calibration may mean, for example, at least one of acquiring the user's eye characteristics and detecting the user's eye movement in order to accurately calculate the user's gaze point in gaze estimation. The eye characteristics may include, for example, eye size, eye color, eye shape, and whether the user wears glasses or contact lenses. For example, the control unit 14 controls the display unit 11 to display an instruction image related to the calibration. The instruction image may be an image containing content for guiding the user's gaze (e.g., at least one of a bright spot, an arrow, a symbol, a figure, and an instruction). Here, the image acquisition unit 12 acquires an image of the user's eye captured while the instruction image is displayed on the display unit 11. The eye image can be said to represent the state of the user's eyes when the user is gazing at a predetermined position on the display unit 11 (i.e., the position to which the user's gaze is guided by the instruction image). The control unit 14 may acquire the user's eye characteristics and / or detect the user's eye movement based on the eye image. The control unit 14 may change a setting value (e.g., a parameter value) related to gaze estimation based on the result of the calibration. Alternatively, the gaze estimation unit 13 may change a setting value related to gaze estimation based on the result of the calibration. The gaze estimation unit 13 may perform gaze estimation of the user using the setting value related to gaze estimation changed based on the result of the calibration.

[0014] In particular, in this embodiment, the image acquisition unit 12 acquires at least a portion of the user's eye image captured when the instruction image is displayed on the display unit 11 as an eye image that can also be used for biometric authentication of the user (e.g., iris authentication).

[0015] The operation of calibration for gaze estimation will be described with reference to the flowchart of FIG. 2 . In FIG. 2 , the control unit 14 controls the display unit 11 to display an instruction image (step S101). While the instruction image is displayed on the display unit 11, the image acquisition unit 12 acquires an image of the user's eyes (step S102). After the process of step S102, the process of step S101 may be performed again. That is, the processes of steps S101 and S102 may be performed repeatedly. In this case, the instruction image displayed in the current process of step S101 may be different from the instruction image displayed in the previous process of step S101. For example, in the previous process of step S101, an instruction image for guiding the user's gaze may be displayed at a certain position on the display unit 11. In the current process of step S101, an instruction image for guiding the user's gaze may be displayed at another position on the display unit 11 (i.e., a position different from the above-mentioned certain position). In particular, the image acquisition unit 12 acquires at least a part of the eye image acquired in the process of step S102 as an eye image that is also used for biometric authentication of the user.

[0016] The information processing device 10 may perform an information processing method in which it acquires an image of the user's eyes, performs gaze estimation to estimate the user's gaze using the eye image, performs calibration related to the gaze estimation, controls the display unit 11 to display an instruction image related to the calibration, and acquires at least a portion of the user's eye image captured while the instruction image is displayed on the display unit 11 as an eye image that is also used for biometric authentication of the user.

[0017] The information processing device 10 described above may be realized by a computer reading a computer program recorded on a recording medium. In this case, the recording medium may store a computer program for causing a computer having a display unit to execute an information processing method that acquires a user's eye image, performs gaze estimation to estimate the user's gaze using the eye image, performs calibration related to the gaze estimation, controls the display unit to display an instruction image related to the calibration, and acquires at least a portion of the user's eye image captured while the instruction image is displayed on the display unit as an eye image that is also used for user biometric authentication.

[0018] (Technical Effect) A gaze operation interface has been proposed as an operation interface for a computer (e.g., corresponding to an information processing device 10). In gaze operations using a gaze operation interface, for example, a gaze operation different from the one intended by the user may be selected. One cause of this is a discrepancy between the user's actual gaze point and the user's gaze point estimated by gaze estimation. In order to suppress this discrepancy, calibration related to gaze estimation is often performed.

[0019] In a computer with a gaze-operated interface, a camera that captures an image of a user's eyes for gaze estimation is placed in a position that does not obstruct the user's view of the computer's display screen. For this reason, when the user is looking at the display screen, the user's gaze and the optical axis of the camera are often different.

[0020] Incidentally, biometric authentication using an image of a user's eyes (e.g., iris authentication) may be required for login authentication, etc. However, the eye image generated by the camera capturing an image of the user's eyes may not be suitable for biometric authentication because the camera is not positioned with the intention of biometric authentication of the user.

[0021] The inventors of the present application have focused on the fact that the user's gaze is guided to various positions during calibration related to gaze estimation. In other words, due to the user's gaze being guided to various positions during calibration, it is expected that one or more eye images generated by the camera capturing images of the user's eyes will include an eye image suitable for biometric authentication.

[0022] Therefore, in the information processing device 10, the control unit 14 controls the display unit 11 to display an instruction image related to calibration, and the image acquisition unit 12 acquires at least a portion of the user's eye image when the instruction image is displayed on the display unit 11 as an eye image to be used for biometric authentication of the user. According to the information processing device 10, a portion of the eye image for calibration related to gaze estimation can also be used for biometric authentication. In other words, the information processing device 10 can efficiently acquire the user's eye image to be used for biometric authentication.

[0023] Note that, as the biometric authentication of the user, for example, in addition to iris authentication, at least one of face authentication, fingerprint authentication, palm print authentication, voice print authentication, and ear acoustic authentication may be performed. In other words, multimodal biometric authentication may be performed as the biometric authentication of the user.

[0024] Second Embodiment A second embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Figures 3 and 4. Hereinafter, the second embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, for the second embodiment, descriptions that overlap with the first embodiment will be omitted as appropriate.

[0025] 3 , the information processing device 100 includes a calculation device 110, a storage device 120, a communication device 130, an input device 140, and an output device 150. The calculation device 110, the storage device 120, the communication device 130, the input device 140, and the output device 150 may be connected via a data bus 160.

[0026] The computing device 110 may include a processor 110a. The computing device 110 may include other processors in addition to the processor 110a. That is, the computing device 110 may include one or more processors. The processor 110a may be a multi-core processor. When the computing device 110 includes a single processor 110a that is a multi-core processor, the computing device 110 can be said to logically include multiple processors.

[0027] The processor 110a may be, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a tensor processing unit (TPU), and a quantum processor.

[0028] The storage device 120 may include a memory 120a. The storage device 120 may include other memories in addition to the memory 120a. That is, the storage device 120 may include one or more memories. The memory 120a may be, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. Therefore, the storage device 120 may include the memory 120a as a non-transitory recording medium.

[0029] The storage device 120 can store desired data. A computer program 121 to be executed by the arithmetic device 110 may be stored in the memory 120a of the storage device 120. The storage device 120 may temporarily store data that is temporarily used by the arithmetic device 110 when the arithmetic device 110 is executing the computer program 121.

[0030] The computer program 121 may be recorded on a computer-readable, non-transitory recording medium. In this case, the computer program 121 may be stored in the memory 120a by reading the recording medium using a recording medium reading device (not shown) included in the information processing device 100. The recording medium may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program. The computer program 121 may be acquired (in other words, downloaded) from a device (not shown) external to the information processing device 100 via the communication device 130. The acquired computer program 121 may be stored in the memory 120a.

[0031] The communication device 130 may be capable of communicating with devices external to the information processing device 100. The communication device 130 may perform wired communication or wireless communication.

[0032] The input device 140 is a device capable of accepting information input to the information processing device 100 from outside. The input device 140 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by a user of the information processing device 100. The input device 140 may include a recording medium reading device that can read information recorded on a recording medium that is detachable from the information processing device 100, such as a USB (Universal Serial Bus) memory. Note that when information is input to the information processing device 100 via the communication device 130 (in other words, when the information processing device 100 acquires information via the communication device 130), the communication device 130 may function as an input device.

[0033] The output device 150 is a device capable of outputting information to the outside of the information processing device 100. The output device 150 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 150 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 150 may be capable of outputting information to a recording medium that is detachable from the information processing device 100, such as a USB memory. Note that when the information processing device 100 outputs information via the communication device 130, the communication device 130 may function as the output device.

[0034] The processor 110a of the arithmetic device 110 may, together with the memory 120a of the storage device 120 in which the computer program 121 is stored (in other words, together with the memory 120a and the computer program 121 stored in the memory 120a), execute the processing to be performed by the information processing device 100. For example, the processor 110a may execute the computer program 121 to realize, within the arithmetic device 110 (for example, within the processor 110a), a logical functional block for executing the processing to be performed by the information processing device 100.

[0035] 4, the arithmetic device 110 may include, as logically realized functional blocks or physically realized processing circuits, an image acquisition unit 111, a gaze estimation unit 112, and a control unit 113. Note that at least one of the image acquisition unit 111, the gaze estimation unit 112, and the control unit 113 may be realized in a form in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed.

[0036] Note that when the image acquisition unit 111, the gaze estimation unit 112, and the control unit 113 are realized as functional blocks, the image acquisition unit 111, the gaze estimation unit 112, and the control unit 113 may be realized by a single processor (for example, the processor 110a). Alternatively, the image acquisition unit 111, the gaze estimation unit 112, and the control unit 113 may be realized by different processors. Alternatively, parts of the image acquisition unit 111, the gaze estimation unit 112, and the control unit 113 may be realized by a single processor, and the remaining parts of the image acquisition unit 111, the gaze estimation unit 112, and the control unit 113 may be realized by one or more processors different from the single processor.

[0037] The output device 150 may include a display device 151. The display device 151 may be, for example, at least one of a liquid crystal display, an organic EL (Electro-Luminescence) display, and a head-mounted display. The "image acquisition unit 111," "gaze estimation unit 112," "control unit 113," and "display device 151" are components corresponding to the "image acquisition unit 12," "gaze estimation unit 13," "control unit 14," and "display unit 11" in the first embodiment described above, respectively.

[0038] The image acquisition unit 111 acquires an eye image of the user of the information processing device 100. The user's eye image may be generated, for example, by a camera arranged around the display device 151 capturing an image of the user's eyes. The camera may be a device different from the information processing device 100. In this case, the image acquisition unit 111 may acquire the user's eye image generated by the camera, for example, via the communication device 130. The camera may constitute a part of the information processing device 100. In this case, the image acquisition unit 111 may acquire the user's eye image generated by the camera, for example, via the data bus 160. The number of cameras capturing the user's eyes is not limited to one, and may be multiple. When there are multiple cameras capturing the user's eyes, there may be a camera capturing the user's left eye and a camera capturing the user's right eye. When there are multiple cameras capturing the user's eyes, there may be multiple cameras capturing the user's left eye, or multiple cameras capturing the user's right eye.

[0039] The gaze estimation unit 112 performs gaze estimation to estimate the user's gaze (for example, the user's gaze toward the display device 151) using the eye image acquired by the image acquisition unit 111. The control unit 113 performs calibration related to the gaze estimation. The gaze estimation unit 112 may perform gaze estimation of the user using setting values ​​related to gaze estimation that have been changed based on the results of the calibration.

[0040] In calibration for gaze estimation, the image acquisition unit 111 acquires at least a portion of the user's eye image captured when the instruction image is displayed on the display device 151 as an eye image that is also used for biometric authentication of the user (e.g., iris authentication).

[0041] (Technical effect) According to the information processing device 100 of the second embodiment, similar to the information processing device 10 of the first embodiment described above, it is possible to efficiently acquire an image of the user's eyes to be used for biometric authentication.

[0042] Third Embodiment A second embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to FIG. 5 . Hereinafter, a third embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100a. Note that, for the third embodiment, descriptions that overlap with the first and second embodiments will be omitted as appropriate. Furthermore, in the drawings, parts common to other embodiments are indicated by the same reference numerals.

[0043] 5, the information processing device 100a includes a calculation device 110, a storage device 120, a communication device 130, an input device 140, and an output device 150. The calculation device 110 may include an image acquisition unit 111, a gaze estimation unit 112, a control unit 113, and an authentication unit 114 as logically realized functional blocks or as physically realized processing circuits.

[0044] At least one of the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, and the authentication unit 114 may be realized in a format that combines logical functional blocks and physical processing circuits (i.e., hardware). When the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, and the authentication unit 114 are realized as functional blocks, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, and the authentication unit 114 may be realized by a single processor (e.g., processor 110a). Alternatively, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, and the authentication unit 114 may be realized by different processors. Alternatively, some of the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, and the authentication unit 114 may be realized by a single processor, and the remaining parts of the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, and the authentication unit 114 may be realized by one or more processors different from the single processor.

[0045] The authentication unit 114 performs biometric authentication of the user using at least a part of the eye image acquired by the image acquisition unit 111. The operation of the authentication unit 114 when performing iris authentication as an example of biometric authentication will be described. The authentication unit 114 may extract an iris image including an iris region, which is an area corresponding to the user's iris, from the eye image. The authentication unit 114 may calculate a matching score by comparing the extracted iris image with a pre-registered iris image. Here, the matching score may refer to a score indicating the degree of match between the extracted iris image and the pre-registered iris image. Note that the higher the degree of match between the extracted iris image and the pre-registered iris image, the higher the matching score may be.

[0046] Alternatively, the authentication unit 114 may extract features related to the extracted iris image. The authentication unit 114 may calculate a matching score by comparing the extracted features with pre-registered features. Here, the matching score may refer to a score indicating the degree of match between the features extracted from the iris image and the pre-registered features. Note that the higher the degree of match between the features extracted from the iris image and the pre-registered features, the higher the matching score may be.

[0047] The authentication unit 114 may compare the matching score with a predetermined threshold. If the matching score is equal to or greater than the predetermined threshold, the authentication unit 114 may determine that the user is a person associated with the pre-registered iris image or feature amount. In this case, it can be said that a person (e.g., identification information) corresponding to the user has been identified through biometric authentication. If the matching score is less than the predetermined threshold, the authentication unit 114 may determine that the user is not a person associated with the pre-registered iris image or feature amount. In this case, it can be said that a person corresponding to the user has not been identified through biometric authentication.

[0048] The authentication unit 114 may calculate a matching score for the user's left eye and a matching score for the user's right eye. However, the authentication unit 114 may calculate only one of the matching score for the left eye and the matching score for the right eye. In other words, the authentication unit 114 may calculate at least one of the matching score for the left eye and the matching score for the right eye.

[0049] The authentication unit 114 may calculate the matching score based on the degree of match between the iris image of the user's left eye and one pre-registered iris image and the degree of match between the iris image of the user's right eye and another pre-registered iris image. Alternatively, the authentication unit 114 may calculate the matching score based on the degree of match between a feature extracted from the iris image of the user's left eye and one pre-registered feature, and the degree of match between a feature extracted from the iris image of the user's right eye and another pre-registered feature.

[0050] When a person corresponding to the user is identified by biometric authentication, the control unit 113 may associate at least one of the results of calibration related to gaze estimation and the setting values ​​related to gaze estimation changed based on the results of the calibration with the identified person (e.g., identification information), and store them in the storage device 120. When a person corresponding to the user is identified by biometric authentication, the control unit 113 may acquire from the storage device 120 at least one of the results of calibration related to gaze estimation and the setting values ​​related to gaze estimation changed based on the results of the calibration, which are associated with the identified person (e.g., identification information).

[0051] (Technical Effect) In calibration related to gaze estimation, the image acquisition unit 111 acquires at least a part of an image of the user's eyes when an instruction image is displayed on the display device 151 as an image of the eyes that is also used for biometric authentication of the user.

[0052] As described above, in the information processing device 100a, at least one of the results of calibration related to gaze estimation and the setting values ​​related to gaze estimation changed based on the results of the calibration may be stored in the storage device 120 in association with a person identified by biometric authentication. Here, the eye image used for biometric authentication by the authentication unit 114 is acquired by the image acquisition unit 111 when calibration related to gaze estimation is performed. Therefore, the image processing unit 111 does not need to separately acquire the eye image used for biometric authentication. In other words, the user does not need to capture an image of the eyes for biometric authentication separately from calibration related to gaze estimation.

[0053] When calibration related to gaze estimation is being performed and a person corresponding to the user is identified by biometric authentication using the eye image acquired by the image acquisition unit 111, the control unit 113 may acquire at least one of the result of the calibration related to gaze estimation associated with the identified person and the setting value related to gaze estimation that has been changed based on the result of the calibration from the storage device 120. In this case, the control unit 113 may end the calibration, for example, when the person corresponding to the user is identified.

[0054] In this way, the information processing apparatus 100a according to the third embodiment can reduce the burden on the user of at least one of biometric authentication and calibration related to gaze estimation.

[0055] Fourth Embodiment A second embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to FIG. 6. Hereinafter, a fourth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, for the fourth embodiment, descriptions that overlap with the first to third embodiments will be omitted as appropriate. Furthermore, in the drawings, parts common to other embodiments are indicated by the same reference numerals.

[0056] The information processing device 100 according to the fourth embodiment may be configured to be able to communicate with an authentication device 200 shown in Fig. 6 via a communication device 130. The authentication device 200 may be, for example, a personal computer or a server device (e.g., a cloud server). The information processing device 100 and the authentication device 200 may constitute a single information processing system.

[0057] The arithmetic device 110 of the information processing device 100 may transmit the user's eye image (here, the eye image used for biometric authentication of the user) acquired by the image acquisition unit 111 to the authentication device 200 via the communication device 130. The authentication device 200 may use the eye image to perform iris authentication as an example of biometric authentication. For a specific example of iris authentication, see the third embodiment described above. The authentication device 200 may transmit the result of the biometric authentication to the information processing device 100.

[0058] When a person corresponding to a user is identified by biometric authentication, the control unit 113 of the computing device 110 may associate at least one of the results of calibration related to gaze estimation and the setting values ​​related to gaze estimation changed based on the results of the calibration with the identified person (e.g., identification information), and store them in the storage device 120. When a person corresponding to a user is identified by biometric authentication, the control unit 113 may acquire from the storage device 120 at least one of the results of calibration related to gaze estimation and the setting values ​​related to gaze estimation changed based on the results of the calibration, which are associated with the identified person (e.g., identification information).

[0059] (Technical effect) According to the information processing device 100 of the fourth embodiment, similar to the information processing device 100a of the third embodiment described above, it is possible to reduce the burden on the user of at least one of biometric authentication and calibration related to gaze estimation.

[0060] Fifth Embodiment A fifth embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to FIGS. 7 to 10. Hereinafter, the fifth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100b. Note that, for the fifth embodiment, descriptions that overlap with the first to fourth embodiments will be omitted as appropriate. Furthermore, in the drawings, parts common to other embodiments are denoted by the same reference numerals.

[0061] In FIG. 7 , the information processing device 100b includes a calculation device 110, a storage device 120, a communication device 130, an input device 140, an output device 150, and cameras 171 and 172. The calculation device 110, the storage device 120, the communication device 130, the input device 140, the output device 150, and the cameras 171 and 172 may be connected via a data bus 160. The cameras 171 and 172 may be cameras capable of capturing near-infrared light. In other words, the cameras 171 and 172 may be cameras equipped with image sensors sensitive to the near-infrared wavelength range. The calculation device 110 may not include the authentication unit 114. In this case, the information processing device 100b may be configured to communicate with an authentication device different from the information processing device 100b (e.g., the authentication device 200 according to the fourth embodiment) via the communication device 130. The user's biometric authentication may be performed by the authentication device.

[0062] The output device 150 may include a head-mounted display (HMD) 152. The head-mounted display 152 will now be described with reference to FIG. 8 . In this embodiment, an immersive head-mounted display (so-called immersive virtual reality (VR) goggles) is given as an example of the head-mounted display 152. Note that the head-mounted display 152 is not limited to an immersive head-mounted display, and may be, for example, a video see-through head-mounted display or an optical see-through head-mounted display (so-called augmented reality (AR) glasses).

[0063] The head-mounted display 152 may be a standalone head-mounted display. In this case, the arithmetic device 110, the storage device 120, the communication device 130, the input device 140, and the cameras 171 and 172 may constitute a part of the head-mounted display 152. The head-mounted display 152 may be a PC-connected head-mounted display that is connected to a personal computer, which is an example of the information processing device 100b. The head-mounted display 152 may be a hybrid head-mounted display (i.e., a head-mounted display that can operate as both a standalone type and a PC-connected type).

[0064] 8 , the head-mounted display 152 includes a display 1521, a lens 1522L for the left eye, a lens 1522R for the right eye, and a plurality of light sources 1523#1, 1523#2, 1523#3, and 1523#4 that emit near-infrared light. The light sources 1523#1 and 1523#2 may be arranged around the lens 1522L. The light sources 1523#3 and 1523#4 may be arranged around the lens 1522R. The lenses 1522L and 1522R may be referred to as VR lenses. The camera 171 may be arranged in the head-mounted display 152 so as to be able to capture an image of the user's left eye. The camera 172 may be arranged in the head-mounted display 152 so as to be able to capture an image of the user's right eye. The cameras 171 and 172 may constitute part of the head-mounted display 152. That is, the head mounted display 152 may have cameras 171 and 172. The head mounted display 152 may have a display for the left eye and a display for the right eye instead of the display 1521. The number of light sources emitting near-infrared light is not limited to four, and may be five or more, or three or less.

[0065] A specific example of calibration related to gaze estimation will be described with reference to FIG. 9 . FIG. 9 shows an example of an instruction image for guiding the user's gaze. In FIG. 9 , the background of the instruction image is set to a white background for convenience. However, the background of an actual instruction image may be a background with a relatively low brightness value, such as black or gray. In FIG. 9 , the instruction image (specifically, images Img#1 to Img#9) includes points for guiding the user's gaze. However, the instruction image is not limited to points, and may include at least one of, for example, an arrow, a symbol, a figure, and an instruction.

[0066] In calibration related to gaze estimation, the control unit 113 may first control the output device 150 to display the image Img#1 shown in FIG. 9A on the head-mounted display 152 (specifically, the display 1521). The control unit 113 may control the output device 150 to display, for example, text saying "Look at the point on the screen" superimposed on the image Img#1. The control unit 113 may control the output device 150 to output, for example, a sound saying "Look at the point on the screen." For example, the control unit 113 may control the output device 150 to display text saying "Look at the point on the screen that will be displayed now" and / or to output a sound saying "Look at the point on the screen that will be displayed now" before the image Img#1 is displayed.

[0067] While image Img#1 is displayed (in other words, while the user is looking at image Img#1 displayed on display 1521), camera 171 may capture the user's left eye, and camera 172 may capture the user's right eye. Eye image EIL#1 may be generated by camera 171 capturing the user's left eye. Eye image EIR#1 may be generated by camera 172 capturing the user's right eye. Image acquisition unit 111 may acquire eye images EIL#1 and EIR#1. Note that eye images EIL#1 and EIR#1 may be at least temporarily stored in storage device 120. Note that when cameras 171 and 172 each capture the user's eyes, at least some of the light sources among multiple light sources 1523#1, 1523#2, and 1523#4 may emit near-infrared light (in other words, at least some of the light sources may illuminate the user with near-infrared light).

[0068] Thereafter, the control unit 113 may control the output device 150 to display the image Img#2 shown in FIG. 9B on the head-mounted display 152 instead of the image Img#1. While the image Img#2 is being displayed (in other words, while the user is looking at the image Img#2 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate an eye image EIL#2. The camera 172 may capture an image of the user's right eye to generate an eye image EIR#2. The image acquisition unit 111 may acquire the eye images EIL#2 and EIR#2. The eye images EIL#2 and EIR#2 may be at least temporarily stored in the storage device 120.

[0069] Here, when image Img#1 is displayed on head-mounted display 152, the point at which the user should look (in other words, the point at which the user should gaze) is displayed near the left edge of the screen. When the user looks at a point included in image Img#1, the user's left eye is expected to face toward camera 171. Therefore, eye image EIL#1 is expected to include most of the iris of the user's left eye.

[0070] When image Img#2 is displayed on head-mounted display 152, the point at which the user should look (in other words, the point at which the user should gaze) is displayed near the right edge of the screen. When the user looks at a point included in image Img#2, the user's right eye is expected to face toward camera 172. Therefore, eye image EIRL#2 is expected to include most of the iris of the user's right eye.

[0071] The image acquisition unit 111 may also acquire the eye images EIL#1 and EIR#2 as eye images to be used for biometric authentication of the user (e.g., iris authentication). The image acquisition unit 111 may transmit the eye images EIL#1 and EIR#2 to the authentication unit 114. The authentication unit 114 may perform biometric authentication of the user using the eye images EIL#1 and EIR#2.

[0072] After the image acquisition unit 111 acquires the eye images EIL#2 and EIR#2, the control unit 113 may control the output device 150 to display the image Img#3 shown in FIG. 9C on the head-mounted display 152 instead of the image Img#2. While the image Img#3 is being displayed (in other words, while the user is looking at the image Img#3 displayed on the display 1521), the camera 171 may capture the user's left eye, and the camera 172 may capture the user's right eye. The camera 171 may capture the user's left eye to generate the eye image EIL#3. The camera 172 may capture the user's right eye to generate the eye image EIR#3. The image acquisition unit 111 may acquire the eye images EIL#3 and EIR#3. The eye images EIL#3 and EIR#3 may be at least temporarily stored in the storage device 120.

[0073] Thereafter, the control unit 113 may control the output device 150 to display image Img#4 shown in FIG. 9( d) on the head-mounted display 152 instead of image Img#3. While image Img#4 is being displayed (in other words, while the user is looking at image Img#4 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate eye image EIL#4. The camera 172 may capture an image of the user's right eye to generate eye image EIR#4. The image acquisition unit 111 may acquire eye images EIL#4 and EIR#4. The eye images EIL#4 and EIR#4 may be at least temporarily stored in the storage device 120.

[0074] Thereafter, the control unit 113 may control the output device 150 to display the image Img#5 shown in FIG. 9E on the head-mounted display 152 instead of the image Img#4. While the image Img#5 is being displayed (in other words, while the user is looking at the image Img#5 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate an eye image EIL#5. The camera 172 may capture an image of the user's right eye to generate an eye image EIR#5. The image acquisition unit 111 may acquire the eye images EIL#5 and EIR#5. The eye images EIL#5 and EIR#5 may be at least temporarily stored in the storage device 120.

[0075] Thereafter, the control unit 113 may control the output device 150 to display image Img#6 shown in FIG. 9( f) on the head-mounted display 152 instead of image Img#5. While image Img#6 is being displayed (in other words, while the user is looking at image Img#6 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate eye image EIL#6. The camera 172 may capture an image of the user's right eye to generate eye image EIR#6. The image acquisition unit 111 may acquire eye images EIL#6 and EIR#6. The eye images EIL#6 and EIR#6 may be at least temporarily stored in the storage device 120.

[0076] Thereafter, the control unit 113 may control the output device 150 to display image Img#7 shown in FIG. 9( g) on ​​the head-mounted display 152 instead of image Img#6. While image Img#7 is being displayed (in other words, while the user is looking at image Img#7 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate eye image EIL#7. The camera 172 may capture an image of the user's right eye to generate eye image EIR#7. The image acquisition unit 111 may acquire eye images EIL#7 and EIR#7. The eye images EIL#7 and EIR#7 may be at least temporarily stored in the storage device 120.

[0077] Thereafter, the control unit 113 may control the output device 150 to display image Img#8 shown in FIG. 9( h) on the head-mounted display 152 instead of image Img#7. While image Img#8 is being displayed (in other words, while the user is looking at image Img#8 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate eye image EIL#8. The camera 172 may capture an image of the user's right eye to generate eye image EIR#8. The image acquisition unit 111 may acquire eye images EIL#8 and EIR#8. The eye images EIL#8 and EIR#8 may be at least temporarily stored in the storage device 120.

[0078] Thereafter, the control unit 113 may control the output device 150 to display image Img#9 shown in FIG. 9( i) on the head-mounted display 152 instead of image Img#8. While image Img#9 is being displayed (in other words, while the user is looking at image Img#9 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate eye image EIL#9. The camera 172 may capture an image of the user's right eye to generate eye image EIR#9. The image acquisition unit 111 may acquire eye images EIL#9 and EIR#9. The eye images EIL#9 and EIR#9 may be at least temporarily stored in the storage device 120.

[0079] The control unit 113 may, for example, acquire the characteristics of the user's eyes and / or detect the eye movement of the user's eyes based on the eye images EIL#1 to EIL#9 and the eye images EIR#1 to EIR#9. For example, the control unit 113 may calculate a calibration value for calibrating a setting value (e.g., a parameter value) related to gaze estimation based on at least one of the characteristics of the user's eyes and the eye movement of the user's eyes. The control unit 113 may output calibration information indicating the calculated calibration value as a result of the calibration.

[0080] For example, when a person corresponding to the user is identified by biometric authentication using the eye images EIL#1 and EIR#2, the control unit 113 may associate the calibration information with the identified person (e.g., identification information) and store it in the storage device 120. For example, when a person corresponding to the user is not identified by biometric authentication using the eye images EIL#1 and EIR#2, the control unit 113 may associate the calibration information with new identification information for identifying the user and store it in the storage device 120.

[0081] For example, when a person corresponding to a user is identified by biometric authentication using eye images EIL#1 and EIR#2, and calibration information (i.e., past calibration information) linked to the identified person (e.g., identification information) is stored in the storage device 120, the control unit 113 may acquire the calibration information from the storage device 120 based on the identified person (e.g., identification information). In this case, the control unit 113 may not control the output device 150 to display images Img#3 to Img#9. In other words, images Img#3 to Img#9 may not be displayed.

[0082] In the calibration related to gaze estimation, the control unit 113 may control the output device 150 to display, for example, one image including multiple points that the user should look at, instead of sequentially displaying the images Img#1 to Img#9. In this case, the control unit 113 may make at least one of the color, brightness, size, and shape of the point that the user should currently look at different from the other points.

[0083] For example, changing from image Img#1 to image Img#2 can be considered as changing the position of a point included in the image. Therefore, in calibration related to gaze estimation, instead of controlling the output device 150 to display an instruction image (e.g., images Img#1 to Img#9), the control unit 113 may, for example, transmit to the output device 150 coordinate information indicating the coordinates of a point displayed to guide the user's gaze, and control the output device 150 to display the point at the coordinate indicated by the coordinate information.

[0084] The operation of information processing device 100b will be described with reference to the flowchart of Fig. 10 . In Fig. 10 , control unit 113 may control output device 150 to display a first instruction image (e.g., image Img#1) on head-mounted display 152 (specifically, display 1521) (step S201). While the first instruction image is displayed, camera 171 may capture an image of the user's left eye, and camera 172 may capture an image of the user's right eye. Image acquisition unit 111 may acquire first eye images (e.g., eye images EIL#1 and EIR#1) generated by cameras 171 and 172 capturing the images of the user's left eye and right eye, respectively (step S202).

[0085] Next, control unit 113 may control output device 150 to display a second instruction image (e.g., image Img#2) on head-mounted display 152 instead of the first instruction image (step S203). While the second instruction image is being displayed, camera 171 may capture an image of the user's left eye, and camera 172 may capture an image of the user's right eye. Image acquisition unit 111 may acquire second eye images (e.g., eye images EIL#2 and EIR#2) generated by cameras 171 and 172 capturing the images of the user's left eye and right eye, respectively (step S204).

[0086] The image acquisition unit 111 may transmit at least a portion of the first eye image and the second eye image (e.g., eye images EIL#1 and EIR#2) to the authentication unit 114. The authentication unit 114 may perform biometric authentication of the user (e.g., iris authentication) using the first eye image and the second eye image (step S205). Note that the authentication unit 114 may perform biometric authentication of the user using at least a portion of the first eye image (e.g., eye image EIL#1) after the process of step S202. Furthermore, the authentication unit 114 may perform biometric authentication of the user using at least a portion of the second eye image (e.g., eye image EIR#2) after the process of step S204. In this case, if a person corresponding to the user is identified through biometric authentication using at least a portion of the first eye image, the authentication unit 114 does not need to perform biometric authentication using at least a portion of the second eye image.

[0087] The control unit 113 may determine whether the biometric authentication has been successful (step S206). Here, if a person corresponding to the user is identified through the biometric authentication, the control unit 113 may determine that the biometric authentication has been successful. On the other hand, if a person corresponding to the user is not identified through the biometric authentication, the control unit 113 may determine that the biometric authentication has not been successful (in other words, that the biometric authentication has failed).

[0088] If it is determined in the processing of step S206 that the biometric authentication is successful (step S206: Yes), the control unit 113 may determine whether or not past calibration information related to the user exists (step S207). Here, if calibration information linked to the person corresponding to the user identified by biometric authentication is stored in the storage device 120, the control unit 113 may determine that past calibration information related to the user exists. On the other hand, if calibration information linked to the person corresponding to the user identified by biometric authentication is not stored in the storage device 120, the control unit 113 may determine that past calibration information related to the user does not exist.

[0089] If it is determined in the processing of step S207 that past calibration information related to the user exists (step S207: Yes), the control unit 113 may acquire calibration information (i.e., past calibration information) linked to the person identified by biometric authentication and corresponding to the user, based on the person's (e.g., identification information) from the storage device 120 (step S208). In this case, the control unit 113 may terminate the calibration related to gaze estimation. Note that in the processing of step S208, the control unit 113 may acquire setting values ​​related to gaze estimation that have been changed based on the results of the past calibration, in addition to or instead of the past calibration information.

[0090] If it is determined in the process of step S206 that biometric authentication has not been successful (step S206: No), or if it is determined in the process of step S207 that past calibration information related to the user does not exist (step S207: No), the control unit 113 may continue calibration related to gaze estimation (step S209). For example, the control unit 113 may control the output device 150 to display a third instruction image (e.g., images Img#3 to Img#9) on the head-mounted display 152 instead of the second instruction image. While the third instruction image is being displayed, the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The image acquisition unit 111 may acquire third eye images (e.g., eye images EIL#3 to EIL#9 and eye images EIR#3 to EIR#9) generated by cameras 171 and 172 capturing images of the user's left and right eyes, respectively. (Technical Effect) When a user wears a head-mounted display (e.g., head-mounted display 152), the relative positional relationship between the user's eyes and the cameras (e.g., cameras 171 and 172) arranged on the head-mounted display is fixed. In addition, the cameras are arranged in positions that do not obstruct the user's view of the display screen. For this reason, it is difficult to use the eye images generated by the cameras capturing images of the user's eyes for biometric authentication of the user.

[0091] In contrast to this, in the present embodiment, in calibration related to gaze estimation, the gaze of the user is guided so that the iris of the user faces the camera (specifically, so that the iris of the user's left eye faces the camera 171, and so that the iris of the user's right eye faces the camera 172). Therefore, in the present embodiment, when calibration is being performed, the cameras 171 and 172 capture images of the user's eyes, thereby making it possible to generate eye images (e.g., eye images EIL#1 and EIR#2) suitable for biometric authentication of the user.

[0092] As described above, when calibration is being performed, if a person corresponding to the user is identified by biometric authentication using the eye image acquired by the image acquisition unit 111 and calibration information (i.e., past calibration information) associated with the identified person exists, the control unit 113 may acquire the calibration information and terminate the calibration, thereby reducing the burden on the user.

[0093] Sixth Embodiment A sixth embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to FIGS. 11 and 12. Hereinafter, the sixth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100c. Note that, for the sixth embodiment, descriptions that overlap with the first to fifth embodiments will be omitted as appropriate. Furthermore, in the drawings, parts common to other embodiments are denoted by the same reference numerals.

[0094] In Figure 11, the calculation device 110 of the information processing device 100c may have an image acquisition unit 111, a gaze estimation unit 112, a control unit 113, an authentication unit 114, and a position estimation unit 115 as logically realized functional blocks or as physically realized processing circuits.

[0095] At least one of the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the position estimation unit 115 may be realized in a format that combines logical functional blocks and physical processing circuits (i.e., hardware). When the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the position estimation unit 115 are realized as functional blocks, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the position estimation unit 115 may be realized by a single processor (e.g., processor 110a). Alternatively, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the position estimation unit 115 may each be realized by different processors. Alternatively, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and part of the position estimation unit 115 may be realized by a single processor, and the remaining parts of the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the position estimation unit 115 may be realized by one or more processors different from the single processor.

[0096] However, different models of head-mounted displays may have different positions of cameras (e.g., cameras 171 and 172) arranged on the head-mounted displays (e.g., head-mounted display 152). In this case, if the position of a point for guiding the user's gaze is fixed in order to orient the iris of the user's eye toward the camera, it may be difficult to acquire an eye image suitable for biometric authentication of the user when calibration related to gaze estimation is performed.

[0097] The shape of the user's eyes included in the eye image changes depending on the relative positional relationship between the user's eyes and the camera. Here, an explanation will be provided with reference to FIG. 12. When the camera is placed at position P#1 in FIG. 12(a), an eye image such as that shown in FIG. 12(b) may be generated. When the camera is placed at position P#2 in FIG. 12(a), an eye image such as that shown in FIG. 12(c) may be generated. When the camera is placed at position P#3 in FIG. 12(a), an eye image such as that shown in FIG. 12(d) may be generated.

[0098] When calibration related to gaze estimation is performed, the control unit 113 may first control the output device 150 to display, for example, image Img#3 shown in FIG. 9C on the head-mounted display 152 (specifically, the display 1521). While image Img#3 is being displayed (in other words, while the user is looking at image Img#3 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate an eye image EIL#3a. The camera 172 may capture an image of the user's right eye to generate an eye image EIR#3a. The image acquisition unit 111 may acquire the eye images EIL#3a and EIR#3a. The eye images EIL#3a and EIR#3a may be at least temporarily stored in the storage device 120.

[0099] The position estimation unit 115 may use the eye images EIL#3a and EIR#3a to estimate the positions of the cameras 171 and 172 on the head-mounted display 152. Thereafter, the control unit 113 may control the output device 150 to display, instead of the image Img#3, an image (e.g., image Img#1 in FIG. 9A ) on the head-mounted display 152 that guides the user's gaze so that the iris of the user's left eye faces the camera 171. In this case, the control unit 113 may determine the positions of points included in the displayed image based on the estimated position of the camera 171.

[0100] Thereafter, the control unit 113 may control the output device 150 to display, on the head-mounted display 152, an image (e.g., image Img#2 in FIG. 9B ) that guides the user's gaze so that the iris of the user's right eye faces the camera 172, instead of the image that guides the user's gaze so that the iris of the user's left eye faces the camera 171. In this case, the control unit 113 may determine the positions of points included in the displayed image based on the estimated position of the camera 172.

[0101] (Technical Effect) According to the information processing device 100c according to the sixth embodiment, when calibration related to gaze estimation is being performed, the user's gaze can be appropriately guided so that the iris of the user's eye faces toward the camera.

[0102] Seventh Embodiment A seventh embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to FIG. 13 . Hereinafter, the seventh embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 100d. Note that, for the seventh embodiment, descriptions that overlap with the first to sixth embodiments will be omitted as appropriate. Furthermore, in the drawings, parts common to other embodiments are indicated by the same reference numerals.

[0103] In Figure 13, the calculation device 110 of the information processing device 100d may have an image acquisition unit 111, a gaze estimation unit 112, a control unit 113, an authentication unit 114, and an information acquisition unit 116 as logically realized functional blocks or as physically realized processing circuits.

[0104] At least one of the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the information acquisition unit 116 may be realized in a format that combines logical functional blocks and physical processing circuits (i.e., hardware). When the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the information acquisition unit 116 are realized as functional blocks, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the information acquisition unit 116 may be realized by a single processor (e.g., processor 110a). Alternatively, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the information acquisition unit 116 may each be realized by a different processor. Alternatively, the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and part of the information acquisition unit 116 may be realized by a single processor, and the remaining parts of the image acquisition unit 111, the gaze estimation unit 112, the control unit 113, the authentication unit 114, and the information acquisition unit 116 may be realized by one or more processors different from the single processor.

[0105] The information acquisition unit 116 may acquire camera position information indicating the positions of each of the cameras 171 and 172 arranged on the head-mounted display 152 from a device external to the information processing device 100d (e.g., a cloud server) via the communication device 130.

[0106] When calibration related to gaze estimation is performed, the control unit 113 may first control the output device 150 to display, on the head-mounted display 152, an image (for example, image Img#1 in FIG. 9A ) that guides the user's gaze so that the iris of the user's left eye faces the camera 171. In this case, the control unit 113 may determine the positions of points included in the displayed image based on the position of the camera 171 indicated by the camera position information.

[0107] Thereafter, the control unit 113 may control the output device 150 to display, on the head-mounted display 152, an image (e.g., image Img#2 in FIG. 9B ) that guides the user's gaze so that the iris of the user's right eye faces the camera 172, instead of the image that guides the user's gaze so that the iris of the user's left eye faces the camera 171. In this case, the control unit 113 may determine the positions of points included in the displayed image based on the position of the camera 172 indicated by the camera position information.

[0108] (Technical effect) According to the information processing device 100d of the seventh embodiment, similar to the information processing device 100c of the sixth embodiment described above, when calibration related to gaze estimation is being performed, the user's gaze can be appropriately guided so that the iris of the user's eye faces toward the camera.

[0109] Eighth Embodiment An eighth embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to Figures 7, 14, and 15. Below, the eighth embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 100b. Note that, for the eighth embodiment, descriptions that overlap with the first to seventh embodiments will be omitted as appropriate. Furthermore, in the drawings, parts common to other embodiments are indicated by the same reference numerals.

[0110] In the eighth embodiment, it is assumed that the user is wearing glasses. For example, as shown in Fig. 14(a), when the user is wearing glasses, a left eye lens 300L of the glasses is located between the user's left eye and the camera 171. A right eye lens 300R of the glasses is located between the user's right eye and the camera 172.

[0111] As described in the fifth embodiment, when cameras 171 and 172 each capture an image of a user's eyes, at least some of light sources 1523#1, 1523#2, and 1523#4 may emit near-infrared light (in other words, at least some of the light sources may illuminate the user with near-infrared light). Therefore, if the user is wearing eyeglasses, the near-infrared light may be reflected by the lenses of the eyeglasses (e.g., lenses 300L and 300R).

[0112] Fig. 14(b) shows an example of an eye image EIRg generated by capturing an image of a user's right eye with camera 172 when the user is wearing eyeglasses. As shown in Fig. 14(b), near-infrared light is reflected by the eyeglass lens (here, lens 300R). As a result, eye image EIRg includes a high-brightness region hba, which is a region of relatively high brightness resulting from the reflected near-infrared light. Note that, to avoid cluttering the drawing, Fig. 14(b) shows only one symbol "hba," but each of the multiple white circles included in image EIRg corresponds to a high-brightness region hba.

[0113] An eye image in which the high-brightness region hba overlaps the iris of a user's eye (e.g., eye image EIRg) is difficult to use for user biometric authentication (e.g., iris authentication) because some of the information related to the iris is lost due to the high-brightness region hba.

[0114] Therefore, the control unit 113 of the information processing device 100b may determine whether the user is wearing eyeglasses based on the eye image generated by the cameras 171 and 172 capturing images of the user's eyes. As shown in FIG. 14(b), if the user is wearing eyeglasses, the eye image EIRg includes the eyeglass frame fr. The control unit 113 may determine whether the eye image includes the eyeglass frame. If it is determined that the eye image includes the eyeglass frame, the control unit 113 may determine that the user is wearing eyeglasses. On the other hand, if it is determined that the eye image does not include the eyeglass frame, the control unit 113 may determine that the user is not wearing eyeglasses.

[0115] If it is determined that the user is wearing eyeglasses, the control unit 113 may detect a high-brightness region hba from the eye image (e.g., eye image EIRg). If the high-brightness region hba is detected, the control unit 113 may determine whether the high-brightness region hba overlaps an iris region, which is a region corresponding to the iris. If it is determined that the high-brightness region hba overlaps the iris region, the control unit 113 may change the illumination mode of the user by the multiple light sources 1523#1, 1523#2, 1523#3, and 1523#4 so that the high-brightness region hba does not overlap the iris region.

[0116] The operation of the information processing device 100b according to the eighth embodiment will be described with reference to the flowchart in Fig. 15. Here, it is assumed that the operation shown in the flowchart in Fig. 15 is performed when calibration related to gaze estimation is being performed. In this case, the operation shown in the flowchart in Fig. 15 may be, for example, a subroutine of the calibration. Note that the operation shown in the flowchart in Fig. 15 may be performed before calibration related to gaze estimation is performed.

[0117] 15 , the control unit 113 may first control the output device 150 to display, for example, image Img#1 shown in FIG. 9A on the head-mounted display 152 (specifically, the display 1521). While image Img#1 is being displayed (in other words, while the user is looking at image Img#1 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate an eye image EIL#1. The camera 172 may capture an image of the user's right eye to generate an eye image EIR#1. The image acquisition unit 111 may acquire eye images EIL#1 and EIR#1 (step S301). The eye images EIL#1 and EIR#1 may be at least temporarily stored in the storage device 120.

[0118] The control unit 113 may determine whether the user is wearing eyeglasses based on at least one of the eye images EIL#1 and EIR#1 (step S302). If it is determined in the processing of step S302 that the user is not wearing eyeglasses (step S302: No), the operation shown in the flowchart of FIG. 15 may be terminated.

[0119] If it is determined in the process of step S302 that the user is wearing eyeglasses (step S302: Yes), the control unit 113 may detect a high-brightness region hba (in other words, a light source reflection) caused by near-infrared light reflected by the lenses of the eyeglasses (e.g., lenses 300L and 300R) (step S303). Here, image Img#1 is an image for guiding the user's gaze so that the iris of the user's left eye faces toward the camera 171. Therefore, in the process of step S303, the control unit 113 may detect only the high-brightness region hba included in the eye image EIL#1.

[0120] Next, the control unit 113 may determine whether to change the user's illumination mode using the multiple light sources 1523#1, 1523#2, 1523#3, and 1523#4 (step S304). Here, if the high-brightness area hba is not detected in the processing of step S303, the control unit 113 may determine not to change the user's illumination mode. If the detected high-brightness area hba does not overlap the iris area, the control unit 113 may determine not to change the user's illumination mode. On the other hand, if the detected high-brightness area hba overlaps the iris area, the control unit 113 may determine to change the user's illumination mode.

[0121] If it is determined in the process of step S304 that the user's illumination mode should not be changed (step S304: No), the operation shown in the flowchart of Fig. 15 may be temporarily terminated. If it is determined in the process of step S304 that the user's illumination mode should be changed (step S304: Yes), the control unit 113 may control the multiple light sources 1523#1, 1523#2, 1523#3, and 1523#4 so that the high-brightness area hba does not overlap the iris area (step S305). As a result, the user's illumination mode may be changed.

[0122] Next, with image Img#1 displayed, camera 171 may capture the user's left eye, and camera 172 may capture the user's right eye. Camera 171 may capture the user's left eye, thereby generating eye image EIL#1a. Camera 172 may capture the user's right eye, thereby generating eye image EIR#1a. Image acquisition unit 111 may acquire eye images EIL#1a and EIR#1a (step S306). That is, in the process of step S306, image acquisition unit 111 may acquire the user's eye images EIL#1a and EIR#1a after the user's lighting condition has been changed. Note that eye images EIL#1a and EIR#1a may be at least temporarily stored in storage device 120.

[0123] Thereafter, the control unit 113 may control the output device 150 to display, for example, image Img#2 shown in FIG. 9B on the head-mounted display 152 instead of image Img#1. While image Img#2 is being displayed (in other words, while the user is looking at image Img#2 displayed on the display 1521), the camera 171 may capture an image of the user's left eye, and the camera 172 may capture an image of the user's right eye. The camera 171 may capture an image of the user's left eye to generate eye image EIL#2. The camera 172 may capture an image of the user's right eye to generate eye image EIR#2. The image acquisition unit 111 may acquire eye images EIL#2 and EIR#2. The eye images EIL#2 and EIR#2 may be at least temporarily stored in the storage device 120.

[0124] If it is determined that the user is wearing eyeglasses based on at least one of the eye images EIL#1 and EIR#1 (step S302: Yes), the control unit 113 may detect a high-brightness region hba caused by near-infrared light reflected by the lenses of the eyeglasses (step S303). Here, the image Img#2 is an image for guiding the user's gaze so that the iris of the user's right eye faces the camera 172. Therefore, in the processing of step S303, the control unit 113 may detect only the high-brightness region hba included in the eye image EIR#2.

[0125] Next, the control unit 113 may determine whether to change the user's illumination mode (step S304). If it is determined in the process of step S304 that the user's illumination mode is not to be changed (step S304: No), the operation shown in the flowchart of FIG. 15 may be temporarily terminated. If it is determined in the process of step S304 that the user's illumination mode is to be changed (step S304: Yes), the control unit 113 may control the multiple light sources 1523#1, 1523#2, 1523#3, and 1523#4 so that the high-brightness area hba does not overlap the iris area (step S305). As a result, the user's illumination mode may be changed.

[0126] Next, with image Img#2 being displayed, camera 171 may capture the user's left eye, and camera 172 may capture the user's right eye. Camera 171 may capture the user's left eye to generate eye image EIL#2a. Camera 172 may capture the user's right eye to generate eye image EIR#2a. Image acquisition unit 111 may acquire eye images EIL#2a and EIR#2a (step S306). That is, in the process of step S306, image acquisition unit 111 may acquire the user's eye images EIL#2a and EIR#2a after the user's lighting condition has been changed. Note that eye images EIL#2a and EIR#2a may be at least temporarily stored in storage device 120.

[0127] (Technical Effect) According to the information processing device 100b of the eighth embodiment, it is possible to determine whether or not the user is wearing eyeglasses. In the information processing device 100b of the eighth embodiment, when the user is wearing eyeglasses, the illumination mode of the user by the multiple light sources 1523#1, 1523#2, 1523#3, and 1523#4 is changed so that the high-brightness area hba does not overlap the iris area. Therefore, according to the information processing device 100b of the eighth embodiment, even when the user is wearing eyeglasses, it is possible to appropriately acquire an eye image used for biometric authentication of the user.

[0128] <Supplementary Notes> A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0129] (Supplementary Note 1) An information processing device comprising: a display unit; a first acquisition means for acquiring an eye image of a subject; a gaze estimation means for performing gaze estimation to estimate the gaze of the subject using the eye image; and a control means for performing calibration related to the gaze estimation, wherein the control means controls the display unit to display an instruction image related to the calibration, and the first acquisition means acquires at least a part of the eye image of the subject, captured while the instruction image is displayed on the display unit, as a first eye image that is also used for biometric authentication of the subject.

[0130] (Supplementary Note 2) The information processing device according to Supplementary Note 1, further comprising: an authentication means for performing biometric authentication of the target using the first eye image; and when the target is identified based on the biometric authentication, the control means acquires past calibration information relating to the target.

[0131] (Supplementary Note 3) The instruction images include a first instruction image for directing one of the subject's left or right eye toward a first camera that captures an image of the one of the left or right eye, a second instruction image for directing the other of the subject's left or right eye toward a second camera that captures an image of the other of the left or right eye, and one or more third instruction images different from the first instruction image and the second instruction image; when the control means performs the calibration, the control means controls the display unit to display the first instruction image, and controls the display unit to display the second instruction image after the first instruction image is displayed; the first acquisition unit acquires, as the first eye image, an eye image of one of the subject's left or right eye when the first instruction image is displayed on the display unit, and acquires, as the first eye image, an eye image of the other of the subject's left or right eye when the second instruction image is displayed on the display unit; the authentication means performs the biometric authentication of the subject using the eye image of one of the left or right eye and the eye image of the other of the left or right eye as the first eye image; The information processing device described in Appendix 2, wherein, when the target is identified by the biometric authentication, the control means controls the display unit not to display the one or more third instruction images and acquires past calibration information related to the target; and when the target is not identified by the biometric authentication, the control means controls the display unit to display the one or more third instruction images.

[0132] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the instruction image is an image including content for guiding the gaze of the target.

[0133] (Supplementary Note 5) The information processing device described in any one of Supplementary Notes 1 to 4, wherein the display unit is a head-mounted display, and further includes a second acquisition means for acquiring position information indicating a position of a camera arranged on the head-mounted display, and the control means changes the instruction image based on the position information.

[0134] (Supplementary Note 6) The information processing device described in any one of Supplementary Notes 1 to 4, wherein the display unit is a head-mounted display, when the control means performs the calibration, the control means controls the display unit to display, as the instruction image, a fourth instruction image that guides the subject's line of sight to a predetermined position on the display unit, the first acquisition unit acquires a second eye image as the eye image of the subject when the fourth instruction image is displayed on the display unit, and the information processing device includes position estimation means that estimates a position of a camera arranged on the head-mounted display using the second eye image.

[0135] (Supplementary Note 7) When the control means performs the calibration, the control means controls the display unit to display one instruction image that guides the subject's gaze at one position on the display unit, and after the subject's gaze is directed to the one position, controls the display unit to display another instruction image that guides the subject's gaze at another position on the display unit. (Supplementary Note 7) An information processing device described in any one of Supplementary Notes 1 to 6.

[0136] (Supplementary Note 8) The information processing device according to any one of Supplementary Notes 1 to 7, further comprising: an illumination unit that illuminates the target with light; and when the target is wearing glasses, the control unit changes an illumination mode of the illumination unit.

[0137] (Supplementary Note 9) An information processing method in an information processing device having a display unit, comprising: acquiring an eye image of a subject; performing gaze estimation to estimate a gaze of the subject using the eye image; performing calibration related to the gaze estimation; controlling the display unit to display an instruction image related to the calibration; and acquiring at least a portion of the eye image of the subject captured while the instruction image is being displayed on the display unit as a first eye image that is also used for biometric authentication of the subject.

[0138] (Supplementary Note 10) A recording medium having recorded thereon a computer program for causing a computer having a display unit to execute an information processing method, which includes acquiring an eye image of a subject, performing gaze estimation to estimate the gaze of the subject using the eye image, performing calibration related to the gaze estimation, controlling the display unit to display an instruction image related to the calibration, and acquiring at least a part of the eye image of the subject captured while the instruction image is being displayed on the display unit as a first eye image that is also used for biometric authentication of the subject.

[0139] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 8, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 9 and 10 in the same dependent relationship as Supplementary Notes 2 to 8. Furthermore, not limited to Supplementary Notes 1, 9, and 10, some or all of the configurations described as Supplements may be made dependent on various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0140] This disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope of the claims and the gist or idea of ​​the invention as can be read from the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical scope of this disclosure.

[0141] 10, 100, 100a, 100b, 100c, 100d Information processing device 11 Display unit 12, 111 Image acquisition unit 13, 112 Gaze estimation unit 14, 113 Control unit

Claims

1. An information processing device comprising: a display unit; a first acquisition means for acquiring an eye image of a subject; a gaze estimation means for performing gaze estimation to estimate the gaze of the subject using the eye image; and a control means for performing calibration related to the gaze estimation, wherein the control means controls the display unit to display an instruction image related to the calibration, and the first acquisition means acquires at least a part of the eye image of the subject, captured while the instruction image is being displayed on the display unit, as a first eye image that is also used for biometric authentication of the subject.

2. An information processing device as described in claim 1, further comprising an authentication means for performing biometric authentication of the target using the first eye image, and when the target is identified based on the biometric authentication, the control means acquires past calibration information related to the target.

3. The instruction images include a first instruction image for directing one of the subject's left or right eye toward a first camera that captures an image of the one of the left or right eye, a second instruction image for directing the other of the subject's left or right eye toward a second camera that captures an image of the other of the left or right eye, and one or more third instruction images different from the first instruction image and the second instruction image; when the control means performs the calibration, the control means controls the display unit to display the first instruction image, and controls the display unit to display the second instruction image after the first instruction image is displayed; the first acquisition unit acquires, as the first eye image, an eye image of one of the subject's left or right eye when the first instruction image is displayed on the display unit, and acquires, as the first eye image, an eye image of the other of the subject's left or right eye when the second instruction image is displayed on the display unit; the authentication means performs the biometric authentication of the subject using the eye image of one of the left or right eye and the eye image of the other of the left or right eye as the first eye image; 3. The information processing device according to claim 2, wherein, when the target is identified by the biometric authentication, the control means controls the display unit not to display the one or more third instruction images and acquires past calibration information related to the target; and when the target is not identified by the biometric authentication, the control means controls the display unit to display the one or more third instruction images.

4. The information processing device according to claim 1, wherein the instruction image is an image containing content for guiding the gaze of the subject.

5. The information processing device according to claim 1, wherein the display unit is a head-mounted display, and further comprising a second acquisition means for acquiring position information indicating the position of a camera arranged on the head-mounted display, and the control means changes the indication image based on the position information.

6. The information processing device according to claim 1, wherein the display unit is a head-mounted display, when the control means performs the calibration, the control means controls the display unit to display, as the instruction image, a fourth instruction image that guides the subject's line of sight to a predetermined position on the display unit, the first acquisition unit acquires a second eye image as the eye image of the subject when the fourth instruction image is displayed on the display unit, and the information processing device comprises position estimation means that estimates the position of a camera arranged on the head-mounted display using the second eye image.

7. An information processing device as described in claim 1, wherein when the control means performs the calibration, the control means controls the display unit to display one instruction image at one position on the display unit to guide the subject's gaze, and after the subject's gaze is directed to the one position, the control means controls the display unit to display another instruction image at another position on the display unit to guide the subject's gaze.

8. An information processing device according to claim 1, further comprising an illumination means for illuminating the subject with light, and wherein if the subject is wearing glasses, the control means changes the illumination mode of the illumination means.

9. The information processing device according to claim 2, wherein the display unit is a head-mounted display.

10. The information processing device according to claim 9, wherein the instruction image is an image containing content for guiding the gaze of the subject.

11. An information processing device according to claim 9, further comprising a second acquisition means for acquiring position information indicating the position of a camera arranged on the head-mounted display, and the control means changes the instruction image based on the position information.

12. The information processing device according to claim 9, wherein when the control means performs the calibration, the control means controls the display unit to display, as the instruction image, a fourth instruction image that guides the subject's line of sight to a predetermined position on the display unit; the first acquisition unit acquires a second eye image as the eye image of the subject when the fourth instruction image is displayed on the display unit; and the information processing device comprises a position estimation means that estimates the position of a camera arranged on the head-mounted display using the second eye image.

13. An information processing device as described in claim 9, wherein when the control means performs the calibration, the control means controls the display unit to display one instruction image at one position on the display unit to guide the subject's gaze, and after the subject's gaze is directed to the one position, the control means controls the display unit to display another instruction image at another position on the display unit to guide the subject's gaze.

14. An information processing device according to claim 9, further comprising an illumination means for illuminating the subject with light, wherein if the subject is wearing glasses, the control means changes the illumination mode of the illumination means.

15. The information processing device according to claim 3, wherein the display unit is a head-mounted display.

16. An information processing device according to claim 15, further comprising a second acquisition means for acquiring position information indicating the position of a camera arranged on the head-mounted display, and the control means changes the instruction image based on the position information.

17. The information processing device according to claim 15, wherein when the control means performs the calibration, the control means controls the display unit to display, as the instruction image, a fourth instruction image that guides the subject's line of sight to a predetermined position on the display unit; the first acquisition unit acquires a second eye image as the eye image of the subject when the fourth instruction image is displayed on the display unit; and the information processing device comprises position estimation means that estimates the position of a camera arranged on the head-mounted display using the second eye image.

18. An information processing device according to claim 15, further comprising an illumination means for illuminating the subject with light, and wherein if the subject is wearing glasses, the control means changes the illumination mode of the illumination means.

19. An information processing method in an information processing device having a display unit, comprising: acquiring an eye image of a subject; performing gaze estimation to estimate the gaze of the subject using the eye image; performing calibration related to the gaze estimation; controlling the display unit to display an instruction image related to the calibration; and acquiring at least a portion of the eye image of the subject captured while the instruction image is being displayed on the display unit as a first eye image that is also used for biometric authentication of the subject.

20. A recording medium having recorded thereon a computer program for causing a computer having a display unit to execute an information processing method, which comprises acquiring an eye image of a subject, performing gaze estimation to estimate the gaze of the subject using the eye image, performing calibration related to the gaze estimation, controlling the display unit to display an instruction image related to the calibration, and acquiring at least a portion of the eye image of the subject captured while the instruction image is being displayed on the display unit as a first eye image that is also used for biometric authentication of the subject.

Images