Optical device, gaze detection device, retinal projection display device, head-mounted display device, optometry device, user state estimation device, driving assistance system, object tilt detection method, and gaze detection method

Abstract

To detect reflection light by an object with high sensitivity while suppressing an increase in a volume of light with which the object is irradiated.SOLUTION: An optical device according to one aspect of the present invention includes: a plurality of light emitting units capable of irradiating an object with light; a detection unit for detecting the light reflected by the object and outputting a signal on the basis of the reflected light; and an output unit for outputting inclination information on the object acquired on the basis of an output signal output from the detection unit. The light emitted from the light emitting units is subjected to time modulation by a code having orthogonality. The plurality of light emitting units include a first light emitting unit and a second light emitting unit. A light emitting timing of the light by the first light emitting unit is different from a light emitting timing of the light by the second light emitting unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an optical device, a gaze detection device, a retinal projection display device, a head-mounted display device, an ophthalmological device, and device, user state estimation device, driving assistance system, Object The tilt detection method and the line of sight detection method Regarding. [Background technology]

[0002] Optical devices, such as devices that optically detect the tilt of an object such as an eyeball, are known. From the viewpoint of safety, such optical devices may be required to irradiate the object with light of a weak intensity and detect the weak light reflected by the object with high sensitivity.

[0003] Furthermore, an optical device that detects weak return light from an object that has been irradiated with light is disclosed, for example, in Patent Document 1, a biological information measuring device. This biological information measuring device includes a pseudo-noise sequence generator, a laser driver that performs amplitude shift keying (ASK) using the pseudo-noise sequence generated by the pseudo-noise sequence generator to output a light intensity amplitude that has been spectrum-spread, and a light emitting unit that includes a semiconductor laser that is assigned and driven by the spectrum-spread laser driver output, for each of two different optical wavelengths. The pseudo-noise sequence is different for each of the two different optical wavelengths, and the device includes a photodetector that receives light that has propagated through the living body and reaches it and generates an electrical detection signal, and an AD converter that receives the detection signal and performs AD conversion on it. Furthermore, a correlation processing unit is provided for each of the two optical wavelengths. The correlation processing unit is made up of a multiplier that inputs the AD conversion detection signal output from the AD converter and the pseudo-noise sequence generated by the pseudo-noise sequence generator, and multiplies both signals over one period or more to perform inverse spectrum spreading, and an accumulator that accumulates the output of the multiplier. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the configuration of Patent Document 1, light is irradiated onto an object such as a living body in parallel from multiple light sources such as semiconductor lasers, which increases the amount of light irradiated onto the object, leaving room for improvement in terms of safety for the object and the efficiency of use of the irradiated light.

[0005] An object of the present invention is to enable highly sensitive detection of reflected light from an object while suppressing an increase in the amount of light irradiated onto the object. [Means for solving the problem]

[0006] An optical device according to one aspect of the present invention comprises: Light temporally modulated by an orthogonal Hadamard coding pattern is Object Light a detection unit that detects the light reflected by the object and outputs a signal based on the reflected light; and an output unit that outputs tilt information of the object obtained based on the output signal output from the detection unit. an encoding unit that selects and outputs the Hadamard coded pattern; an inner product calculation unit that calculates an inner product of digital voltage data for one cycle of the Hadamard coded pattern and reference voltage data input from the encoding unit; a determination unit that determines whether the detection unit has received light reflected by the object illuminated with the light emitted by the plurality of light emitting units, based on an inner product value that is a result of the inner product calculation by the inner product calculation unit; a probability selection unit that selects, from the plurality of light emitting units, a light emission probability distribution according to a usage frequency of the light emitting unit that was used when the determination unit determined that the detection unit received light; and a stochastic encoding unit that selects, from the plurality of Hadamard coded patterns that encode the light, a Hadamard coded pattern based on a coding probability distribution according to a usage frequency of the Hadamard coded pattern that was used when the determination unit determined that the detection unit received light. and before the plurality of light-emitting units include a first light-emitting unit and a second light-emitting unit, and a timing at which the first light-emitting unit emits light is different from a timing at which the second light-emitting unit emits light; before The Hadamard coding pattern does not have all elements that are 1, but includes both 1 and -1. [Effects of the Invention]

[0007] According to the present invention, it is possible to detect light reflected by an object with high sensitivity while suppressing an increase in the amount of light irradiated onto the object. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a gaze detection device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a processing unit according to the first embodiment. [Figure 3] FIG. 2 is a block diagram illustrating an example of the functional configuration of a processing unit according to the first embodiment. [Figure 4]10 is a flowchart of an example of processing by a processing unit according to the first embodiment. [Figure 5] 5A and 5B are diagrams illustrating Hadamard codes, where FIG. 5A is a diagram of an encoding pattern and FIG. 5B is a diagram of an inner product operation. [Figure 6A] 10A and 10B are diagrams showing examples of experimental results of weak light detection. [Figure 6B] 10A and 10B are diagrams showing examples of experimental results of detecting gaze direction using Hadamard coding patterns. [Figure 7] FIG. 10 is a diagram illustrating labeled decoding processing. [Figure 8] FIG. 10 is a block diagram illustrating an example of the functional configuration of a processing unit according to the second embodiment. [Figure 9] 10 is a flowchart of an example of processing by a processing unit according to the second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of the configuration of a retinal projection display device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes the preferred embodiments of the present invention with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations may be omitted.

[0010] Furthermore, the embodiments shown below are intended to exemplify optical devices embodying the technical concepts of the present invention, and the present invention is not limited to the embodiments shown below. Unless otherwise specified, the shapes of the components described below, their relative locations, parameter values, and the like are intended to be illustrative and not to limit the scope of the present invention. Furthermore, the sizes and positional relationships of components shown in the drawings may be exaggerated for clarity of explanation.

[0011] The optical device of the embodiment has a plurality of light-emitting units capable of irradiating light onto an object, a detection unit that detects the light reflected by the object and outputs a signal based on the reflected light, and an output unit that outputs tilt information of the object obtained based on the output signal output from the detection unit.

[0012] The object is, for example, a human eyeball. Since the eyeball tilts in the direction the human gaze is directed, the optical device outputs this eyeball tilt information (tilt information) as information indicating the human gaze direction (gaze direction information). Note that the eyeball tilt information includes not only information directly indicating the tilt angle, but also information regarding tilt angles other than the eyeball tilt angle.

[0013] The gaze direction information output by the optical device is used, for example, in an eye tracking device, an optometry device, a user state estimation device, a driving assistance system, etc. Alternatively, it is used to correct the position or content of a projected image according to the gaze direction when an image is projected onto the retina or the like in a head-mounted display device such as a retinal projection display device or a head-mounted display (HMD).

[0014] In such optical devices, when light is irradiated in parallel onto an object such as a living body from multiple light sources including semiconductor lasers, the amount of light irradiated onto the object increases, leaving room for improvement in terms of safety for the object and the efficiency of use of the irradiated light.

[0015] In the embodiment, the light emitted from the light-emitting unit is time-modulated by an orthogonal code, the plurality of light-emitting units includes a first light-emitting unit and a second light-emitting unit, and the timing of emitting light by the first light-emitting unit is different from the timing of emitting light by the second light-emitting unit. As a result, light is not emitted from all of the plurality of light sources in parallel onto the object, which prevents an increase in the amount of light irradiated onto the object and enables highly sensitive detection of reflected light from the object while improving safety for the object and light utilization efficiency.

[0016] Hereinafter, an embodiment will be described using a gaze detection device mounted on a glasses-type support and configured to detect the inclination angle of the eyeball of a person wearing the glasses-type support as the gaze direction as an example of an optical device. The human eyeball is an example of an object and an example of a three-dimensional object. The gaze direction is an example of the inclination of the object and an example of the inclination of a three-dimensional object.

[0017] Although the embodiment illustrates a right eyeball of a human being, the same applies to a left eyeball. Two gaze detection devices can also be applied to both eyeballs.

[0018] [First embodiment] (Configuration example of gaze detection device 10) First, a configuration of the gaze detection device 10 according to the first embodiment will be described.

[0019] As shown in FIG. 1, the gaze detection device 10 includes a light source 1, a concave mirror 2, a light detection unit 3, and a processing unit 100.

[0020] The light source 1, the concave mirror 2, and the light detection unit 3 are arranged on an optical system support 4. The optical system support 4 is tiltably fixed to an eyeglass frame 22 of an eyeglass-type support 20, which includes an eyeglass lens 21 and an eyeglass frame 22, via a ball joint 4a.

[0021] The use of the ball joint 4a makes it possible to adjust the tilt of the optical system support 4. As the ball joint 4a constituting the installation fine adjustment mechanism, a method of fixing using mechanical pressure acting between a spherical structure and its outer shell structure, as well as a method of fixing using magnetic force acting between a magnetized spherical structure and a metal opening structure can be used.

[0022] The eyeglass-type support body 20 can be worn on a person's head. When the eyeglass-type support body 20 is worn, the optical system including the light source 1, the concave mirror 2, and the light detection unit 3 is placed in a position close to the eyeball 30 (in front of the eye).

[0023] The light source 1 is a VCSEL (Vertical Cavity Surface Emitting Laser) array in which VCSEL elements are arranged two-dimensionally, and each VCSEL element can be driven individually. Hereinafter, the smallest unit of an individually driven VCSEL element will be referred to as a light-emitting element. Each light-emitting element emits laser light with directionality and a finite divergence angle.

[0024] Here, the laser light emitted by the light source 1 is an example of light. Also, one VCSEL element is an example of a light-emitting unit, and the light source 1 is an example of a plurality of light-emitting units capable of emitting laser light that is time-modulated by orthogonal codes.

[0025] However, the light source 1 is not limited to a VCSEL array as long as it includes a plurality of light-emitting units that emit light. The light source 1 may also be configured by arranging a plurality of LDs (semiconductor lasers; laser diodes) or LEDs (light-emitting diodes; light-emitting diodes) that emit laser light two-dimensionally within a plane. The light source 1 may also be configured by including a plurality of pulsed laser light sources that emit pulsed laser light. Furthermore, the light source 1 may also be configured by combining a plurality of types of light sources. VCSELs are preferable in that they can be easily integrated within a two-dimensional plane and can be mounted in a compact size on a wearable device.

[0026] Furthermore, the wavelength of the laser light emitted by the light source 1 is preferably a wavelength of near-infrared light, which is invisible light, so as not to impede the visibility of the "person" whose line of sight is to be detected. However, the wavelength is not limited to this and may be visible light.

[0027] Laser light L0 emitted by light source 1 propagates through the space in front of the eye in a direction substantially parallel to the face of a person wearing eyeglass-type support 20 or the lens surface of the eyeglass lens 21 worn, toward concave mirror 2. Laser light L0 is divergent light that propagates while widening its beam diameter due to diffraction at the opening of the light-emitting section of light source 1. The divergence angle of the divergent light can be controlled by the shape of the opening of the light-emitting section. Laser light L0 propagates while diverging, and is incident on concave mirror 2.

[0028] In this embodiment, the light source 1 is illustrated as having light-emitting units arranged two-dimensionally on a single substrate, but the number of substrates on which the light-emitting units are arranged does not necessarily have to be one. For example, the light source 1 may have a first substrate and a second substrate different from the first substrate, with the first light-emitting units provided on the first substrate and the second light-emitting units provided on the second substrate, and the first substrate and the second substrate arranged in parallel.

[0029] Concave mirror 2, which is an example of focusing / reflecting means, has a reflective surface with a curvature, reflects incident laser light L0, and irradiates focused laser light L1 onto eyeball 30. The focused laser light L1 is incident near pupil 31 of eyeball 30. The center of curvature of the concave surface of concave mirror 2 is located off the optical axis of the optical path from light source 1 to concave mirror 2, and the optical system including light source 1, concave mirror 2, and light detection unit 3 constitutes a so-called off-axis optical system.

[0030] Although FIG. 1 shows a concave mirror 2 as an example of a focusing reflecting means, the present invention is not limited to this as long as it can focus light. For example, a configuration combining a convex lens and a plane mirror, a wavefront control element using a hologram, a diffractive optical element, etc. may also be used.

[0031] Note that the gaze detection device 10 may have a focusing means that has a function of focusing light but does not have a function of reflecting light, instead of the focusing and reflecting means such as the concave mirror 2. For example, in FIG. 1 , instead of the light detection unit 3, a light source 1 and a focusing optical element such as a convex lens or a microlens array may be provided at the position of the light detection unit 3, and instead of the concave mirror 2, the light detection unit 3 may be provided at the position of the concave mirror 2.

[0032] Furthermore, if an anamorphic aspherical surface having different curvatures in two orthogonal directions in a plane intersecting the optical axis of the focused laser light L1 is used as the concave surface of the concave mirror 2, the diameter of the beam of reflected laser light L2 obtained by reflecting the focused laser light L1 off the eyeball 30 can be further reduced and the beam can be shaped into an isotropic state. Here, the reflected laser light L2 is an example of reflected light.

[0033] The angle of incidence of the focused laser light L1 on the eyeball 30 is adjusted so that it is incident at a predetermined angle on the center of the pupil 31 of the eyeball 30 when the eye is looking straight ahead. The light source 1 can emit laser light L0 from each of a plurality of light-emitting units. The laser light L0 emitted by the light source 1 is irradiated onto a plurality of locations on the eyeball 30, or is irradiated onto the eyeball 30 at a plurality of angles.

[0034] The pupil surface (corneal surface) of the eyeball 30, which is an example of an irradiated surface, is a transparent body containing water and generally has a reflectance of approximately 2 to 4%. The focused laser light L1 incident on the vicinity of the pupil 31 of the eyeball 30 is reflected by the surface of the pupil 31 of the eyeball 30, and the reflected laser light L2 propagates toward the light detection unit 3. For ease of explanation, hereinafter, the surface of the eyeball 30 may be referred to as the eyeball 30, the surface of the pupil 31 may be referred to as the pupil 31, and the surface of the cornea 32 may be referred to as the cornea 32.

[0035] Furthermore, the illuminated surface is not limited to the eyeball 30, and this embodiment can be applied to any object that is a three-dimensional object having a curvature.

[0036] In addition, the gaze detection device 10 adjusts the spacing between each light-emitting unit in the light source 1, the radius of curvature of the concave mirror 2, and the installation position so that only one of the laser beams L0 emitted by the multiple light-emitting units in the light source 1 ultimately enters the light detection unit 3, depending on the tilt of the eyeball 30.

[0037] The light detection unit 3 is an example of a detection unit that detects light reflected by an object and outputs a signal based on the reflected light. The light detection unit 3 is a single-pixel photo detector (PD) that outputs a signal indicating the intensity of the received light, or a position sensitive detector (PSD) that can output both a signal indicating the intensity of the received light and a signal indicating the position of the reflected laser light L2 incident on the light detection unit 3.

[0038] 1 , as long as only one of the laser beams L0 emitted by the multiple light-emitting units of the light source 1 ultimately enters the photodetector 3 in accordance with the tilt of the eyeball 30. Also, as long as the configuration satisfies the above, a deflection optical element may be provided between the eyeball 30 and the photodetector 3. The photodetector 3 is not limited to a PSD, and may be an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0039] Using a PSD as the light detection unit 3 is more preferable because it can detect the reflected light from the eyeball 30 with high sensitivity using information on light intensity, and can detect the gaze direction more accurately based on the incident position of the reflected laser light L2 on the PSD and the position of the light emitting unit in the light source 1.

[0040] The processing unit 100 outputs a control signal to sequentially cause each light-emitting unit of the light source 1 to emit light. The processing unit 100 also receives an output signal from the light detection unit 3 and executes processing to estimate the gaze direction. The processing unit 100 can be disposed in the eyeglass frame 22, for example.

[0041] Since the origin of the reflected laser light L2 incident on the light detection unit 3 from which light-emitting unit included in the light source 1 emitted the laser light L0 differs depending on the line of sight direction, the processing unit 100 can estimate the line of sight direction based on the output signal from the light detection unit 3 and the position of the light-emitting unit in the light source 1.

[0042] The gaze direction is estimated using the position of the light emitting unit in the light source 1, the position of the reflected laser light L2 incident on the light detecting unit 3, and a predetermined eyeball model.

[0043] Here, the light source 1 has multiple light-emitting units and is capable of high-speed time modulation. The gaze detection device 10 time-modulates the laser light emitted by the light source 1 according to an orthogonal encoding pattern, and extracts a component having an encoding pattern that matches the incident laser light from the output signal of the light detection unit 3. This makes it possible to remove the influence of light from the external environment (not involving modulation) and improve the signal-to-noise ratio of the output signal, making it suitable for detecting the gaze direction in a bright environment and enabling a reduction in the amount of focused laser light L1 irradiated onto the eyeball 30.

[0044] The gaze detection device 10 sequentially emits light from each light-emitting unit included in the light source 1, which is preferable in that the light intensity of the focused laser light L1 irradiated onto the eyeball 30 can be reduced compared to when the light-emitting units emit light in parallel. Furthermore, when all of the multiple light-emitting units emit light in parallel, it is necessary to prepare light source driving units such as light source modulation units equal to the number of light-emitting units, but the gaze detection device 10 does not require this because it emits light one by one, which is preferable in that the light source driving units can be implemented in a small and lightweight manner.

[0045] Although FIG. 1 shows an example in which the optical system and the processing unit are arranged in the eyeglass frame 22, the present invention is not limited to this, and a head-mounted display, a headgear-type holding structure, or the like may also be used.

[0046] (Example of hardware configuration of processing unit 100) Next, the hardware configuration of the processing unit 100 will be described with reference to Fig. 2. Fig. 2 is a block diagram illustrating an example of the hardware configuration of the processing unit 100.

[0047] 2, the processing unit 100 has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an SSD (Solid State Drive) 104. The processing unit 100 also has a light source driving circuit 105, an A / D (Analog / Digital) conversion circuit 106, and an input / output I / F (Interface) 107. These are connected via a system bus B so as to be able to transmit and receive data or signals to and from each other.

[0048] The CPU 101 is a computing device that reads programs and data from storage devices such as the ROM 102 and the SSD 104 onto the RAM 103 and executes the programs to control the entire processing unit 100 and to realize functions described below. Note that some or all of the functions of the CPU 101 may be realized by electronic circuits such as an ASIC (application specific integrated circuit) or an FPGA (field-programmable gate array).

[0049] The ROM 102 is a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 102 stores programs and data such as a BIOS (Basic Input / Output System) that is executed when the processing unit 100 starts up, OS settings, and network settings. The RAM 103 is a volatile semiconductor memory (storage device) that temporarily retains programs and data.

[0050] The SSD 104 is a non-volatile memory that stores various data and programs that execute the processing by the processing unit 100. The SSD may be an HDD (Hard Disk Drive).

[0051] The light source drive circuit 105 is an electric circuit that is electrically connected to the light source 1 and outputs a drive voltage to the light source 1 in accordance with a control signal. The light source drive circuit 105 can cause a plurality of light emitting units included in the light source 1 to emit light sequentially.

[0052] The drive voltage can have a voltage waveform of a square wave, a sine wave, or a predetermined waveform, and the light source drive circuit 105 can change the period (frequency) of these voltage waveforms to modulate the period of the drive voltage.

[0053] The A / D conversion circuit 106 is an electric circuit electrically connected to the photodetector unit 3, and outputs digital voltage data Dat obtained by A / D converting the output signal Sen, which is an analog voltage signal output by the photodetector unit 3.

[0054] The input / output I / F 107 is an interface for connecting to external devices such as a PC (Personal Computer) and video equipment.

[0055] (Example of functional configuration of processing unit 100) 3 is a block diagram showing an example of the functional configuration of the processing unit 100. As shown in FIG. 3, the processing unit 100 includes a selection unit 111, an encoding unit 112, a light source modulation unit 113, a switching unit 114, an A / D conversion unit 115, an accumulation unit 116, an inner product calculation unit 117, a determination unit 118, an estimation unit 119, and an output unit 120.

[0056] Of these, the functions of the selection unit 111, the encoding unit 112, the inner product calculation unit 117, the determination unit 118, and the estimation unit 119 are realized by the CPU 101 in Fig. 2 executing a predetermined program stored in the ROM 102, etc. The functions of the light source modulation unit 113 and the switching unit 114 are realized by the light source drive circuit 105 etc. in Fig. 2. The function of the A / D conversion unit 115 is realized by the A / D conversion circuit 106 etc. in Fig. 2, the function of the accumulation unit 116 is realized by the RAM 103 etc. in Fig. 2, and the function of the output unit 120 is realized by the input / output I / F 107 etc. in Fig. 2.

[0057] The selection unit 111 selects a light-emitting unit to emit light from among the eight light-emitting units included in the light source 1, namely, the first light-emitting unit 11, the second light-emitting unit 12, ..., and the eighth light-emitting unit 18, and outputs information indicating the selected light-emitting unit to the switching unit 114. Note that the number of light-emitting units included in the light source 1, which is eight, is just an example, and any number of light-emitting units can be selected as appropriate as long as it is plural.

[0058] The encoding unit 112 selects an encoding pattern as an orthogonal code and outputs the encoding pattern data to the light source modulation unit 113. The encoding unit 112 also outputs the selected encoding pattern data to the inner product calculation unit 117 as reference voltage data Ref for inner product calculation by the inner product calculation unit 117. The orthogonal code is, for example, a Hadamard code. Here, the Hadamard code refers to a coding system used for detecting and correcting errors in signals.

[0059] Also, different coding patterns are prepared in advance depending on the position of the light-emitting unit of the light source 1, and are stored in the SSD 104 or the like in Fig. 2. The coding unit 112 can acquire the coding pattern corresponding to the position of the light-emitting unit selected by the selection unit 111 by referring to the SSD 104 or the like.

[0060] The light source modulation unit 113 modulates the drive current for driving the light source 1 based on the encoding pattern data input from the encoding unit 112, and outputs the modulated drive current to the light emitting unit selected by the selection unit 111 via the switching unit 114.

[0061] The switching unit 114 switches so that the drive current from the light source modulation unit 113 is output to the light-emitting unit selected by the selection unit 111. For example, the function of the switching unit 114 is realized by a demultiplexer or the like. In this embodiment, the demultiplexer is included in the light source drive circuit 105.

[0062] The A / D conversion unit 115 converts the output signal Sen from the photodetection unit 3 that has received the reflected laser light L2 into digital voltage data Dat and outputs it to the storage unit 116. The storage unit 116 stores the digital voltage data Dat input from the A / D conversion unit 115 for one period of the encoding pattern. The light source 1 continues to emit light in accordance with the encoding pattern until the storage unit 116 stores the digital voltage data Dat for one period of the encoding pattern.

[0063] The inner product calculation unit 117 acquires digital voltage data Dat for one period of the encoding pattern by referring to the storage unit 116, and calculates the inner product of this digital voltage data Dat and the reference voltage data Ref input from the encoding unit 112.

[0064] The determination unit 118 determines, based on the dot product value that is the dot product calculation result by the dot product calculation unit 117, whether or not the light detection unit 3 has received reflected laser light L2 of the focused laser light L1 by the eyeball 30 that is irradiated with the focused laser light L1 that is based on the laser light L0 emitted by the light source 1. Specifically, if the dot product value is greater than a predetermined threshold, the determination unit 118 determines that the light detection unit 3 has received the reflected laser light L2, and if the dot product value is not greater than the predetermined threshold, the determination unit 118 determines that the light detection unit 3 has not received the reflected laser light L2.

[0065] It is preferable to set an appropriate value in advance for this predetermined threshold value depending on the brightness of the surroundings in which the gaze detection device 10 is used, the reflectance or shape of the eyeball 30 of the user using the gaze detection device 10, etc.

[0066] The estimation unit 119 estimates the gaze direction by calculation based on the position of the light-emitting unit of the light source 1 detected based on the output signal Sen by the light detection unit 3. Furthermore, when the light detection unit 3 is a PSD, the estimation unit 119 estimates the gaze direction by calculation based on the position of the light-emitting unit of the light source 1 detected based on the output signal Sen by the light detection unit 3 and the position of the reflected laser light L2 incident on the light detection unit 3, and acquires gaze direction information. The estimation unit 119 outputs the acquired gaze direction information to an external device via the output unit 120.

[0067] The output unit 120 can output tilt information of the object obtained based on the output signal output from the light detection unit 3.

[0068] The external device is a retinal projection display device, a head-mounted display device, an ophthalmoscope, or the like, which uses the gaze direction information acquired by the gaze detection device 10. However, the external device is not limited to these, and the output unit 120 can also output the gaze direction information to an external device such as a PC (Personal Computer), a storage device, a communication device connected to the Internet, or the like.

[0069] In this embodiment, the light source 1 includes eight light-emitting units, and the light-emitting units selected by the selection unit 111 are sequentially caused to emit light one by one. If the eight light-emitting units are caused to emit light in parallel, the focused laser beam L1 irradiated onto the eyeball 30 will be added together, increasing the light intensity of the focused laser beam L1 and possibly reducing safety for the eyeball 30. Furthermore, only some of the laser beams emitted by the eight light-emitting units are used for detecting the gaze direction, and the remaining laser beams are not used. This reduces the utilization efficiency of the light irradiated onto the eyeball 30 for detecting the gaze direction, i.e., the ratio of the number of laser beams used for detection to the number of emitted laser beams.

[0070] By sequentially causing the light-emitting units selected by the selection unit 111 to emit light one by one, an increase in the light intensity of the focused laser light L1 irradiated onto the eyeball 30 is suppressed, and a decrease in safety for the eyeball 30 and a decrease in the efficiency of light use in the line of sight are suppressed. Furthermore, if eight light-emitting units are caused to emit light in parallel, drive circuits for eight light-emitting units are required, but by causing them to emit light sequentially, the number of drive circuits is also reduced.

[0071] However, in this embodiment, it is not limited to sequentially emitting light from the eight light-emitting units one by one, and it is sufficient that all eight light-emitting units do not emit light in parallel. In other words, the plurality of light-emitting units include a first light-emitting unit 11 and a second light-emitting unit 12, and it is sufficient that the timing of emitting laser light L0 by the first light-emitting unit 11 is different from the timing of emitting laser light L0 by the second light-emitting unit 12.

[0072] For example, four adjacent light-emitting elements out of the eight may be sequentially emitted in two separate bursts, two adjacent light-emitting elements in four separate bursts, or two adjacent light-emitting elements may be sequentially emitted the first time, three adjacent light-emitting elements the second time, two adjacent light-emitting elements the third time, and one light-emitting element the fourth time. However, from the viewpoint of increasing the detection resolution of the position of the light-emitting element that emitted the laser beam L0, it is preferable to sequentially emit light from each of the light-emitting elements eight times. In this embodiment, by sequentially emitting light from the eight light-emitting elements included in the light source 1, the emission timing of the first light-emitting element 11 and the emission timing of the second light-emitting element 12 are different.

[0073] (Example of processing by the processing unit 100) Next, processing by the processing unit 100 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of processing by the processing unit 100. Fig. 4 shows processing by the processing unit 100 triggered by the start of gaze detection by the gaze detection device 10. Gaze detection by the gaze detection device 10 is started, for example, when an operation unit or the like of the gaze detection device 10 receives an operation to start gaze detection by a user of the gaze detection device 10.

[0074] First, in step S41, the selection unit 111 selects which of the eight light-emitting units included in the light source 1 is to emit light, and outputs to the encoding unit 112 information indicating the selected light-emitting unit.

[0075] Subsequently, in step S42, the encoding unit 112 selects an encoding pattern as a code having orthogonality, and outputs the encoding pattern data to the light source modulation unit 113. Furthermore, the encoding unit 112 outputs the generated encoding pattern data to the inner product calculation unit 117 as reference voltage data Ref, for the inner product calculation by the inner product calculation unit 117.

[0076] Subsequently, in step S43, the light source modulation unit 113 modulates the drive current for driving the light source 1 based on the encoded pattern data input from the encoding unit 112, and outputs the modulated drive current to the light emitting unit selected by the selection unit 111 via the switching unit 114. The light emitting unit emits laser light L0 in accordance with the input drive current.

[0077] Next, in step S44, the A / D conversion unit 115 converts the output signal Sen from the photodetection unit 3 that receives the laser light L2 reflected by the eyeball 30 from the focused laser light L1 into digital voltage data Dat and outputs it to the storage unit 116. The storage unit 116 stores the digital voltage data Dat input from the A / D conversion unit 115 for one period of the encoding pattern.

[0078] Next, in step S45, the inner product calculation unit 117 refers to the storage unit 116 to obtain digital voltage data Dat for one period of the encoding pattern, and calculates the inner product of this digital voltage data Dat and the reference voltage data Ref input from the encoding unit 112.

[0079] Subsequently, in step S46, the determination unit 118 determines whether or not the inner product value calculated by the inner product calculation unit 117 is greater than a predetermined threshold value.

[0080] If it is determined in step S46 that the value is greater than the threshold value (step S46, Yes), the processing unit 100 proceeds to step S47, and if it is determined that the value is not greater than the threshold value (step S46, No), the processing unit 100 proceeds to step S41.

[0081] When the process moves from step S46 to step S41, the selection unit 111 sequentially selects different ones of the eight light-emitting units. For example, if the selection unit 111 selects the first light-emitting unit 11 in the first step S41, the selection unit 111 selects the second light-emitting unit 12 in the second step S41. After proceeding up to the eighth step S41, the selection unit 111 returns to the first light-emitting unit 11 and selects the first light-emitting unit 11 again in the ninth step S41. This causes the eight light-emitting units included in the light source 1 to emit light sequentially.

[0082] Subsequently, in step S47, the estimation unit 119 estimates the gaze direction by calculation based on the position of the light-emitting unit of the light source 1 detected based on the output signal Sen by the light detection unit 3. Furthermore, if the light detection unit 3 is a PSD, the estimation unit 119 estimates the gaze direction by calculation based on the position of the light-emitting unit of the light source 1 detected based on the output signal Sen by the light detection unit 3 and the position of the reflected laser light L2 incident on the light detection unit 3, and acquires gaze direction information.

[0083] Subsequently, in step S48, the output unit 120 outputs the gaze direction information acquired by the estimation unit 119 to an external device via the output unit 120.

[0084] Subsequently, in step S49, the processing unit 100 determines whether or not to terminate the processing. For example, the gaze detection device 10 determines to terminate the processing when a termination operation by the user of the gaze detection device 10 is received via an operation unit or the like of the gaze detection device 10.

[0085] If it is determined in step S49 that the process is to be ended (step S49, Yes), the processing unit 100 ends the process. On the other hand, if it is determined that the process is not to be ended (step S49, No), the processing unit 100 returns to step S41 and performs the process from step S41 onwards again.

[0086] When the process moves from step S49 to step S41, the selector 111 selects a predetermined light-emitting unit, such as the first light-emitting unit 11, from among the eight light-emitting units as the initial state. In this case, the same light-emitting unit that was previously activated to emit light is activated. In other words, the same light-emitting unit emits light as long as the output signal Sen is detected.

[0087] In this way, the processing unit 100 can execute processing for gaze detection by the gaze detection device 10.

[0088] (Example of encoding pattern) Next, the encoding pattern will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating a Hadamard code as an example of the encoding pattern. Fig. 5(a) is a diagram illustrating the encoding pattern, and Fig. 5(b) is a diagram illustrating the inner product operation.

[0089] Generally, a square matrix whose elements are either 1 or -1 and whose row vectors are orthogonal is called a Hadamard matrix. As an example, an 8x8 Hadamard matrix H8 is shown in the following equation (1).

number

[0090] Fig. 5(a) shows an example of an 8-bit Hadamard coding pattern, illustrating eight Hadamard coding patterns designated #0 to #7. Fig. 5(b) shows an example of decoding a detection signal coded by an 8-bit Hadamard code. The horizontal axis of each graph shown in Fig. 5 represents time t, and the vertical axis represents the voltage V of either the digital voltage data Dat or the reference voltage data Ref.

[0091] 5(b), Dat denotes digital voltage data obtained by A / D converting the output signal Sen by the A / D conversion unit 115. Ref denotes reference voltage data. The reference voltage data Ref is Hadamard encoding pattern data selected by the encoding unit 112 and output to the inner product calculation unit 117.

[0092] Since each encoding pattern is orthogonal, when an inner product of the encoded digital voltage data Dat and the reference voltage data Ref is calculated, the inner product value is equal to or greater than 0 only if the digital voltage data Dat and the reference voltage data Ref are generated from the same encoding pattern. This property can be used to decode an encoded signal.

[0093] In the example shown in Fig. 5(b), when the digital voltage data Dat and the reference voltage data Ref are both the same Hadamard coding pattern #2, the inner product value is greater than or equal to 0. Otherwise, the digital voltage data Dat and the reference voltage data Ref are different, so the inner product value is 0.

[0094] In the gaze detection device 10, the light source modulation unit 113 time-modulates the laser light L0 emitted from each light emitting unit of the light source 1 in accordance with the Hadamard coding pattern selected by the coding unit 112.

[0095] At this time, the light source modulation unit 113 performs time modulation by turning on (turning on) the laser light L0 at the parts of the elements of the Hadamard coding pattern that are 1, and turning off (turning off) the laser light L0 at the parts of the elements of the Hadamard coding pattern that are -1. Turning off the laser light L0 corresponds to replacing the parts of the elements of the Hadamard coding pattern that are -1 with 0. Furthermore, by not using the first row of the Hadamard matrix (a row where all elements are 1) for coding, it is possible to remove noise whose time fluctuations are sufficiently slow compared to the modulation period.

[0096] In FIG. 5, the digital voltage data Dat in which -1 is replaced with 0 is displayed as a Hadamard coding pattern.

[0097] Here, even when replacing the -1 elements of the Hadamard coding pattern with 0, the reason why decoding is possible will be explained. N×N Hadamard matrix H N Let each row vector of be h k (where k is an integer satisfying 0 ≦ k < N), then the N×N Hadamard matrix H N can be expressed by the following equation (2). [Equation] Note that T in equation (2) represents the transpose matrix.

[0098] The optical intensity I of the laser beam L0 modulated with the Hadamard coding pattern in which the -1 elements are replaced with 0 can be expressed by the following equation (3). [Equation] Note that I0 in equation (3) represents the optical intensity of the laser beam L0. Also, in equation (3), patterns h other than h0 as the Hadamard coding pattern i (where i is an integer satisfying 1 ≦ i < N - 1) are used. Also, I nz represents the optical intensity of the noise light.

[0099] h m (where m is an integer satisfying i ≦ m < N - 1) is used as the reference signal. As a result of the inner product operation, it can be seen that the component of h0 in equation (3) becomes 0, and the inner product value becomes a value of 0 or more only when i = m. Thereby, the optical intensity I of the noise light nz can be removed and only the desired output signal Sen can be detected.

[0100] Also, when i = m, since the inner product operation hi·hm = N, the signal-to-noise ratio improves as the number of bits (number of elements) of the Hadamard coding pattern increases.

[0101] (Example of experimental results for detecting weak light) Next, Fig. 6A shows an example of the results of an experiment to detect weak light using a Hadamard coding pattern. In Fig. 6A, the horizontal axis represents the light intensity of the laser light emitted from the VCSEL as the light emitter, and the vertical axis represents the S / N ratio of the output signal from the PSD as the light detector. The S / N ratio is calculated as 10 × log (mean value / standard deviation).

[0102] In the detection experiment, the laser light emitted by the light-emitting part was Hadamard-encoded by repeatedly switching the VCSEL light emission on and off using digital voltage data from a PC. The VCSEL used had an oscillation wavelength of 940 nm.

[0103] The output signal from the PSD was A / D converted and input to a PC as digital voltage data. After the digital voltage data was accumulated for the elements of the Hadamard coding pattern, an inner product calculation was performed using software processing on the PC. One set of PSD detection and inner product calculation of the digital voltage data was performed 95 times, and the average and standard deviation of the inner product values ​​were calculated, and these were used to calculate the S / N ratio. A 16-bit coding pattern (1010101010101010) was used as the Hadamard coding pattern.

[0104] The horizontal axis of Figure 6A shows the optical intensity measured with a power meter just before the PSD, when the VCSEL was on and when it was off, and the difference between the output signal from the PSD and the output signal from the power meter was calculated. The origin of the horizontal axis corresponds to the result of the dot product calculation when the VCSEL was off. The value measured by the power meter when the VCSEL was off was approximately 35 μW. In other words, the optical intensity under indoor lighting conditions was approximately 35 μW.

[0105] The results in Figure 6A show that laser light with a light intensity of about 1 μW can be detected with high sensitivity under an indoor lighting environment of about 35 μW. More specifically, for laser light with a light intensity of 1.4 μW, the average value of the dot product was found to be about 45 times higher than the standard deviation.

[0106] (Example of gaze detection experiment results) 6B is a diagram showing an example of the results of an experiment to detect the gaze direction using a Hadamard coding pattern. In Fig. 6B, the horizontal and vertical axes respectively represent the X and Y coordinates of the beam spot position output by the photodetector 3.

[0107] The output signal from the photodetector 3 when the horizontal tilt θx of the eyeball model was changed was A / D converted and input to a PC (Personal Computer) as digital voltage data. After the digital voltage data for the elements of the Hadamard coding pattern was accumulated, an inner product calculation was performed using software processing by the PC. The horizontal tilt θx of the eyeball model was changed in 2° increments within a range of ±4°, with 0° representing the state in which the eyeball 30 is facing forward. Figure 6B shows the results when the tilt θx was -4°, -2°, 0°, 2°, and 4°.

[0108] The experimental method for FIG. 6B is as follows. (1) First, measurements were performed in a darkroom environment without Hadamard encoding (black circle plot in Figure 6B). (2) Next, measurements were performed without Hadamard encoding under an environment where a halogen lamp was used as background light (plots marked with "x" in Figure 6B). (3) Finally, measurements were performed using Hadamard encoding under an environment where a halogen lamp was used as background light (plots indicated by open circles in Figure 6B).

[0109] The optical intensity of the signal light was approximately 1 [μW], and the optical intensity of the halogen lamp was approximately 150 [μW]. Furthermore, a 16-bit coding pattern (1001100110011001) was used as the Hadamard coding pattern.

[0110] As shown in Figure 6B, in a darkroom environment where the background light was negligible, the coordinate values ​​indicated by the output signal of the light detection unit 3 changed in response to the tilt of the eyeball. However, when background light was irradiated, the coordinate values ​​indicated by the output signal of the light detection unit 3 did not change in response to the tilt of the eyeball. Therefore, when the signal light was coded using a Hadamard pattern, the coordinate values ​​indicated by the output signal of the light detection unit 3 changed in response to the tilt of the eyeball 30, even when irradiated by a halogen lamp. This result showed that the tilt of the eyeball 30 can be measured with high accuracy while maintaining the intensity of the signal light at a low level, even in an environment where the intensity of the optical signal is 1 / 150 of the background light intensity.

[0111] (Label processing example) Next, we will explain the label processing by the estimation unit 119. The estimation unit 119 can also decode the output signal Sen from the light detection unit 3 using a label corresponding to the position information of the light-emitting unit that emitted the laser light L0 to be irradiated onto the eyeball 30 as the focused laser light L1, among the multiple light-emitting units included in the light source 1, and the encoding pattern of the laser light L0.

[0112] In this way, the position of the light-emitting unit is detected from the inner product value based on the output signal Sen without inputting the coded pattern data from the coding unit 112. The processing using the labels by this estimation unit 119 will be described below.

[0113] The reference voltage data Ref is expressed by the following equations (4) and (5).

number

number

[0114] L kis used to provide the final output result for the output signal Sen. More specifically, when the dot product of the digital voltage data Dat based on the output signal Sen expressed in equation (3) and the reference voltage data Ref expressed in equation (4) is calculated, the following equation (6) is obtained. Similarly, when the dot product of the digital voltage data Dat based on the output signal Sen and the reference voltage data Ref expressed in equation (5) is calculated, the following equation (7) is obtained.

number

number

[0115]

number

[0116] Here, Fig. 7 is a diagram for explaining an example of labeled decoding processing. In Fig. 7, an explanation will be given using #6 (1, -1, 1, -1, -1, 1, -1, 1) as the Hadamard coding pattern of an 8-bit Hadamard code (see Fig. 4).

[0117] After the output signal Sen from the light detection unit 3 is A / D converted by the A / D conversion unit 115, the estimation unit 119 performs labeled decoding processing. The output result is expressed as L kTherefore, when expressed in binary, the bit corresponding to the number of the Hadamard coding pattern is 1. This allows the position of the light emitting element to be detected as the position where the bit is 1.

[0118] (Actions and Effects of the Line-of-Sight Detection Device 10) Next, the effects of the line of sight detection device 10 will be described.

[0119] Recently, technologies related to "Augmented Reality (AR)" and "Virtual Reality (VR)" have been attracting attention. AR technology in particular is expected to be applied in a wide range of fields, not only for entertainment purposes but also for work support in manufacturing and medical settings.

[0120] Glasses-type image display devices have been developed as devices for displaying AR. Among these, "retinal projection displays" that project images directly onto the retina have the advantage of eliminating the need to focus the eyes on the projected image. This allows users to view clear AR images at all times while focusing their eyes on objects in the outside world. Furthermore, by combining this with "eye tracking" technology that tracks the user's line of sight, it becomes possible to widen the display's field of view and control the displayed image with the user's gaze.

[0121] A gaze detection device has been disclosed that estimates the gaze direction of an eyeball from the position of the light reflected by the eyeball of irradiated laser light. In this configuration, from the viewpoint of safety, it is preferable that the light intensity of the laser light irradiated onto the eyeball is weak. Therefore, in order to detect the position of the reflected light with a high signal-to-noise ratio, a lock-in detector and a modulator that modulates the light emitted by a light-emitting unit are used.

[0122] However, in conventional configurations, lock-in detectors and modulators are used in the same number as the number of light-emitting units, so if the number of light-emitting units is increased to expand the range in which gaze detection is possible, the gaze detection device may become larger and consume more power.

[0123] Furthermore, since a lock-in detector separates signals derived from time-modulated light by frequency, a high frequency resolution is desirable for highly accurate detection. However, the detection time increases depending on the frequency resolution, which can make it difficult to quickly detect the line of sight direction.

[0124] Furthermore, Patent Document 1 discloses a configuration for separating and detecting a plurality of weak optical signals using the orthogonality of a pseudo-random sequence including a Hadamard code in order to separate and detect a plurality of weak optical signals without being affected by ambient noise.

[0125] However, in the configuration of Patent Document 1, the laser light emitted from multiple light-emitting elements is irradiated in parallel, which increases the amount of light irradiated onto an object such as an eyeball, leaving room for improvement in terms of safety for the object and light utilization efficiency.

[0126] In this embodiment, the device includes a light source 1 (plurality of light-emitting units) capable of emitting laser light L0 (light) time-modulated by a code having orthogonality, a light detection unit 3 that outputs a signal corresponding to reflected laser light L2 (reflected light) of the focused laser light L1 by an eyeball 30 (object) irradiated with focused laser light L1 based on the laser light L0, and an output unit 120 that outputs gaze direction information (information indicating the inclination of the object) obtained based on the output signal Sen from the light detection unit 3.

[0127] For example, codes having orthogonality include Hadamard codes.

[0128] Furthermore, the light source 1 includes a first light-emitting unit 11 and a second light-emitting unit 12, and the timing at which the first light-emitting unit 11 emits the laser light L0 is different from the timing at which the second light-emitting unit 12 emits the laser light L0.

[0129] As a result, the focused laser light L1 based on the laser light L0 emitted by the multiple light-emitting elements is not irradiated parallel to the eyeball 30, thereby suppressing an increase in the light intensity of the focused laser light L1 irradiated to the eyeball 30, and suppressing a decrease in safety for the eyeball 30 and a decrease in the efficiency of light utilization in the line of sight.

[0130] Furthermore, in this embodiment, the gaze direction is estimated based on a signal corresponding to the laser light L2 reflected by the eyeball 30 of the focused laser light L1 based on the laser light L0 time-modulated by a code having orthogonality, and gaze direction information acquired by the estimation is output. This makes it possible to improve the signal-to-noise ratio of the signal by removing the influence of light from the external environment, and to detect the weak laser light L2 reflected by the eyeball 30 with high sensitivity. As a result, the laser light L2 reflected by the eyeball 30 can be detected with high sensitivity while improving safety for the eyeball 30 and light utilization efficiency.

[0131] Furthermore, in this embodiment, the output unit 120 outputs gaze direction information acquired based on an inner product calculation of the output signal Sen from the light detection unit 3 and the coded pattern of the laser light L0 emitted by the light source 1. This allows noise to be reduced through a simple calculation, and the laser light L2 reflected by the eyeball 30 to be detected with high sensitivity.

[0132] Furthermore, in this embodiment, the light detection unit 3 can output position information at which the laser light L2 reflected by the eyeball 30 is incident on the light detection unit 3. This allows the gaze direction to be detected with higher accuracy based on the position information of the light-emitting units among the multiple light-emitting units and the position information of the laser light L2 reflected by the eyeball 30.

[0133] Furthermore, the output signal Sen from the light detection unit 3 may be decoded using a label corresponding to position information of the light-emitting unit that emitted the laser light L0 among the multiple light-emitting units and the coding pattern of the laser light L0 emitted by the light-emitting unit. In this way, the position of the light-emitting unit can be detected from the inner product value based on the output signal Sen from the light detection unit 3, without inputting the coding pattern data from the coding unit 112. As a result, a configuration for irradiating laser light and a configuration for receiving laser light can be provided separately, further simplifying the configuration of the processing unit 100.

[0134] In the present embodiment, the configuration in which the focused laser light L1 by the concave mirror 2 is irradiated onto the eyeball 30 has been exemplified, but the present invention is not limited to this. The laser light L0 emitted by the light source 1 may be directly irradiated onto the eyeball 30, or the laser light L0 may be converted using an optical element other than the concave mirror 2 and the converted laser light may be irradiated onto the eyeball 30.

[0135] [Second embodiment] Next, a gaze detection device 10a according to a second embodiment will be described. Note that the same components as those described in the above embodiments are given the same part numbers, and duplicated descriptions will be omitted as appropriate.

[0136] In the first embodiment described above, there is a great degree of freedom in the selection of the light-emitting unit by the selection unit 111 and the selection of the coding pattern by the coding unit 112. As an example, there is a method of selecting the light-emitting unit and the coding pattern by cyclically selecting the coding patterns shown in equation (1).

[0137] However, this method is not necessarily efficient, since the probability of finding the selected coding pattern in one process is 1 / N, where N is the number of bits that make up the coding pattern.

[0138] Therefore, in this embodiment, from among the multiple light-emitting units, a light-emitting unit that emits light is selected based on a light-emitting probability distribution according to the frequency of use of the light-emitting unit, and from among the multiple coding patterns that encode light, a coding pattern is selected based on a coding probability distribution according to the frequency of use of the coding pattern.

[0139] This reduces the frequency of light emission that does not involve the detection of reflected light from an object, improves the efficiency of light emitted by multiple light-emitting elements, and eliminates unnecessary detection of the light-emitting element positions, thereby increasing the speed of line-of-sight detection.

[0140] 8 is a block diagram showing an example of the functional configuration of a processing unit 100a included in the gaze detection device 10a. As shown in FIG. 8, the processing unit 100a includes a storage unit 121, a probability selection unit 122, a probability encoding unit 123, and an update unit 124.

[0141] Of these, the function of the storage unit 121 is realized by the SSD 104 in Figure 2, etc., and the functions of the probability selection unit 122, the probability encoding unit 123, and the update unit 124 are realized by the CPU 101 executing a predetermined program stored in either the ROM 102 or the SSD 104 in Figure 2, etc.

[0142] The storage unit 121 stores information on a light emission probability distribution P1(A) and an encoding probability distribution P2(A). The light emission probability distribution P1(A) is usage frequency information of a light emission unit that was used when the determination unit 118 determined that the light detection unit 3 received light in a past gaze detection by the gaze detection device 10a. Similarly, the encoding probability distribution P2(A) is usage frequency information of a coding pattern that was used when the determination unit 118 determined that the light detection unit 3 received light in a past gaze detection by the gaze detection device 10a.

[0143] In gaze detection by the gaze detection device 10a, if the determination unit 118 determines that the light detection unit 3 is not receiving light, the probability selection unit 122 selects a light-emitting unit that emits laser light L0 based on random numbers that follow the emission probability distribution P1(A), and the probability encoding unit 123 selects a coding pattern that encodes the laser light L0 based on random numbers that follow the encoding probability distribution P2(A). As a result, light-emitting units and coding patterns that are used frequently are preferentially selected.

[0144] The update unit 124 updates the light emission probability distribution P1(A) and the encoding probability distribution P2(A) after the probability selection unit 122 and the probability encoding unit 123 have selected the light emission unit and the encoding pattern, respectively. The update method may be a method according to a frequency distribution of a predetermined number of times, or a method of updating the probability distribution using Bayes' theorem. Furthermore, it is preferable to assign a finite low probability to light emission units and encoding patterns that occur infrequently, so that the gaze direction is not lost for a long period of time when the gaze direction changes instantaneously.

[0145] Fig. 9 is a flowchart showing an example of processing by the processing unit 100a. The processing in steps S93 to S98 in Fig. 9 is the same as the processing in steps S43 to S48 in Fig. 4. The processing in step S100 in Fig. 9 is the same as the processing in step S99 in Fig. 4. Therefore, overlapping explanations will be omitted here, and the explanation will focus on the differences.

[0146] First, in step S91, the probability selection unit 122 selects a light-emitting unit based on a random number according to the light-emitting probability distribution P1(A) stored in the storage unit 121.

[0147] Subsequently, in step S92, the stochastic encoding unit 123 selects an encoding pattern based on random numbers according to the encoding probability distribution P2(A) stored in the storage unit 121.

[0148] In addition, in step S99, the update unit 124 performs update processing on each of the light emission probability distribution P1(A) and the encoding probability distribution P2(A).

[0149] In this way, the processing unit 100a can execute processing for gaze detection by the gaze detection device 10a.

[0150] (Functions and Effects of the Line-of-Sight Detection Device 10a) As described above, in this embodiment, the probability selection unit 122 selects a light-emitting unit that emits laser light L0 from among the multiple light-emitting units included in the light source 1, based on the light emission probability distribution P(A) in accordance with the frequency of use of the light-emitting units. Also, the probability encoding unit 123 selects a coding pattern from among the multiple coding patterns for encoding the laser light L0, based on the coding probability distribution P2(A) in accordance with the frequency of use of the coding pattern.

[0151] This reduces the frequency of light emission that does not involve the detection of reflected light from an object, improving the efficiency of light emitted by multiple light-emitting elements. It also eliminates unnecessary detection of the positions of the light-emitting elements, and increases the speed of line-of-sight detection.

[0152] [Third embodiment] Next, a retinal projection display device 60 according to a third embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram illustrating an example of the configuration of the retinal projection display device 60.

[0153] As shown in FIG. 10, the retinal projection display device 60 includes an RGB (Red, Green, Blue) laser light source 61, a scanning mirror 62, a plane mirror 63, a half mirror 64, an image generation unit 65, and a gaze detection device 10.

[0154] The RGB laser light source 61 modulates and outputs laser light of three RGB colors over time. The scanning mirror 62 scans the light from the RGB laser light source 61 two-dimensionally. The scanning mirror 62 is a MEMS mirror or the like. However, it is not limited to this, and any other mirror having a reflective portion for scanning light, such as a polygon mirror or a galvanometer mirror, can be used. The MEMS mirror is advantageous in terms of size and weight reduction. The driving method of the MEMS mirror may be any of electrostatic, piezoelectric, electromagnetic, etc.

[0155] The plane mirror 63 reflects the scanning light emitted by the scanning mirror 62 toward the half mirror 64. The half mirror 64 transmits a portion of the incident light and reflects a portion of it toward the eyeball 30. The half mirror 64 has a concave curved surface and focuses the reflected light near the pupil 31 of the eyeball 30, forming an image at the position of the retina 33. In this way, the image formed by the scanning light is projected onto the retina 33. Light 61a indicated by a dashed line in the figure represents light that forms an image on the retina 33. It should be noted that the light amounts of the reflected light and transmitted light of the half mirror 64 do not necessarily have to be one to one.

[0156] The gaze detection device 10 transmits a feedback signal of the inclination of the eyeball 30, that is, the gaze direction, to the image generation unit 65.

[0157] The image generation unit 65 has a function of controlling the deflection angle of the scanning mirror 62 and a function of controlling the light emission of the RGB laser light source 61. The image generation unit 65 also receives a feedback signal of the gaze direction from the gaze detection device 10. According to the gaze direction detected by the gaze detection device 10, the image generation unit 65 controls the deflection angle of the scanning mirror 62 and the light emission of the RGB laser light source 61, and rewrites the projection angle of the image or the image content. This makes it possible to form an image on the retina 33 that follows changes in the gaze direction (eye tracking).

[0158] In this embodiment, an example has been shown in which the retinal projection display device 60 is a head-mounted display, which is a wearable terminal, but the retinal projection display device 60 may not only be worn directly on a person's head, but may also be worn indirectly on a person's head via a member such as a fixing part (a head-mounted display device). Also, a binocular retinal projection display device may be used in which a pair of retinal projection display devices 60 is provided for the left and right eyes.

[0159] Furthermore, in this embodiment, the retinal projection display device 60 is illustrated as including the gaze detection device 10, but the retinal projection display device 60 may also be provided with a gaze detection device 10a.

[0160] Although examples of embodiments of the present invention have been described above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

[0161] For example, in the above-described embodiment, the device for detecting the tilt of the eyeball 30 is shown as an example of an optical device, but the present invention is not limited to this. For example, an optical device may be mounted on a robot hand or the like to detect the tilt of the robot hand, which is an example of an object.

[0162] The present invention can also be adopted in optometry devices capable of detecting the tilt of the eyeball and pupil position (cornea). An optometry device is a device capable of performing various tests, such as visual acuity tests, eye refraction tests, intraocular pressure tests, and axial length tests. An optometry device is a device capable of performing non-contact tests on the eyeball. It comprises a support unit that supports the subject's face, an examination window, a display unit that displays information to keep the subject's eyeball (gaze direction) constant during the eye examination, a control unit, and a measurement unit. To improve the measurement accuracy of the measurement unit, the subject is required to gaze at a single point without moving the eyeball (gaze direction). The subject fixes their face on the support unit and gazes at an object displayed on the display unit through the examination window. The eyeball tilt position detection device of this embodiment can be used to detect the tilt position of the eyeball. The eyeball tilt position detection device is positioned to the side of the measurement unit so as not to interfere with the measurement. The eyeball tilt position (gaze) information obtained by the eyeball tilt position detection device can be fed back to the control unit, allowing measurements to be made according to the eyeball tilt position information.

[0163] The gaze detection device 10 can also be applied to a user state estimation device that estimates the state of a user based on information about the tilt of the eyeball 30, the pupil position (cornea), or the gaze direction. The user refers to a person using the user state estimation device.

[0164] The user's state includes at least one of the user's fatigue level and the user's attention level. The user's fatigue level is an index that indicates, for example, the degree of mental fatigue of the user. The user's attention level is an index that indicates the level of the user's attention.

[0165] For example, a user state estimation device that estimates a user's fatigue level includes a gaze detection device 10 and a fatigue level estimation unit that estimates the fatigue level based on gaze direction information of the user detected by the gaze detection device 10. The fatigue level estimation unit is an example of a state estimation unit.

[0166] An example of a method for estimating a user's mental fatigue level using the fatigue level estimation unit is described in non-patent literature (Tseng, V. W. S., Valliappan, N., Ramachandran, V. et al. Digital biomarker of mental fatigue. npj Digit. Med. 4, 47 (2021)). This method involves performing a task of tracking the trajectory of an object displayed on a monitor for several minutes, and measuring the user's gaze movements during that time to estimate the user's mental fatigue level. The gaze detection device 10 can detect reflected light from an object with high sensitivity while suppressing an increase in the amount of light irradiated onto the object. Therefore, a fatigue level estimation device including the gaze detection device 10 can safely and accurately estimate a user's mental fatigue level. The fatigue level estimation device may also include a notification means for notifying the user of information, such as a prompt to take a break, based on the estimated mental fatigue level.

[0167] Furthermore, a user state estimation device that estimates a user's attention level includes a gaze detection device 10 and an attention level estimation unit that estimates a user's attention level based on gaze direction information detected by the gaze detection device 10. The attention level estimation unit is an example of a state estimation unit.

[0168] One example of a method for estimating a user's level of attention in the attention level estimation unit is to detect microsaccades, which are small vibrations of the eyeball 30, and estimate the user's level of attention based on their frequency. According to a non-patent document (Pastukhov A, Braun J. Rare but precious: microsaccades are highly informative about attentional allocation. Vision Res. 2010 Jun 11;50(12):1173-84), microsaccades are relatively large-amplitude, high-speed movements among fixational eye movements (small vibrations of the eyeball 30 with an amplitude of approximately ±3.0° that occur when a person gazes at an object), and are known to be correlated with a person's level of attention. The gaze detection device 10 can measure the tilt of the eyeball 30 quickly and accurately, and therefore can detect microsaccades with higher accuracy than conventional gaze detection devices.

[0169] Therefore, a user state estimation device that estimates a user's level of attention can detect reflected light from an object with high sensitivity while suppressing an increase in the amount of light irradiated onto the object, thereby enabling safe and highly accurate estimation of a user's level of attention.

[0170] Furthermore, a user state estimation device having the gaze detection device 10 can also be applied to a driving assistance system. This driving system includes a user state estimation device having the gaze detection device 10 and an operation control unit that controls the operation of a mobile object based on the level of attention estimated by the user state estimation device. For example, if the level of attention of the user estimated by the user state estimation device falls below a predetermined standard, the operation control unit controls the operation mode of the mobile object, such as a vehicle, to switch from manual driving mode to automatic driving mode. The gaze detection device 10 can detect reflected light from an object with high sensitivity while suppressing an increase in the amount of light irradiated onto the object, allowing the driving assistance system to provide safe and highly accurate driving assistance.

[0171] Furthermore, these embodiments may be configured such that two or more image generating units and a state estimating unit that estimates the state of the user each use one piece of information related to the tilt of the eyeball 30, the pupil position (cornea), or the gaze direction detected by the gaze detection device 10. With this configuration, it is possible to detect the light reflected by the eyeball 30 with high sensitivity while suppressing an increase in the amount of light irradiated onto the user's eyeball 30, and it is also possible to miniaturize the gaze detection device 10.

[0172] For example, the tilt of the eyeball 30, the pupil position (cornea), or the gaze direction information detected by the gaze detection device 10 may be used as a feedback signal for the image generation unit of the retinal projection display device, and may also be used for fatigue estimation by the fatigue estimation unit of the fatigue estimation device. In this case, the image generation unit and the fatigue estimation unit as functional components may be provided in the same information processing device, or may be provided in separate information processing devices.

[0173] The embodiments also include a method for detecting the tilt of a three-dimensional object. For example, the method for detecting the tilt of a three-dimensional object includes the steps of: emitting light from a plurality of light-emitting units capable of irradiating light onto an object; detecting the light reflected by the object and outputting a signal based on the reflected light using a detection unit; and outputting tilt information of the object obtained based on the output signal output from the detection unit. The light emitted from the light-emitting units is time-modulated using an orthogonal code, and the plurality of light-emitting units include a first light-emitting unit and a second light-emitting unit, and the timing of light emission by the first light-emitting unit is different from the timing of light emission by the second light-emitting unit. This method for detecting the tilt of a three-dimensional object can achieve the same effect as the above-mentioned gaze detection device.

[0174] The embodiments also include a gaze detection method. For example, the gaze detection method includes a step of emitting light from a plurality of light-emitting units capable of irradiating a target object, a step of detecting the light reflected by the target object and outputting a signal based on the reflected light using a detection unit, and a step of outputting tilt information of the target object obtained based on the output signal output from the detection unit, wherein the light emitted from the light-emitting units is time-modulated using an orthogonal code, the plurality of light-emitting units include a first light-emitting unit and a second light-emitting unit, and the timing of light emission by the first light-emitting unit is different from the timing of light emission by the second light-emitting unit. Such a gaze detection method can achieve the same effect as the gaze detection device described above.

[0175] Furthermore, all ordinal numbers, quantitative numbers, etc. used above are merely examples for specifically explaining the technology of the present invention, and the present invention is not limited to the exemplified numbers. Furthermore, the connection relationships between the components are merely examples for specifically explaining the technology of the present invention, and the connection relationships for realizing the functions of the present invention are not limited to these.

[0176] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. [Explanation of symbols]

[0177] 1. Light source (an example of multiple light-emitting units) 2 Concave mirror 3. Light detection unit (example of detection unit) 4 Optical system support 4a ball joint 10. Eye gaze detection device (an example of an optical device) 11 First light-emitting part 12 Second light-emitting part 18 8th light-emitting part 20 Spectacle-shaped support 21 Eyeglass lenses 22 eyeglass frames 30 Eyeball (example of object) 31 Pupil 32 Cornea 100 Processing section 111 Selection section 112 Encoding section 113 Light source modulation section 114 Switching section 115 A / D conversion section 116 Storage Unit 117 Inner product calculation unit 118 Judgment section 119 Estimation Department 120 Output section 121 Storage area 122 Probability Selection Section 123 Probability coding unit L0 laser light L1 Focused laser light L2 Reflected laser light (an example of reflected light) P1(A) Emission probability distribution P2(A) coding probability distribution Sen output signal Dat Digital voltage data Ref Reference voltage data [Prior art documents] [Patent documents]

[0178] [Patent Document 1] Patent No. 3623743

Claims

1. A plurality of light-emitting units capable of irradiating an object with light that has been time-modulated by an orthogonal Hadamard coding pattern; a detection unit that detects the light reflected by the object and outputs a signal based on the reflected light; an output unit that outputs tilt information of the object acquired based on the output signal output from the detection unit; an encoding unit that selects and outputs the Hadamard encoding pattern; an inner product calculation unit that calculates an inner product of the digital voltage data for one period of the Hadamard coding pattern and the reference voltage data input from the coding unit; a determination unit that determines whether or not the detection unit has received reflected light from the object irradiated with the light emitted by the plurality of light-emitting units, based on an inner product value that is the inner product calculation result by the inner product calculation unit; and a probability selection unit that selects, from the plurality of light-emitting units, a light-emitting unit that emits the light based on a light emission probability distribution according to a usage frequency of the light-emitting unit that was used when the determination unit determined that the detection unit received light; and a stochastic encoding unit that selects a Hadamard coding pattern from the plurality of Hadamard coding patterns that encode the light, based on an encoding probability distribution according to a frequency of use of the Hadamard coding pattern that was used when the determination unit determined that the detection unit received light, the plurality of light-emitting units include a first light-emitting unit and a second light-emitting unit, the timing of emitting the light by the first light-emitting unit is different from the timing of emitting the light by the second light-emitting unit; An optical device, wherein the Hadamard coding pattern does not have all elements of 1, but includes both 1 and -1.

2. 2. The optical device according to claim 1, wherein the output unit outputs tilt information of the object estimated based on an inner product operation of the output signal and the Hadamard coding pattern.

3. 3. The optical device according to claim 1, wherein the detector is capable of outputting a signal indicating a position of the reflected light incident on the detector.

4. 4. The optical device according to claim 1, wherein the output signal is decoded using a label corresponding to information indicating the position of the light-emitting element that emitted the light among the plurality of light-emitting elements and the Hadamard coding pattern.

5. An optical device according to any one of claims 1 to 4, the object is the user's eyeball, The optical device is a gaze detection device that detects the inclination of the user's eyeball as the user's gaze direction.

6. A retinal projection display device having the line of sight detection device according to claim 5.

7. A head-mounted display device comprising the gaze detection device according to claim 5.

8. An optometry apparatus comprising the gaze detection device according to claim 5.

9. The gaze detection device according to claim 5, a state estimation unit that estimates the state of the user based on information about the gaze direction detected by the gaze detection device.

10. The user state estimation device according to claim 9 , wherein the state estimation unit estimates the state of the user based on a frequency of occurrence of micro-vibrations of the user's eyeballs.

11. The user state estimation device according to claim 9 , wherein the state of the user includes at least one of a fatigue level of the user and an alertness level of the user.

12. A user state estimation device according to any one of claims 9 to 11; an operation control unit that controls the operation of a mobile object driven by the user based on the state of the user estimated by the user state estimation device.

13. A process in which a plurality of light-emitting units are capable of irradiating an object with light that has been time-modulated by an orthogonal Hadamard coding pattern; detecting the light reflected by the object and outputting a signal based on the reflected light by a detection unit; outputting tilt information of the object obtained based on the output signal output from the detection unit; an encoding step of selecting and outputting the Hadamard coding pattern; an inner product calculation step of calculating an inner product of the digital voltage data for one period of the Hadamard coding pattern and the reference voltage data obtained in the coding step; a determination step of determining whether or not the detection unit has received reflected light from the object irradiated with the light emitted by the plurality of light-emitting units, based on an inner product value that is a result of the inner product calculation in the inner product calculation step; selecting, from the plurality of light-emitting units, a light-emitting unit that emits the light based on a light emission probability distribution according to a usage frequency of the light-emitting unit that was used when it was determined in the determination step that the detection unit received light; selecting a Hadamard coding pattern from the plurality of Hadamard coding patterns that encode the light based on a coding probability distribution according to a frequency of use of the Hadamard coding pattern that was used when it was determined in the determination step that the detection unit received light; Including, the plurality of light-emitting units include a first light-emitting unit and a second light-emitting unit, the timing of emitting the light by the first light-emitting unit is different from the timing of emitting the light by the second light-emitting unit; The object tilt detection method is characterized in that the Hadamard coding pattern does not have all elements of 1, but includes both 1 and -1.

14. A step of emitting light from a plurality of light-emitting units capable of irradiating an eyeball with light that has been time-modulated by an orthogonal Hadamard coding pattern; detecting the light reflected by the eyeball and outputting a signal based on the reflected light by a detection unit; outputting the eyeball tilt information acquired based on the output signal output from the detection unit; an encoding step of selecting and outputting the Hadamard coding pattern; an inner product calculation step of calculating an inner product of the digital voltage data for one period of the Hadamard coding pattern and the reference voltage data obtained in the coding step; a determination step of determining whether or not the detection unit has received reflected light from the eyeball irradiated with the light emitted by the plurality of light-emitting units, based on an inner product value that is a result of the inner product calculation in the inner product calculation step; selecting, from the plurality of light-emitting units, a light-emitting unit that emits the light based on a light emission probability distribution according to a usage frequency of the light-emitting unit that was used when it was determined in the determination step that the detection unit received light; selecting a Hadamard coding pattern from the plurality of Hadamard coding patterns that encode the light based on a coding probability distribution according to a frequency of use of the Hadamard coding pattern that was used when it was determined in the determining step that the detection unit received light, the plurality of light-emitting units include a first light-emitting unit and a second light-emitting unit, the timing of emitting the light by the first light-emitting unit is different from the timing of emitting the light by the second light-emitting unit; The gaze detection method is characterized in that the Hadamard coding pattern does not have all elements of 1, but includes both 1 and -1.

Images