Line-of-sight detection device and camera device

Abstract

The present disclosure relates to a line-of-sight detection apparatus and an imaging apparatus. The line-of-sight detection apparatus includes: an acquisition component configured to acquire an eye image; an irradiation component configured to irradiate an eye with light in an infrared wavelength band to form a plurality of point-like optical images on the eye image; and a calculation component configured to calculate line-of-sight information based on the eye image and the plurality of point-like optical images, wherein the irradiation component includes: at least one single light source configured to emit light in the infrared wavelength band; and a division element configured to divide the light emitted from the single light source into a plurality of light beams of emitted light to form the plurality of point-like optical images.

Description

Technical Field

[0001] This invention relates to a line-of-sight detection device capable of detecting line of sight. Background Technology

[0002] Gaze detection technology has been studied and practical progress has been made in fields such as gaze detection for drivers of motor vehicles, research on human gaze behavior, and technologies for supporting people with disabilities. The corneal reflection method is one of several proposed methods for gaze detection and is advantageous in terms of accuracy and miniaturization. The corneal reflection method is a method for calculating gaze information based on near-infrared eye images and is based on a Purkinje image arrangement within the eye image.

[0003] To obtain visual information using corneal reflection methods, multiple Purkinje images need to be formed within the eye image. For example, in Japanese Patent Application Publication No. 2019-139743, this is achieved by arranging multiple infrared light sources on the glasses.

[0004] However, although a smaller and more accurate device is needed for line-of-sight detection technology, it is difficult to achieve such a smaller and more accurate device using known technologies due to the need for a large number of light sources and the low degree of freedom in the arrangement of the light sources. Summary of the Invention

[0005] The present invention was made in view of the above-mentioned problems, and provides a line of sight detection device with small size and capable of detecting line of sight with high precision.

[0006] According to a first aspect of the present invention, a gaze detection device is provided, comprising: an acquisition component configured to acquire an eye image; an illumination component configured to illuminate the eye with light in the infrared band to form a plurality of dot-shaped optical images on the eye image; and a calculation component configured to calculate gaze information based on the eye image and the plurality of dot-shaped optical images, wherein the illumination component comprises: at least one single light source configured to emit light in the infrared band; and a segmentation element configured to segment the light emitted from the single light source into a plurality of beams of emitted light to form the plurality of dot-shaped optical images.

[0007] According to a second aspect of the present invention, a camera device is provided, comprising: a camera component configured to capture an image of a subject; a display device configured to display the image captured by the camera component; and the aforementioned gaze detection device.

[0008] According to a third aspect of the present invention, a gaze detection device is provided, comprising: an acquisition component configured to acquire an eye image; an illumination component configured to illuminate the eye with light in the infrared band to form a plurality of dot-shaped optical images on the eye image; and a calculation component configured to calculate gaze information based on the eye image and the plurality of dot-shaped optical images, wherein the illumination component is a micro-light source array comprising a micro-light source arranged in a two-dimensional array.

[0009] According to a fourth aspect of the present invention, a camera device is provided, comprising: a camera component configured to capture an image of a subject; a display device configured to display the image captured by the camera component; and the aforementioned gaze detection device.

[0010] Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1A This is a schematic diagram illustrating the configuration of a line-of-sight detection device according to a first embodiment of the present invention.

[0012] Figure 1B This is a schematic diagram illustrating the configuration of a line-of-sight detection device according to a modified example of the first embodiment.

[0013] Figure 2 This is a schematic diagram illustrating an example of an eyeball image.

[0014] Figure 3 This is a diagram illustrating an example of a known line-of-sight detection device.

[0015] Figure 4 This is a schematic diagram illustrating an example of an eyeball image according to the first embodiment.

[0016] Figure 5A This is a schematic diagram showing an image of the eyeball used to describe individual differences in human vision.

[0017] Figure 5B This is a schematic diagram showing an image of the eyeball used to describe individual differences in human vision.

[0018] Figure 6 This is a schematic diagram illustrating the Purkinje image, which is difficult to reduce individual differences in the human eye.

[0019] Figure 7A This is a schematic diagram showing a Purkinje image in the form of a two-dimensional array on an eyeball image.

[0020] Figure 7B This is a schematic diagram showing a Purkinje image in the form of a two-dimensional array on an eyeball image.

[0021] Figure 8 This is a schematic diagram illustrating the configuration of a gaze detection device according to a third embodiment.

[0022] Figure 9 This is a schematic diagram illustrating the configuration of the gaze detection device according to the fourth embodiment.

[0023] Figure 10 This is a schematic diagram illustrating the configuration of a camera device according to the fifth embodiment. Detailed Implementation

[0024] In the following description, embodiments will be presented in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but the invention is not limited to requiring all such features, and multiple such features can be suitably combined. Furthermore, in the drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions thereof are omitted.

[0025] First Embodiment

[0026] The first embodiment of the present invention will now be described.

[0027] Line of sight detection equipment

[0028] Figure 1 is a diagram illustrating the configuration of a first embodiment of the gaze detection device of the present invention. In this embodiment, a configuration is employed that identifies the gaze position of the eyeball 102a by using a corneal reflection method.

[0029] The eyeball 102a includes an iris portion 102b in the sclera (white part) of the eye, wherein the iris portion 102b includes a pupil. The gaze detection device 101 of this embodiment includes a single light source 101e, wherein the single light source 101e is configured to illuminate the surface of the eyeball (including the surface portion of the cornea) with near-infrared light 101f. The light 101f emitted from the single light source 101e is split into light in two directions by an illumination light splitting element 101g and emitted onto the surface of the eyeball 102a.

[0030] The light-splitting element 101g is not limited to a specific element, and any element that splits light, such as a diffractive optical element (DOe) using optical interference or a micro-scanner element, can be used.

[0031] exist Figure 1A In the configuration shown, light emitted onto the surface of the eyeball is partially reflected by the eyeball surface, then further reflected by the reflecting mirror 101c, and reaches the image sensor 101a, which serves as an eyeball image acquisition unit, through the imaging lens 101b. The point illumination unit of this embodiment includes a single light source 101e and an illumination light splitting element 101g.

[0032] The gaze information includes the direction / angle of the visual axis, the position of the gaze across the screen in the area of ​​the screen facing the eye, etc. The gaze information is obtained from the gaze information calculation unit 103 based on the eye image acquired by the image sensor 101a.

[0033] In this embodiment, the reflector 101c is a dichroic mirror that reflects near-infrared light and transmits visible light, thereby enabling the subject to see the scene through the mirror. However, this embodiment is not necessarily limited to configurations involving the use of dichroic mirrors for wavelength selection. Figure 1A The configuration shown should not be interpreted as limiting, and for example, it can be adopted... Figure 1B The configuration shown involves near-infrared reflected light from the eyeball being incident on the image sensor 101a through a bandpass filter 101h that allows only infrared light to pass through.

[0034] Using this configuration, the gaze information calculation unit 103 can obtain near-infrared eye images.

[0035] Figure 2 The eyeball image 301 shown is obtained using Figure 1A or Figure 1B An example of an eye image obtained with the configuration shown. In eye image 301, the eyelid portion 305, the sclera portion 304 of the eyeball therein, the iris portion 303 therein, and the pupil 302, which is an opening near the center of the iris portion 303, are captured. This example is a typical example of an eye image obtained by a corneal reflection method.

[0036] On the eye image 301, light divided into two parts by the illumination light splitting element 101g forms Purkinje images 306 as two point-like optical images. Here, a pair of Purkinje images 306 are in relation to... Figure 2 The x-direction in the image is formed substantially parallel to the x-direction and is adjusted to sandwich the center of the eye image or the pupil region. This allows for the calculation of gaze information in the x-direction using the position of the Purkinje image 306 on the eye image and the contour shape and position of the pupil 302.

[0037] In this embodiment, unit miniaturization can be achieved by setting the illumination light splitting element 101g. Utilizing... Figure 3 The known configuration shown lacks an illumination light segmentation element, thus requiring a light source corresponding to each Purkinje image to form multiple Purkinje images, making it difficult to achieve unit miniaturization.

[0038] like Figure 4 As shown, by illuminating the eye with light to form a two-dimensional arrangement of Purkinje images centered on the pupil 302 in at least the horizontal and vertical directions, respectively, gaze information in at least the x and y directions can be detected. Therefore, two-dimensional gaze information can be detected.

[0039] exist Figure 4 In this configuration, the pair of Purkinje images 306 are used for gaze information in the x-direction, and the pair of Purkinje images 401 are used for gaze information in the y-direction. Note that the pair of Purkinje images 401 are also configured such that the pupil 302 at the center of the eye image is positioned between them and the other image.

[0040] It is known that the accuracy of gaze detection is greatly affected by individual differences in the eyes of the subjects (i.e., individuals). In particular, the amount of eyelid coverage of the eyeball varies among individuals, and therefore, the amount of coverage significantly impacts the accuracy of gaze detection. Figure 5A An example image of a person's eyeball with the eyelids ideally open. Figure 5B Examples of images showing the eyes of a person whose eyeballs are relatively largely covered by his or her eyelids.

[0041] It is relatively common for the human eyeball to be covered by the upper eyelid, but it is rare for the eyeball to be largely covered by the lower eyelid. Figure 5A The images captured clearly include the four Purkinje images as expected. On the other hand, in Figure 5B In the image, Purkinje image 501 (one of the Purkinje images), which is believed to be on the +y side, is hidden by the upper eyelid.

[0042] However, even in Figure 5B In this case, according to the arrangement of the four Purkinje images in this embodiment, there is a Purkinje image on the -y side, and only the Purkinje image on the +y side is hidden. Therefore, the captured image includes a pair of Purkinje images 306 and a Purkinje image 401, and the three Purkinje images in the Purkinje image are arranged in two dimensions, thereby enabling the acquisition of line-of-sight information in the x and y directions.

[0043] Figure 6 This illustrates the formation of paired Purkinje images in the x and y directions, but in each direction, the Purkinje images are not formed opposite each other with the pupil 302 at the center of the eye image 301 between them. Figure 6 The two-dimensional line-of-sight information cannot be obtained in the case shown because the +y side of the Purkinje image 601' of the four target Purkinje images 601 and 601' is hidden.

[0044] As described above, using the gaze detection device of this embodiment, two pairs of Purkinje images in the x and y directions are each formed so that the pupil 302 is between the pairs of Purkinje images and opposite to each other, so that gaze detection can be performed with high accuracy even when vignetting occurs on the eyeball image due to eyelids.

[0045] Using the line-of-sight detection device of this embodiment, such as Figure 7Aand Figure 7B As shown, a Purkinje image is formed in the form of a two-dimensional array on the eye image, which allows for higher precision acquisition of gaze information. Examples of two-dimensional array arrangements include, but are not limited to, those shown. Figure 7A The grid pattern shown, such as Figure 7B The Purkinje images shown are arranged in concentric forms on a corresponding straight line passing through the center, and so on.

[0046] When multiple corresponding single light sources are used to form multiple Purkinje image groups, the effects of stray and scattered light can be suppressed by changing the formation time between Purkinje image groups.

[0047] Second Embodiment

[0048] Line of sight detection equipment

[0049] The gaze detection device of the second embodiment is configured by using a two-dimensional micro-light source array instead of the point illumination unit in the gaze detection device 101 of the first embodiment. The micro-light source forming the two-dimensional micro-light source array can be a light-emitting diode or a vertical-cavity surface-emitting laser diode (VCSEL). Light sources arranged in two dimensions in a plane parallel to the substrate can be used as a two-dimensional micro-light source array. In addition, a stack of laser bars, which are stacks of edge-emitting laser elements arranged in one dimension on the substrate, can be used as a two-dimensional micro-light source array. Using such a two-dimensional micro-light source array as a point illumination unit, an array of point-like optical images can be efficiently formed on the eye image in a compact configuration.

[0050] Third Embodiment

[0051] electronic devices

[0052] Figure 8 This is a schematic diagram illustrating the configuration of the gaze detection device 701 according to the third embodiment. The third embodiment relates to a configuration obtained by providing a display device capable of displaying images for the gaze detection device of the first embodiment.

[0053] exist Figure 8 In this device, the line-of-sight detection device 701 includes a display 801 used as a display device and a controller 802 used to control the content displayed on the display. When video or the like is displayed on the display 801, an observer can observe the video on the display 801 through a dichroic mirror 101c.

[0054] In this embodiment, the gaze information calculation unit 103 calculates the observer's gaze direction from the eye image acquired by the image sensor 101a. Using the information related to this gaze direction sent to the display controller 802, the gaze position can be displayed on the display in real time in an overlay manner. Figure 8The configuration shown allows a television monitor, a monitor for a personal computer, or the like to be used as a display 801.

[0055] Fourth embodiment

[0056] electronic devices

[0057] Figure 9 This is a schematic diagram illustrating the configuration of the gaze detection device 901 according to the fourth embodiment. The configuration of the fourth embodiment is obtained by providing an eyepiece 902 to the gaze detection device 701 of the third embodiment, wherein the eyepiece 902 is used to optimize the video on the display device 801 and transmit the video to the eyeball.

[0058] When the gaze detection device is incorporated into a small unit such as a camera's electronic viewfinder, the display device used for the gaze detection device is small. Therefore, in this embodiment, the eyepiece 902 is arranged in front of the display device 801 so that the observer can comfortably view the video on the small display device 801.

[0059] In this embodiment, two individual light sources 101e and two illumination light splitting elements 101g are provided to utilize the emitted light from each individual light source, such as... Figure 4 The Purkinje images 306 and 401 are shown formed on the eyeball image.

[0060] Fifth Embodiment

[0061] camera equipment

[0062] Figure 10 This is a diagram illustrating the configuration of a camera device 1000 including an electronic viewfinder 1020 according to a fifth embodiment. A gaze detection device 901 is incorporated into the electronic viewfinder 1020.

[0063] exist Figure 10 The camera device 1000 includes: an image sensor 1001 configured to capture images; an MPU 1011 configured to process various signals, data, etc., for use with the camera device; a liquid crystal display 1013 responsible for input / output with external devices; an operation switch group 1014; and a memory 1015. An image sensor driving circuit 1008 configured to drive the image sensor 1001 and an image processing circuit 1009 configured to process image signals from the image sensor 1001 are also provided.

[0064] When a focus detection operation is instructed via a focus detection operation unit 1012, which includes a button for focus detection operation, focus detection is performed based on the captured image signal from the image sensor 1001. The image sensor 1001 is an element configured to acquire images, which is different from an image sensor configured to capture an eye image set in the gaze detection device 901 in the electronic viewfinder 1020.

[0065] In this embodiment, focus detection and focus adjustment are performed using the image plane phase difference method. Focus adjustment is achieved by performing focus detection, calculating the driving amount of the focusing lens in the imaging optical system 1003 required for focusing, and moving the focusing lens to the focusing position.

[0066] Image data acquired by the image sensor 1001 via the imaging optical system 1003 is processed by the image processing circuit 1009 into image data to be stored, and then stored in a memory 1015 (such as an SD card) detachably attached to the camera device 1000. Live view data of the state where the shutter button is not pressed is processed and displayed on the electronic viewfinder 1020 or the LCD screen 1013, and then displayed on the screen. The user viewing the electronic viewfinder 1020 can observe his or her gaze position pointer displayed in a manner superimposed on the video.

[0067] Using the user's gaze position pointer displayed in real time on the electronic viewfinder 1020's display, the area to be focused can be selected by the user's gaze. The camera device 1000 of this embodiment is provided with a gaze operation unit 1016, which can be used to perform a gaze area selection operation to determine the camera's focus detection area. The camera device 1000 then performs focus detection on the determined area.

[0068] As described in the first to fourth embodiments, the gaze detection device 901 of the camera device 1000 used in this embodiment can have a small size and perform gaze detection with high accuracy that is unlikely to be affected by individual differences between people. Therefore, the accuracy and speed of a series of operations such as autofocus and subject selection performed by the camera device 1000 of this embodiment can be improved.

[0069] Other embodiments

[0070] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0071] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. A line-of-sight detection device, comprising: Acquisition component, which is configured to acquire eye images; An illumination component is configured to illuminate the eyeball with light in the infrared band, forming a plurality of dot-shaped optical images on the eyeball image; as well as A computing unit is configured to calculate gaze information based on the eye image and the plurality of point-like optical images, wherein, The illumination component includes: two individual light sources configured to emit light in the infrared band; and two splitting elements configured to split the light emitted from each individual light source into multiple beams of emitted light to form the multiple point-like optical images. The irradiation component forms a first pair of dot-shaped optical images in the horizontal direction and a second pair of dot-shaped optical images in the vertical direction on the eye image, wherein the dot-shaped optical images in each pair are positioned opposite each other such that the center of the eye image is located between the two pairs. The first pair is formed by light emitted from one of the two single light sources and divided by one of the two dividing elements, and the second pair is formed by light emitted from the other of the two single light sources and divided by the other of the two dividing elements.

2. The line-of-sight detection device according to claim 1, wherein, The irradiation component forms a plurality of two-dimensionally arranged point-like optical images on the eyeball image.

3. The line-of-sight detection device according to claim 1, wherein, The irradiation component forms the plurality of dot-shaped optical images in such a way that even when a portion of the plurality of dot-shaped optical images is hidden by the eyelid, the remaining dot-shaped optical images are arranged in two dimensions.

4. The line-of-sight detection device according to claim 1, wherein, The illumination component forms the optical image on a straight line passing through the center of the eyeball image.

5. A camera device, comprising: A camera component configured to capture images of a subject; A display device configured to display the image captured by the camera component; as well as The line-of-sight detection device according to claim 1.

6. The camera device according to claim 5, wherein, The display device can also display the gaze information.

7. The camera device according to claim 5, wherein, Users can observe the images on the display device through the gaze detection device.

Images