Information processing device, information processing method, and program

Abstract

This information processing device comprises a processor. The processor: acquires first image data and second image data; outputs the first image data or the second image data to a display; acquires line-of-sight data during the period in which the second image data is being output to the display; and on the basis of the line-of-sight data, detects a line-of-sight position with respect to the display.

Description

Technical Field

[0001] The technology of the present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Japanese Patent Application Laid-Open No. 2019-159982 discloses a gaze detection device including an imaging unit that captures an image of a person's face, a projection unit that projects a predetermined pattern light onto the person's face, a control unit that controls the presence or absence of the projection of the pattern light by the imaging unit and the projection unit, and a gaze detection processing unit that detects a gaze from an image of the person's face captured by the imaging unit.

[0003] Japanese Patent Application Laid-Open No. 7-104170 discloses a gaze detection device including at least first and second light projecting means for projecting light onto the eyeball of a photographer, light receiving means for receiving each reflected light from the eyeball of the photographer of the light projected by the first and second light projecting means, light projecting control means for controlling the light emission pattern and light emission timing of the first and second light projecting means to include a different part and a common part in a time series, and feature extraction means for processing a gaze signal received by the light receiving means as a result of the first and second light projecting means projecting light based on the light emission pattern and light emission timing controlled by the light projecting control means, and extracting a feature signal from the gaze signal.

[0004] Japanese Patent Application Laid-Open No. 2019-161322 discloses a photographic apparatus having a photographing room and photographing a subject in the photographing room, including a photographing unit that photographs at least a face portion of the subject in the photographing room, a light source unit that irradiates light onto the subject in the photographing room, a gaze sensor that detects the gaze of the subject in the photographing room, and a control unit that controls the photographing unit, the light source unit, and the gaze sensor. The control unit includes a gaze direction determination unit that determines whether or not the gaze of the subject is directed in a predetermined direction based on a signal from the gaze sensor, and a still image acquisition unit that operates the photographing unit and the light source unit based on a signal from the gaze direction determination unit to obtain a still image.

Summary of the Invention

[0005] One embodiment of the technology of this disclosure provides an information processing device, an information processing method, and a program that can improve the accuracy of detecting the gaze position relative to a display compared to, for example, detecting the gaze position relative to a display based on gaze data acquired during the period in which first image data is output to the display. [Means for solving the problem]

[0006] A first aspect of the technology of this disclosure is an information processing device comprising a processor, wherein the processor acquires first image data and second image data, outputs either the first image data or the second image data to a display, acquires gaze data during the period in which the second image data is output to the display, and detects the gaze position relative to the display based on the gaze data.

[0007] A second aspect of the technology of this disclosure is an information processing device according to the first aspect, wherein the processor is an information processing device that selectively outputs first image data and second image data to a display.

[0008] A third aspect of the technology of this disclosure is an information processing device according to the first or second aspect, further comprising a first image sensor, wherein the first image data is image data obtained by imaging with the first image sensor.

[0009] A fourth aspect of the technology of this disclosure is an information processing device relating to any one of the first to third aspects, wherein the processor does not detect the gaze position during the period when the first image data is output to the display.

[0010] A fifth aspect of the technology of this disclosure is an information processing device relating to any one of the first to fourth aspects, wherein the second image data is first gaze detection image data for detecting the gaze position.

[0011] A sixth aspect of the technology of this disclosure is an information processing device according to the fifth aspect, wherein the gaze detection image shown by the first gaze detection image data includes a grayscale region.

[0012] A seventh aspect of the technology of this disclosure is an information processing apparatus according to the sixth aspect, further comprising a second image sensor, wherein the gaze detection image includes an image region obtained by imaging with the second image sensor and a grayscale region, and the processor is an information processing apparatus that changes the positions of the image region and the grayscale region for each frame of the gaze detection image.

[0013] An eighth aspect of the technology of this disclosure is an information processing device according to the seventh aspect, wherein the processor sets the position of an image region based on the gaze position for each frame of the gaze detection image.

[0014] A ninth aspect of the technology of this disclosure is an information processing device relating to any one of the sixth to eighth aspects, wherein the achromatic area is a black area or a gray area.

[0015] A tenth aspect of the technology of this disclosure is an information processing device according to any one of the first to ninth aspects, further comprising a light source that illuminates an eye that is the target of gaze detection, wherein the processor outputs second image data to a display while illuminating with light from the light source when there is no object in the position through which the light passes, and outputs second gaze detection image data to the display while illuminating with light from the light source when an object is present.

[0016] An eleventh aspect of the technology of this disclosure is an information processing device according to the tenth aspect, wherein the second gaze detection image data is image data that includes a pattern for detecting the gaze position relative to a display.

[0017] A twelfth aspect of the technology of this disclosure is an information processing device according to the tenth or eleventh aspect, wherein the processor detects a gaze position based on the degree of difference between first gaze data, which is gaze data acquired when an object is present and the eyes are closed, and second gaze data, which is gaze data acquired when an object is present and the eyes are open.

[0018] A thirteenth aspect of the technology of this disclosure is an information processing device relating to any one of the first to twelfth aspects, further comprising a third image sensor, wherein the processor detects a first line of sight direction of a subject appearing in the first image shown by the first image data, and acquires a third image data obtained by imaging with the third image sensor based on a second line of sight direction corresponding to the line of sight position and the first line of sight direction.

[0019] A fourteenth aspect of the technology of this disclosure is an information processing device according to the thirteenth aspect, wherein the processor is an information processing device that acquires a third image data when the first line of sight direction and the second line of sight direction coincide.

[0020] A fifteenth aspect of the technology of this disclosure is an information processing device relating to any one of the first to fourteenth aspects, wherein the number of first frames, which is the number of frames of first image data output to a display per unit time, is greater than the number of second frames, which is the number of frames of second image data output to a display per unit time.

[0021] A sixteenth aspect of the technology of this disclosure is an information processing device according to the fifteenth aspect, wherein the processor sets the ratio of the number of first frames and the number of second frames per unit time according to the reliability of the gaze position detection result.

[0022] A 17th aspect of the technology of this disclosure is an information processing device relating to any one of the first to 16th aspects, wherein the processor sets the brightness of a third image represented by a second image according to the brightness of a second image represented by a first image.

[0023] The 18th aspect according to the technology of the present disclosure is an information processing apparatus according to any one of the 1st to 17th aspects, further including an acceleration sensor, and the processor generates orientation information regarding the orientation of the information processing apparatus based on the acceleration data of the acceleration sensor and the line-of-sight position.

[0024] The 19th aspect according to the technology of the present disclosure is an information processing apparatus according to the 18th aspect, further including a fourth image sensor, and the processor adds orientation information to the fourth image data obtained by imaging with the fourth image sensor.

[0025] The 20th aspect according to the technology of the present disclosure is an information processing apparatus according to any one of the 1st to 19th aspects, further including a touch panel, and the processor executes a first determination process for determining which of the left eye and the right eye is the line-of-sight detection target based on the line-of-sight data, and a second determination process for determining the orientation of the left eye or the right eye determined to be the line-of-sight detection target in the first determination process based on the line-of-sight position. Based on the first determination result by the first determination process and the second determination result by the second determination process, the information processing apparatus sets the valid area and / or the invalid area of the touch panel.

[0026] The 21st aspect according to the technology of the present disclosure is an information processing apparatus according to any one of the 1st to 20th aspects, and the processor detects the size of the pupil based on the line-of-sight data, and adjusts the brightness of the display based on the result of detecting the size of the pupil.

[0027] The 22nd aspect according to the technology of the present disclosure is an information processing apparatus that is an imaging apparatus according to any one of the 1st to 21st aspects.

[0028] A 23rd aspect of the technology according to the present disclosure is an information processing apparatus including a processor, the processor determining whether a display is in either a first state or a second state, acquiring gaze data during a period when the display is in the second state, and detecting a gaze position with respect to the display based on the gaze data.

[0029] A 24th aspect of the technology according to the present disclosure is an information processing method including acquiring first image data and second image data, outputting either the first image data or the second image data to a display, acquiring gaze data during a period when the second image data is being output to the display, and detecting a gaze position with respect to the display based on the gaze data.

[0030] A 25th aspect of the technology according to the present disclosure is a program for causing a computer to execute a process including acquiring first image data and second image data, outputting either the first image data or the second image data to a display, acquiring gaze data during a period when the second image data is being output to the display, and detecting a gaze position with respect to the display based on the gaze data.

Brief Description of the Drawings

[0031] [Figure 1] It is a two - view drawing showing an example of the side and the back of an imaging device. [Figure 2] It is a block diagram showing an example of the hardware configuration of an imaging device. [Figure 3] It is an explanatory drawing showing an example of a method for detecting a gaze position. [Figure 4] It is a block diagram showing an example of the functional configuration of a CPU according to the first embodiment. [Figure 5] It is an explanatory drawing showing an example of a first image and a second image according to the first embodiment. [Figure 6] It is an explanatory drawing showing an example of a first operation of a CPU according to the first embodiment. [Figure 7]This is an explanatory diagram showing an example of the second operation of the CPU according to the first embodiment. [Figure 8] This is an explanatory diagram showing an example of a third operation of the CPU according to the first embodiment. [Figure 9] This is an explanatory diagram showing an example of a fourth operation of the CPU according to the first embodiment. [Figure 10] This flowchart shows an example of the eye-tracking detection process according to the first embodiment. [Figure 11] This flowchart shows an example of the flow of the frame count setting process according to the first embodiment. [Figure 12] This is an explanatory diagram showing an example of the first and second images according to the second embodiment. [Figure 13] This is an explanatory diagram showing an example of the operation of the CPU according to the second embodiment. [Figure 14] This is an explanatory diagram showing an example of the first and second images according to the third embodiment. [Figure 15] This is an explanatory diagram showing an example of the operation of the CPU according to the third embodiment. [Figure 16] This is an explanatory diagram showing an example of the first and second images according to the fourth embodiment. [Figure 17] This is an explanatory diagram showing an example of the operation of the CPU according to the fourth embodiment. [Figure 18] This is an explanatory diagram showing an example of the first operation of the CPU according to the fifth embodiment. [Figure 19] This is an explanatory diagram showing an example of the second operation of the CPU according to the fifth embodiment. [Figure 20] This is an explanatory diagram showing an example of the third operation of the CPU according to the fifth embodiment. [Figure 21] This is a flowchart showing an example of the eye-tracking detection process according to the fifth embodiment. [Figure 22] This is an explanatory diagram showing an example of the first operation of the CPU according to the sixth embodiment. [Figure 23] This is an explanatory diagram showing an example of the second operation of the CPU according to the sixth embodiment. [Figure 24] This is an explanatory diagram showing an example of the third operation of the CPU according to the sixth embodiment. [Figure 25] This flowchart shows an example of the eye-tracking detection process according to the sixth embodiment. [Figure 26] This is an explanatory diagram showing an example of the first image according to the seventh embodiment. [Figure 27] This is an explanatory diagram showing an example of the first operation of the CPU according to the seventh embodiment. [Figure 28] This is an explanatory diagram showing an example of the second operation of the CPU according to the seventh embodiment. [Figure 29] This is a flowchart showing an example of the gaze detection process according to the seventh embodiment. [Figure 30] This is an explanatory diagram showing an example of the first operation of the CPU according to the eighth embodiment. [Figure 31] This is an explanatory diagram showing an example of the second operation of the CPU according to the eighth embodiment. [Figure 32] This is a flowchart showing an example of the eye-tracking detection process according to the eighth embodiment. [Figure 33] This is an explanatory diagram showing an example of the first operation of the CPU according to the ninth embodiment. [Figure 34] This is an explanatory diagram showing an example of the second operation of the CPU according to the ninth embodiment. [Figure 35] This flowchart shows an example of the eye-tracking detection process according to the ninth embodiment. [Figure 36] This is an explanatory diagram showing an example of the first operation of the CPU according to the 10th embodiment. [Figure 37] This is an explanatory diagram showing an example of the second operation of the CPU according to the 10th embodiment. [Figure 38] This flowchart shows an example of the eye-tracking detection process according to the 10th embodiment. [Modes for carrying out the invention]

[0032] Hereinafter, an example of an embodiment of the image processing apparatus, image processing method, and program relating to the technology of this disclosure will be described with reference to the attached drawings.

[0033] First, let's explain the terminology used in the following explanation.

[0034] I / F stands for "Interface". CPU stands for "Central Processing Unit". NVM stands for "Non-Volatile Memory". RAM stands for "Random Access Memory". EEPROM stands for "Electrically Erasable and Programmable Read Only Memory". HDD stands for "Hard Disk Drive". CMOS stands for "Complementary Metal Oxide Semiconductor". CCD stands for "Charge Coupled Device". SSD stands for "Solid State Drive". EL stands for "Electro Luminescence". LED stands for "light emitting diode". OLED stands for "Organic Light-Emitting Diode". fps stands for "frames per second". GPU stands for "Graphics Processing Unit". TPU stands for "Tensor Processing Unit". USB stands for "Universal Serial Bus". ASIC stands for "Application Specific Integrated Circuit". FPGA stands for "Field-Programmable Gate Array". PLD stands for "Programmable Logic Device". SoC stands for "System-on-a-chip". IC stands for "Integrated Circuit". AI stands for "Artificial Intelligence".

[0035] In this specification, “parallel” means not only perfect parallelism but also an error that is generally acceptable in the art to which the disclosed technology belongs, provided that the error is not contrary to the spirit of the disclosed technology. In this specification, “identical” means not only perfect identicalness but also an error that is generally acceptable in the art to which the disclosed technology belongs, provided that the error is not contrary to the spirit of the disclosed technology.

[0036] [First Embodiment] First, the first embodiment will be described.

[0037] As an example, as shown in Figure 1, the imaging device 10 is a digital camera and comprises a lens unit 12 and an imaging device body 14. The imaging device 10 is an example of an "information processing device" and an "imaging device" related to the technology of this disclosure. The imaging device 10 has defined pitch axis, yaw axis, and roll axis. In the example shown in Figure 1, axis P represents the pitch axis of the imaging device 10, axis Y represents the yaw axis of the imaging device 10, and axis R represents the roll axis of the imaging device 10. In the following description, the lateral direction of the imaging device 10 corresponds to the pitch axis direction, and the vertical direction of the imaging device 10 corresponds to the yaw axis direction.

[0038] The lens unit 12 is attached to the imaging device body 14. A touch panel display 34 is provided on the back of the imaging device body 14. The touch panel display 34 is formed by a touch panel 34A and a display 34B, for example, with the touch panel 34A superimposed on the display 34B. The touch panel 34A is an example of a "touch panel" according to the technology of this disclosure.

[0039] An electronic viewfinder 28 is provided at the top of the imaging device body 14. The electronic viewfinder 28 has a display 72. A viewfinder aperture 16 is provided at the top of the imaging device body 14, and the display 72 is provided inside the viewfinder aperture 16. The display 72 is positioned so that the user can see the display 72 with their eye 302 when looking through the viewfinder aperture 16. The display 72 is an example of a "display" related to the technology of this disclosure.

[0040] Inside the viewfinder aperture 16, a light source 82 and a gaze sensor 86 for gaze detection are provided. The user's eye 302 looking through the viewfinder aperture 16 is the target of gaze detection. The light source 82 has multiple light emitters 82A. The multiple light emitters 82A are arranged, for example, in a line laterally across the imaging device 10.

[0041] The light source 82 is positioned so as to illuminate the user's eye 302 when the user looks through the viewfinder aperture 16. For example, the light source 82 emits near-infrared light. The gaze sensor 86 is positioned so as to image the user's eye 302 when the user looks through the viewfinder aperture 16. The light source 82 is an example of a "light source" in the technology of this disclosure. The gaze sensor 86 is an example of a "gaze sensor" in the technology of this disclosure.

[0042] A half-mirror (not shown) may be placed inside the viewfinder aperture 16. A display 72 may be positioned on the optical axis through which the half-mirror passes, and a gaze sensor 86 may be positioned on the optical axis reflected by the half-mirror.

[0043] As an example, as shown in Figure 2, the imaging device body 14 includes a computer 20, an acceleration sensor 22, an image sensor 24, an image sensor control circuit 26, an electronic viewfinder 28, an eye-tracking unit 30, an image memory 32, a touch panel display 34, a touch panel display control circuit 36, and an input / output interface 38. The acceleration sensor 22, image sensor 24, image sensor control circuit 26, image memory 32, and touch panel display control circuit 36 ​​are connected to the input / output interface 38.

[0044] Computer 20 comprises a processor 42, an NVM 44, and RAM 46. The processor 42 controls the entire imaging device 10. The processor 42 is, for example, a processing unit including a CPU and a GPU, where the GPU operates under the control of the CPU and is responsible for performing image-related processing. Here, a processing unit including a CPU and a GPU is given as an example of the processor 42, but this is merely an example, and the processor 42 may be one or more CPUs with integrated GPU functionality, or one or more CPUs without integrated GPU functionality. The processor 42, NVM 44, and RAM 46 are connected via a bus 48, which is connected to an input / output I / F 38. Computer 20 is an example of a “computer” relating to the technology of this disclosure. The processor 42 is an example of a “processor” relating to the technology of this disclosure.

[0045] NVM44 is a non-temporary storage medium that stores various parameters and programs. For example, NVM44 is flash memory (e.g., EEPROM). However, this is merely one example, and HDDs, etc., may also be used as NVM44 along with flash memory. RAM46 temporarily stores various information and is used as work memory.

[0046] The processor 42 reads the necessary program from the NVM 44 and executes the read program in the RAM 46. The processor 42 controls the acceleration sensor 22, image sensor 24, image sensor control circuit 26, electronic viewfinder 28, gaze detection unit 30, and image memory 32 according to the program executed in the RAM 46.

[0047] The acceleration sensor 22 detects acceleration in the pitch axis, yaw axis, and roll axis directions of the imaging device 10. The acceleration sensor 22 outputs acceleration data corresponding to the acceleration in each axis direction of the imaging device 10. The acceleration sensor 22 is an example of an "acceleration sensor" related to the technology of this disclosure.

[0048] The image sensor 24 is, for example, a CMOS image sensor. Here, a CMOS image sensor is used as an example of the image sensor 24, but the technology of this disclosure is not limited to this, and the technology of this disclosure is valid even if the image sensor 24 is another type of image sensor, such as a CCD image sensor. An image sensor control circuit 26 is connected to the image sensor 24. The image sensor control circuit 26 controls the image sensor 24 according to the imaging control signal from the processor 42. The image sensor 24 is an example of the "first image sensor," "second image sensor," "third image sensor," and "fourth image sensor" related to the technology of this disclosure.

[0049] Subject light enters the imaging lens 52 of the lens unit 12, which will be described later. The subject light is imaged onto the light-receiving surface of the image sensor 24 by the imaging lens 52. A photoelectric conversion element (not shown) is provided on the light-receiving surface of the image sensor 24. Under the control of the control circuit 26 for the image sensor, the photoelectric conversion element converts the subject light received by the light-receiving surface into electrical signals and outputs an electrical signal corresponding to the amount of subject light as analog image data representing the subject light. The image sensor 24 has a signal processing circuit (not shown). The signal processing circuit generates a first digital image data by digitizing the analog image data and outputs the first image data.

[0050] The image memory 32 temporarily stores the first image data generated by the image sensor 24. The processor 42 retrieves the first image data from the image memory 32 and performs various processing using the retrieved first image data.

[0051] The lens unit 12 includes an imaging lens 52. The imaging lens 52, for example, has an objective lens 54, a focus lens 56, a zoom lens 58, and an aperture 60. The lens unit 12 also includes a lens control circuit 62, a first actuator 64, a second actuator 66, and a third actuator 68. The lens control circuit 62 is connected to the first actuator 64, the second actuator 66, and the third actuator 68, and the lens control circuit 62 is connected to the input / output interface 38. The lens control circuit 62 controls the first actuator 64, the second actuator 66, and the third actuator 68 according to lens control signals from the processor 42.

[0052] The first actuator 64 moves the focus lens 56 along the optical axis OA. The position of the focus is adjusted by changing the position of the focus lens 56. The second actuator 66 moves the zoom lens 58 along the optical axis OA. The focal length is adjusted by changing the position of the zoom lens 58. The third actuator 68 changes the size of the aperture 60. The amount of aperture 60 changes as the size of the aperture 60 changes, thereby adjusting the exposure. The first actuator 64, the second actuator 66, and the third actuator 68 are, for example, piezoelectric elements or voice coil motors.

[0053] The electronic viewfinder 28 includes a display 72, a display control circuit 74, and an eyepiece 76. The display 72 is, for example, a liquid crystal display or an EL display. The display control circuit 74 is connected to the input / output interface 38. The processor 42 selectively outputs first image data and second image data to the display control circuit 74, as will be described later. The display control circuit 74 displays the first image 200 on the display 72 according to the first image data and displays the second image 202 on the display 72 according to the second image data. The eyepiece 76 is positioned opposite the screen of the display 72.

[0054] The gaze detection unit 30 includes a light source 82, a control circuit 84 for the light source, a gaze sensor 86, and a control circuit 88 for the gaze sensor. The light source 82 is, for example, an LED that outputs near-infrared light. Near-infrared light is an example of "light" in the art of this disclosure. The control circuit 84 for the light source is connected to the input / output I / F 38. The control circuit 84 for the light source controls the light source 82 according to a light source control signal from the processor 42.

[0055] The gaze sensor 86 is, for example, a CMOS image sensor sensitive to near-infrared light. Here, a CMOS image sensor is given as an example of the gaze sensor 86, but the technology of this disclosure is not limited to this, and the technology of this disclosure is valid even if the gaze sensor 86 is another type of image sensor, such as a CCD image sensor. A gaze sensor control circuit 88 is connected to the gaze sensor 86. The gaze sensor control circuit 88 controls the gaze sensor 86 according to the imaging control signal from the processor 42. The gaze sensor 86 images a subject (for example, the user's eyes 302) and outputs gaze data obtained by imaging.

[0056] The touch panel display 34 is connected to the touch panel display control circuit 36. The touch panel display control circuit 36 ​​outputs received data to the processor 42 in response to user instructions received by the touch panel 34A. The touch panel display control circuit 36 ​​also sets the active and / or inactive areas (see Figure 34, described later) of the touch panel 34A, for example, in accordance with the area control signal output from the processor 42. Furthermore, the touch panel display control circuit 36 ​​causes the display 34B to display an image based on the display control signal output from the processor 42. The display control signal includes, for example, data for displaying an image on the display 34B.

[0057] Figure 3 shows an example of a method for detecting gaze position based on gaze data. In the example shown in Figure 3, near-infrared light is irradiated onto the eyeball 306 from multiple light emitters 82A, and a dot-like pattern 204 is formed on the surface of the cornea 308 by the reflection of the near-infrared light from the surface of the cornea 308. The distance L1 between the patterns 204 is obtained from the image 208 obtained by imaging the eyeball 306 with the gaze sensor 86.

[0058] The distance L3 between the gaze sensor 86 and the eyeball 306 can be determined from the known distance L2 between multiple light emitters 82A, the known mounting angle α of the multiple light emitters 82A, the known mounting angle β of the gaze sensor 86, and the distance L1 between the patterns 204. In addition, the position of the center 310A and the outline 310B of the pupil 310 relative to the center 208A of the image 208 obtained by the gaze sensor 86 imaging the eyeball 306 can be determined. Based on the positions of the center 310A and the outline 310B of the pupil 310 relative to the center 208A of the image 208, the distance L3 between the gaze sensor 86 and the eyeball 306, and a predetermined radius of curvature for the eyeball 306, the coordinates of the center 306A of the eyeball 306 and the coordinates of the center 310A of the pupil 310 can be determined. For the radius of curvature of the eyeball 306, for example, a statistically determined value is applied.

[0059] The direction in which the line 312 connecting the coordinates of the center 306A of the eyeball 306 and the coordinates of the center 310A of the pupil 310 extends corresponds to the direction of the line of sight of the eye 302. Furthermore, the extension line 314 obtained by extending the line 312 connecting the coordinates of the center 306A of the eyeball 306 and the coordinates of the center 310A of the pupil 310 indicates the line of sight of the eye 302, and the point P where the line of sight of the eye 302 intersects with the screen of the display 72 corresponds to the position of the line of sight of the eye 302. In this manner, the position of the line of sight of the eye 302 is detected based on the line of sight data obtained by imaging the eyeball 306 with the line of sight sensor 86. Hereafter, "position of the line of sight of the eye 302" will also be referred to as "line of sight position" or "line of sight position relative to the display 72". The line of sight position is an example of a "line of sight position" related to the technology of this disclosure.

[0060] It should be noted that the method for detecting the gaze position described above is merely one example, and the technology of this disclosure is not limited thereto. In addition to the method for detecting the gaze position described above, various methods for detecting the gaze position relative to the display 72 based on gaze data obtained by imaging the eyeball 306 with the gaze sensor 86 while a pattern 204 is formed on the surface of the cornea 308 are applicable to the technology of this disclosure. Furthermore, in the method for detecting the gaze position described above, the gaze position is detected based on reflected light reflected from the surface of the cornea 308, but the gaze position may also be detected based on reflected light reflected from a location other than the cornea 308 (for example, the retina).

[0061] As an example, as shown in Figure 4, the gaze detection program 100 is stored in the NVM 44 of the imaging device 10. The gaze detection program 100 is an example of a "program" related to the technology of this disclosure. The processor 42 reads the gaze detection program 100 from the NVM 44 and executes the read gaze detection program 100 in the RAM 46. The processor 42 performs gaze detection processing according to the gaze detection program 100 executed in the RAM 46. The gaze detection processing is performed by the processor 42 operating as a first image display control unit 102, a second image display control unit 104, a gaze detection processing unit 106, and a frame count setting processing unit 108 according to the gaze detection program 100.

[0062] Figure 5 shows a time chart Tc of the image displayed on the display 72 in the first embodiment. The first frame number N1 is the number of frames of the first image 200, and the second frame number N2 is the number of frames of the second image 202. As shown in the time chart Tc, the period during which the first image 200 with the first frame number N1 is displayed on the display 72 and the period during which the second image 202 with the second frame number N2 is displayed on the display 72 alternate. The first image 200 and the second image 202 constitute a live view image. The first image 200 and the second image 202 will be described in detail later. The first frame number N1 is an example of the "first frame number" in the technology of this disclosure. The second frame number N2 is an example of the "second frame number" in the technology of this disclosure.

[0063] As an example, as shown in Figure 6, the first image display control unit 102 displays the first image 200 on the display 72. Specifically, the first image display control unit 102 outputs an imaging control signal to the image sensor control circuit 26, causing the image sensor 24 to image a subject (not shown) via the image sensor control circuit 26. The first image display control unit 102 acquires the first image data obtained when the subject is imaged by the image sensor 24, and outputs the acquired first image data to the display 72. The display 72 displays the first image 200 indicated by the first image data. As a result, the first image 200, in which the subject appears as an image, is displayed on the display 72.

[0064] The first image display control unit 102 causes the display 72 to display a number of first images 200 corresponding to the first frame number N1. When the first images 200 are displayed on the display 72, the first images 200 are projected onto the surface of the cornea 308 by the reflection of light irradiated from the display 72. The first image data is an example of the "first image data" relating to the technology of this disclosure. The state in which the first image 200 is displayed on the display 72 is an example of the "first state" relating to the technology of this disclosure.

[0065] As an example, as shown in Figure 7, the second image display control unit 104 causes the second image 202 to be displayed on the display 72. Specifically, the NVM 44 has second image data representing the second image 202 stored in it beforehand. The second image 202 is, for example, a black image in which the entire area is black.

[0066] The second image display control unit 104 acquires the second image data stored in the NVM 44 and outputs the acquired second image data to the display 72. The display 72 displays the second image 202 indicated by the second image data. As a result, the second image 202, which is a black image, is displayed on the display 72. The second image data is an example of the "second image data" and "first gaze detection image data" related to the technology of this disclosure. The second image 202 is an example of the "gaze detection image" related to the technology of this disclosure. The state in which the second image 202 is displayed on the display 72 is an example of the "second state" related to the technology of this disclosure.

[0067] The second image display control unit 104 displays a number of second images 202 corresponding to the second frame number N2 on the display 72. When the second image 202, which is a black image, is displayed on the display 72, the second image 202, which is a black image, is projected onto the surface of the cornea 308. When the second image 202, which is a black image, is projected onto the surface of the cornea 308, the amount of light reflected by the cornea 308 is suppressed compared to when the first image 200 is projected onto the surface of the cornea 308 (see Figure 6). The second image 202 is a gaze detection image for detecting the gaze position, as will be described later. The black region forming the second image 202 is an example of a "chromatic region" related to the technology of this disclosure.

[0068] As an example, as shown in Figure 8, the gaze detection processing unit 106 detects the gaze position of the user looking at the display 72 during the period when the second image 202 is displayed on the display 72 (i.e., the period when the second image data is output to the display 72). Specifically, the gaze detection processing unit 106 outputs an imaging control signal to the gaze sensor control circuit 88, causing the gaze sensor 86 to image the subject (for example, the user's eyes 302) via the gaze sensor control circuit 88. The gaze detection processing unit 106 then acquires gaze data obtained from the image captured by the gaze sensor 86 and detects the gaze position relative to the display 72 based on the acquired gaze data. The method for detecting the gaze position is as described in Figure 3. The gaze data is an example of "gaze data" related to the technology of this disclosure.

[0069] The gaze detection processing unit 106 does not detect the gaze position during the period when the first image 200 is displayed on the display 72 (i.e., the period when the first image data is output to the display 72).

[0070] Furthermore, the light source 82 may continue to emit near-infrared light when the imaging device 10 is powered on. In addition, the light source 82 may emit near-infrared light when the gaze position is detected by the gaze detection processing unit 106, and stop emitting near-infrared light when the gaze position is not detected by the gaze detection processing unit 106.

[0071] As an example, as shown in Figure 9, the frame count setting processing unit 108 executes the frame count setting process. The frame count setting process is the process of setting the ratio of the first frame count N1 and the second frame count N2 per unit time.

[0072] In this case, the unit time is, for example, one second. The sum of the first number of frames N1 and the second number of frames N2 per unit time is the number of frames per unit time. The number of frames per unit time is set to, for example, 60 frames or 30 frames. That is, if the number of frames per unit time is 60 frames, the frame rate is 60fps, and if the number of frames per unit time is 30 frames, the frame rate is 30fps.

[0073] The frame count setting processing unit 108 includes a reliability determination unit 112, a frame count determination unit 114, and a frame count setting unit 116 in order to perform frame count setting processing.

[0074] The confidence level determination unit 112 determines the confidence level of the gaze position detection result. Specifically, the confidence level determination unit 112 determines whether the confidence level of the gaze position detection result is below a predetermined threshold. The confidence level of the gaze position detection result may be determined using, for example, an artificial intelligence processing technique such as a neural network. Alternatively, the confidence level of the gaze position detection result may be determined based on the spatial variability when the gaze positions are arranged in a time series. The threshold is set to the minimum value that ensures the accuracy of gaze position detection.

[0075] If the confidence level determination unit 112 determines that the confidence level regarding the detection result of the gaze position is below a threshold, the frame count determination unit 114 determines whether the second frame count N2 is less than a predetermined maximum value.

[0076] In this case, when a second image 202, which is a black image, is inserted between the first image 200, which is an image capture image, the time-averaged brightness of the live view image is compensated by increasing the brightness of the first image 200 compared to the case where the second image 202 is not inserted, in order to maintain the brightness of the live view image composed of the first image 200 and the second image 202. Therefore, the maximum value is set to, for example, the minimum value at which the time-averaged brightness of the live view image is compensated.

[0077] However, if the brightness of the first image 200 is increased using, for example, a pulse width modulation method, the power efficiency of the LED type (e.g., OLED type) display 72 will decrease. Therefore, the maximum value is set to, for example, the minimum value that ensures the power efficiency of the display 72.

[0078] If the frame count determination unit 114 determines that the second frame count N2 is less than the maximum value, the frame count setting unit 116 increases the second frame count N2 by 1. If the frame count determination unit 114 determines that the second frame count N2 is at its maximum value, the frame count setting unit 116 does not increase the second frame count N2, and the second frame count N2 is maintained at its maximum value.

[0079] On the other hand, if the reliability determination unit 112 determines that the reliability of the gaze position detection result exceeds a threshold, the frame count determination unit 114 determines whether the second frame count N2 exceeds a predetermined minimum value.

[0080] Here, since the display 72 cannot be driven beyond its rated capacity, its brightness cannot be increased beyond its rated capacity. Therefore, the number of frames of the second image 202, which is a black image inserted between the first images 200, is set within a range that ensures the frequency of updating the gaze position detection. Accordingly, the minimum value is set to the minimum value that ensures the frequency of updating the gaze position detection.

[0081] If the frame count determination unit 114 determines that the second frame count N2 exceeds the minimum value, the frame count setting unit 116 decreases the second frame count N2 by 1. If the frame count determination unit 114 determines that the second frame count N2 is at the minimum value, the frame count setting unit 116 does not decrease the second frame count N2, and the second frame count N2 is kept at the minimum value. Then, the ratio of the first frame count and the second frame count is set according to the second frame count N2 set by the frame count setting unit 116.

[0082] In this way, the frame count setting processing unit 108 sets the ratio of the first frame count N1 and the second frame count N2 per unit time according to the reliability of the gaze position detection result. As an example, the frame count setting processing unit 108 sets the ratio of the first frame count N1 and the second frame count N2 per unit time such that the first frame count N1 is greater than the second frame count N2.

[0083] Next, the operation of the imaging device 10 according to the first embodiment will be explained with reference to Figure 10. Figure 10 shows an example of the flow of the gaze detection process according to the first embodiment.

[0084] In the gaze detection process shown in Figure 10, first, in step ST10, the first image display control unit 102 causes the image sensor 24 to capture an image of the subject. Then, the first image display control unit 102 acquires the first image data obtained from the image of the subject captured by the image sensor 24. After the process in step ST10 is executed, the gaze detection process proceeds to step ST11.

[0085] In step ST11, the first image display control unit 102 outputs the first image data acquired in step ST10 to the display 72. As a result, the first image 200 is displayed on the display 72. After the processing in step ST11 is completed, the gaze detection process proceeds to step ST12.

[0086] In step ST12, the first image display control unit 102 determines, based on the first image data acquired in step ST10, whether the number of frames n1 of the first image 200 is less than the first number of frames N1. If, in step ST12, the number of frames n1 of the first image 200 is less than the first number of frames N1, the determination is affirmed, and the gaze detection process proceeds to step ST10. If, in step ST12, the number of frames n1 of the first image 200 reaches the first number of frames N1, the determination is denied, and the gaze detection process proceeds to step ST13.

[0087] In step ST13, the second image display control unit 104 acquires the second image data stored in the NVM44. After the processing in step ST13 is completed, the gaze detection process proceeds to step ST14.

[0088] In step ST14, the second image display control unit 104 outputs the second image data acquired in step ST13 to the display 72. As a result, the second image 202, which is a black image, is displayed on the display 72. After the processing in step ST14 is completed, the gaze detection process proceeds to step ST15.

[0089] In step ST15, the gaze detection processing unit 106 acquires gaze data obtained by the gaze sensor 86. After the processing in step ST15 is completed, the gaze detection process proceeds to step ST16.

[0090] In step ST16, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the gaze data acquired in step ST15. After the processing in step ST16 is completed, the gaze detection process proceeds to step ST17.

[0091] In step ST17, the second image display control unit 104 determines, based on the second image data acquired in step ST13, whether the number of frames n2 of the second image 202 is less than the number of second frames N2. If, in step ST17, the number of frames n2 of the second image 202 is less than the number of second frames N2, the determination is affirmed, and the gaze detection process proceeds to step ST13. If, in step ST17, the number of frames n2 of the second image 202 reaches the number of second frames N2, the determination is denied, and the gaze detection process proceeds to step ST18.

[0092] In step ST18, the frame count setting processing unit 108 executes a frame count setting process, which is the process of setting the ratio of the first frame count N1 and the second frame count N2 per unit time. After the process in step ST18 is executed, the gaze detection process moves on to step ST19.

[0093] In step ST19, the processor 42 determines whether the conditions for terminating the gaze detection process have been met. An example of a condition for terminating the gaze detection process is that the conditions for switching from the mode performing the gaze detection process to another mode have been met. If the conditions for terminating the gaze detection process have not been met in step ST19, the determination is denied, and the gaze detection process proceeds to step ST10. If the conditions for terminating the gaze detection process have been met in step ST19, the determination is affirmed, and the gaze detection process terminates.

[0094] Next, we will describe the frame count setting process in detail. Figure 11 shows an example of the flow of the frame count setting process.

[0095] In the frame count setting process shown in Figure 11, first, in step ST20, the reliability determination unit 112 determines whether the reliability of the gaze position detection result is below a threshold. If the reliability of the gaze position detection result is below the threshold in step ST20, the determination is affirmed, and the frame count setting process proceeds to step ST21. If the reliability of the gaze position detection result exceeds the threshold in step ST20, the determination is denied, and the frame count setting process proceeds to step ST23.

[0096] In step ST21, the frame count determination unit 114 determines whether the second frame count N2 is less than the maximum value. If the second frame count N2 is less than the maximum value in step ST21, the determination is affirmed, and the frame count setting process proceeds to step ST22. If the second frame count N2 is the maximum value in step ST21, the determination is denied, and the frame count setting process ends. In this case, the second frame count N2 is kept at the maximum value.

[0097] In step ST22, the frame count setting unit 116 increments the second frame count N2 by 1. After the processing in step ST22 is completed, the frame count setting process ends.

[0098] In step ST23, the frame count determination unit 114 determines whether the second frame count N2 exceeds the minimum value. If the second frame count N2 exceeds the minimum value in step ST23, the determination is affirmed, and the frame count setting process proceeds to step ST24. If the second frame count N2 is the minimum value in step ST23, the determination is denied, and the frame count setting process ends. In this case, the second frame count N2 is kept at the minimum value.

[0099] In step ST24, the frame count setting unit 116 decreases the second frame count N2 by 1. After the processing in step ST24 is completed, the frame count setting process ends.

[0100] The information processing method described above as the operation of the imaging device 10 is an example of an "information processing method" relating to the technology of this disclosure.

[0101] As described above, in the first embodiment, the processor 42 acquires first image data and second image data, and outputs either the first image data or the second image data to the display 72. The processor 42 then acquires gaze data during the period when the second image data is output to the display 72, and detects the gaze position relative to the display 72 based on the gaze data. Therefore, compared to, for example, the case where the gaze position relative to the display 72 is detected based on gaze data acquired during the period when the first image data is displayed on the display 72, the accuracy of gaze position detection can be improved.

[0102] Furthermore, the processor 42 selectively outputs the first image data and the second image data to the display 72. Therefore, an image including the first image 200 and the second image 202 (for example, a live view image) can be displayed on the display 72.

[0103] Furthermore, the imaging device 10 is equipped with an image sensor 24, and the first image data is image data obtained by capturing images with the image sensor 24. Therefore, the subject can be represented as an image in the first image 200.

[0104] Furthermore, the processor 42 does not detect the gaze position while outputting the first image data to the display 72. Therefore, it is possible to avoid detecting the gaze position while the first image 200 is displayed on the display 72.

[0105] Furthermore, the second image data is gaze detection image data for detecting the gaze position. Therefore, the gaze position relative to the display 72 can be detected based on the gaze data acquired during the period when the gaze detection image data, as the second image data, is displayed on the display 72.

[0106] Furthermore, the second image 202 includes achromatic regions (for example, black regions). Therefore, compared to, for example, the case where the second image 202 is a chromatic image, the influence of reflected light on the line-of-sight data can be suppressed.

[0107] Furthermore, the achromatic regions forming the second image 202 are black regions. Therefore, compared to, for example, the case where the achromatic regions forming the second image 202 are gray or white regions, the influence of reflected light on the line-of-sight data can be suppressed.

[0108] Furthermore, the first frame count N1, which is the number of frames of the first image data output to the display 72 per unit time, is greater than the second frame count N2, which is the number of frames of the second image data output to the display 72 per unit time. Therefore, for example, compared to the case where the first frame count N1 is less than the second frame count, the image quality of the image including the first image 200 and the second image 202 (for example, a live view image) can be ensured.

[0109] Furthermore, the processor 42 sets the ratio of the number of first frames and the number of second frames per unit time according to the confidence level of the gaze position detection result. Therefore, the number of first frames and the number of second frames can be adjusted to a ratio corresponding to the confidence level of the gaze position detection result.

[0110] In the first embodiment described above, the processor 42 displays the second image 202, which is a black image, on the display 72 by outputting the second image data stored in the NVM 44 to the display 72. However, the processor 42 may also display the second image 202, which is a black image, on the display 72 by stopping the output of image data to the display 72. In this case, the processor 42 may determine whether the display 72 is displaying the first image 200 or the second image 202, and acquire gaze data when it determines that the display 72 is in the second state (i.e., during the period when the display 72 is in the second state). In this case, the state in which the first image 200 is displayed on the display 72 is an example of the "first state" according to the technology of this disclosure, and the state in which the second image 202 is displayed on the display 72 is an example of the "second state" according to the technology of this disclosure.

[0111] [Second Embodiment] Next, a second embodiment will be described.

[0112] Figure 12 shows the time chart Tc of the image displayed on the display 72 in the second embodiment. In the second embodiment, the second image 202 is modified compared to the first embodiment.

[0113] As an example, as shown in Figure 13, the second image data stored in the NVM44 is image data representing the second image 202, which is a gray image. The gray image representing the second image 202 is, for example, an image in which the entire area is gray. Each time the first image 200 is displayed on the display 72, the NVM44 stores second image data representing the second image 202, which has a brightness corresponding to the brightness of the first image 200.

[0114] The brightness of the second image 202 is set, for example, to the average brightness or representative brightness of part or all of the first image 200 displayed on the display 72 before the second image 202. The number of frames of the first image 200 used to calculate the brightness of the second image 202 is set to a number that ensures the accuracy of the brightness of the second image 202 when setting the brightness of the second image 202 to a brightness corresponding to the brightness of the first image 200. In the second embodiment, the first image 200 is an example of the "second image" relating to the technology of this disclosure, and the second image 202 is an example of the "third image" relating to the technology of this disclosure.

[0115] The second image display control unit 104 acquires the second image data stored in the NVM 44 and displays the second image 202, which is a gray image, on the display 72 based on the acquired second image data. When the second image 202, which is a gray image, is displayed on the display 72, the second image 202, which is a gray image, is projected onto the surface of the cornea 308. When the second image 202, which is a gray image, is projected onto the surface of the cornea 308, the amount of light reflected by the cornea 308 is suppressed compared to when the first image 200 is projected onto the surface of the cornea 308 (see Figure 6). The gray region forming the second image 202 is an example of a "chromatic region" related to the technology of this disclosure.

[0116] As described above, in the second embodiment, the processor 42 sets the brightness of the second image 202 shown by the second image data according to the brightness of the first image 200 shown by the first image data. Therefore, the brightness of the second image 202 can be adjusted to the brightness corresponding to the brightness of the first image 200.

[0117] Furthermore, the second image 202 includes achromatic regions (for example, gray regions). Therefore, compared to, for example, the case where the second image 202 is a chromatic image, the influence of reflected light on the line-of-sight data can be suppressed.

[0118] [Third Embodiment] Next, a third embodiment will be described.

[0119] Figure 14 shows the time chart Tc of the image displayed on the display 72 in the third embodiment. In the third embodiment, the second image 202 is modified compared to the first embodiment.

[0120] As an example, as shown in Figure 15, the NVM44 stores first gaze detection image data and second gaze detection image data. The first gaze detection image data and the second gaze detection image data are image data that represent the black region 202A, respectively. The first gaze detection image data is image data that displays the black region 202A on the upper half of the second image 202. The second gaze detection image data is image data that displays the black region 202A on the lower half of the second image 202. The image memory 32 stores the first image data obtained each time it is captured by the image sensor 24.

[0121] The second image display control unit 104 acquires first image data from the image memory 32 for each frame of the second image 202, and selectively combines the first gaze detection image data and the second gaze detection image data with the acquired first image data. By combining the first gaze detection image data with the first image data, second image data A is generated. Similarly, by combining the second gaze detection image data with the first image data, second image data B is generated.

[0122] The first image data acquired by the second image display control unit 104 is, for example, the first image data showing the first image 200 displayed on the display 72 one frame before the second image 202, but it may also be image data obtained by capturing the subject by the image sensor 24 for each frame of the second image 202.

[0123] The second image display control unit 104 alternately generates second image data A and second image data B, and outputs the generated second image data A or second image data B to the display 72. As a result, the second image 202, which includes a black area 202A in the upper half and an image area 202B in the lower half, and the second image 202, which includes a black area 202A in the lower half and an image area 202B in the upper half, are alternately displayed on the display 72 for each frame of the second image 202. In other words, the positions of the image area 202B and the black area 202A are changed for each frame of the second image 202. Image area 202B is an example of an "image area" related to the technology of this disclosure.

[0124] When the second image 202, which includes the black region 202A, is displayed on the display 72, the second image 202, which includes the black region 202A, is projected onto the surface of the cornea 308. When the second image 202, which includes the black region 202A, is projected onto the surface of the cornea 308, the amount of light reflected by the cornea 308 is suppressed compared to when the first image 200 is projected onto the surface of the cornea 308 (see Figure 6). The black region 202A included in the second image 202 is an example of a "chromatic region" related to the technology of this disclosure.

[0125] As described above, in the third embodiment, the second image 202 includes an image region 202B obtained by imaging with the image sensor 24 and a black region 202A. Therefore, compared to the case where the entire surface of the second image 202 is the image region 202B, for example, the influence of reflected light on the line-of-sight data can be suppressed. Also, compared to the case where the entire surface of the second image 202 is the black region 202A, for example, the image quality of the image including the first image 200 and the second image 202 (for example, a live view image) can be ensured.

[0126] Furthermore, the processor 42 changes the positions of the image region 202B and the black region 202A for each frame of the second image 202. Therefore, compared to, for example, a case where the positions of the image region 202B and the black region 202A are fixed, the image quality of the image including the first image 200 and the second image 202 (for example, a live view image) can be ensured.

[0127] Furthermore, the achromatic regions included in the second image 202 are black regions 202A. Therefore, compared to, for example, the case where the achromatic regions included in the second image 202 are gray or white regions, the influence of reflected light on the line-of-sight data can be suppressed.

[0128] Note that the achromatic areas in the second image 202 may also be gray areas. In this case, for example, the influence of reflected light on the line-of-sight data can be suppressed compared to the case where the achromatic areas in the second image 202 are white areas.

[0129] [Fourth Embodiment] Next, a fourth embodiment will be described.

[0130] Figure 16 shows the time chart Tc of the image displayed on the display 72 in the fourth embodiment. In the fourth embodiment, the second image 202 is modified compared to the first embodiment.

[0131] As an example, as shown in Figure 17, the NVM 44 stores gaze detection image data. The gaze detection image data is image data that shows the black area 202A. The RAM 46 stores gaze data each time the gaze position is detected by the gaze sensor 86. The RAM 46 may store all gaze data obtained since the start of the gaze detection process, or it may store only a predetermined number of gaze data in the order they were acquired, starting with the most recent. The predetermined number is set to, for example, the number that ensures the accuracy of gaze position detection. The predetermined number may be a fixed value predetermined as the number that ensures the accuracy of gaze position detection through actual equipment testing and / or computer simulation, or it may be a variable value that is changed according to instructions given by the user, etc., within a certain limit range. The image memory 32 stores the first image data obtained each time it is captured by the image sensor 24.

[0132] The second image display control unit 104 acquires multiple gaze data from RAM 46 for each frame of the second image 202. The second image display control unit 104 also acquires first image data from image memory 32. Then, for each frame of the second image 202, the second image display control unit 104 generates second image data showing the second image 202, in which the position of the image region 202B is set based on multiple gaze positions (i.e., multiple points P) indicated by the multiple gaze data.

[0133] Image region 202B is set to a position that includes multiple gaze positions. For example, image region 202B may be a rectangular region that includes all of the multiple gaze positions. Alternatively, image region 202B may be a circular region centered on the average of the multiple gaze positions and with a radius that is a constant multiple of the standard deviation. Alternatively, image region 202B may be a circular region centered on a gaze position predicted from the multiple gaze positions. As a result, the second image 202, in which image region 202B is set to a range that includes multiple gaze positions, is displayed on the display 72 for each frame of the second image 202. In other words, the positions of image region 202B and black region 202A are changed for each frame of the second image 202 according to the multiple gaze positions.

[0134] When the second image 202, which includes the black region 202A, is displayed on the display 72, the second image 202, which includes the black region 202A, is projected onto the surface of the cornea 308. When the second image 202, which includes the black region 202A, is projected onto the surface of the cornea 308, the amount of light reflected by the cornea 308 is suppressed compared to when the first image 200 is projected onto the surface of the cornea 308 (see Figure 6). The black region 202A included in the second image 202 is an example of a "chromatic region" related to the technology of this disclosure.

[0135] As described above, in the fourth embodiment, the second image 202 includes an image region 202B obtained by imaging with the image sensor 24 and a black region 202A. Therefore, for example, compared to the case where the entire surface of the second image 202 is the image region 202B, the influence of reflected light on the line-of-sight data can be suppressed. Also, for example, compared to the case where the entire surface of the second image 202 is the black region 202A, the image quality of the image including the first image 200 and the second image 202 (for example, a live view image) can be ensured.

[0136] Furthermore, if the viewing position changes with each frame of the second image 202, the processor 42 changes the positions of the image region 202B and the black region 202A with each frame of the second image 202. Therefore, compared to, for example, the case where the positions of the image region 202B and the black region 202A are fixed, the image quality of the image including the first image 200 and the second image 202 (for example, a live view image) can be ensured.

[0137] Furthermore, the processor 42 sets the position of the image region 202B based on the gaze position for each frame of the second image 202. Therefore, compared to, for example, the case where the position of the image region 202B is set independently of the gaze position, it is possible to ensure image quality of the region corresponding to the gaze position in an image including the first image 200 and the second image 202 (for example, a live view image).

[0138] Furthermore, the achromatic regions included in the second image 202 are black regions 202A. Therefore, compared to, for example, the case where the achromatic regions included in the second image 202 are gray or white regions, the influence of reflected light on the line-of-sight data can be suppressed.

[0139] Note that the achromatic areas in the second image 202 may also be gray areas. In this case, for example, the influence of reflected light on the line-of-sight data can be suppressed compared to the case where the achromatic areas in the second image 202 are white areas.

[0140] [Fifth Embodiment] Next, a fifth embodiment will be described.

[0141] Figures 18 to 21 show the fifth embodiment. The fifth embodiment is modified from the first embodiment as follows.

[0142] As an example, as shown in Figure 18, the processor 42 operates as a light source control unit 122 and an object detection processing unit 124. The light source control unit 122 controls the light source 82 via the light source control circuit 84 by outputting a light source control signal to the light source control circuit 84. When the second image 202, which is a black image, is displayed on the display 72, the light source control unit 122 causes the light source 82 to emit near-infrared light.

[0143] The object detection processing unit 124 acquires gaze data obtained by imaging with the gaze sensor 86. Based on the acquired gaze data, the object detection processing unit 124 determines whether or not an object 304 exists at a position through which near-infrared light passes. The object 304 is, for example, eyeglasses and / or contact lenses. The object 304 can be any object that has the property of absorbing near-infrared light. For example, if the object 304 is eyeglasses and / or contact lenses, the object 304 is located between the cornea 308 and the gaze sensor 86, as an example of a position through which near-infrared light passes. The object 304 can be located at any position as long as it is positioned through which near-infrared light irradiated from the light source 82 passes. The object 304 is an example of an "object" related to the technology of this disclosure.

[0144] The object detection processing unit 124 determines, for example, that an object 304 exists at the position through which the near-infrared light passes if the intensity of the reflected light indicated by the line-of-sight data (i.e., the reflected light of near-infrared light irradiated from the light source 82 and reflected by the cornea 308) is below a threshold. The threshold is determined, for example, based on the intensity of the reflected light when the near-infrared light irradiated from the light source 82 passes through the eyeglasses and / or contact lenses, and the reflected light of the near-infrared light reflected by the cornea 308 passes through the eyeglasses and / or contact lenses again. The threshold may be a fixed value or a variable value that is changed according to instructions given by the user or the like.

[0145] As an example, as shown in Figure 19, if the object detection processing unit 124 determines that no object 304 exists, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the gaze data acquired by the object detection processing unit 124 (i.e., gaze data when a black image is displayed).

[0146] As an example, as shown in Figure 20, if the object detection processing unit 124 determines that an object 304 is present, the second image display control unit 104 displays the second image 202, which is a patterned black image, on the display 72. Specifically, the NVM 44 has a third image data, which is a patterned black image for detecting the line of sight, pre-stored in it. The second image display control unit 104 acquires the third image data stored in the NVM 44 and outputs the acquired third image data to the display 72. As a result, the second image 202, which is a patterned black image, is displayed on the display 72.

[0147] The patterned black image is an image having multiple patterns 206 and black areas 202A. The multiple patterns 206 are, for example, displayed as dots at the four corners of the black areas 202A. The multiple patterns 206 are formed by visible light. Thus, the third image data is image data that includes patterns 206 for detecting the gaze position relative to the display 72. The third image data is an example of the "second gaze detection image data" related to the technology of this disclosure.

[0148] When the second image 202, which is a patterned black image, is displayed on the display 72, multiple patterns 206 and black areas 202A are projected onto the surface of the cornea 308. When the second image 202, which is a patterned black image, is projected onto the surface of the cornea 308, the amount of light reflected by the cornea 308 is suppressed compared to when the first image 200 is projected onto the surface of the cornea 308 (see Figure 6).

[0149] When the second image 202, which is a patterned black image, is displayed on the display 72, the light source control unit 122 stops emitting near-infrared light from the light source 82. The gaze sensor 86 is sensitive to, for example, visible light and near-infrared light.

[0150] The gaze detection processing unit 106 acquires gaze data obtained by the gaze sensor 86 (i.e., gaze data when a patterned black image is displayed). Then, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the acquired gaze data. In the method of detecting the gaze position based on a patterned black image, the gaze position relative to the display 72 is detected based on multiple patterns 206 projected onto the surface of the cornea 308. The method of detecting the gaze position relative to the display 72 based on multiple patterns 206 projected onto the surface of the cornea 308 is the same as the method described in Figure 3.

[0151] Next, the operation of the imaging device 10 according to the fifth embodiment will be explained with reference to Figure 21. Figure 21 shows an example of the flow of the gaze detection process according to the fifth embodiment.

[0152] In the gaze detection process according to the fifth embodiment shown in Figure 21, steps ST30 to ST38 are executed in addition to the gaze detection process according to the first embodiment described above (see Figure 10). Step ST30 is executed after the processing of step ST14 is executed. In the gaze detection process according to the fifth embodiment shown in Figure 21, the processing before step ST14 is the same as in the gaze detection process according to the first embodiment described above.

[0153] In step ST30, the light source control unit 122 irradiates near-infrared light from the light source 82. After the processing in step ST30 is completed, the gaze detection process proceeds to step ST31.

[0154] In step ST31, the object detection processing unit 124 acquires gaze data obtained by the gaze sensor 86. After the processing in step ST31 is completed, the gaze detection process proceeds to step ST32.

[0155] In step ST32, the object detection processing unit 124 determines, based on the line-of-sight data acquired in step ST31, whether or not an object 304 exists at the position through which the near-infrared light passes. If, in step ST32, object 304 does not exist at the position through which the near-infrared light passes, the determination is denied, and the line-of-sight detection process proceeds to step ST33. If, in step ST32, object 304 exists at the position through which the near-infrared light passes, the determination is affirmed, and the line-of-sight detection process proceeds to step ST34.

[0156] In step ST33, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the gaze data acquired in step ST31 (i.e., gaze data when a black image is displayed). After the processing in step ST33 is executed, the gaze detection process proceeds to step ST17 (see Figure 10). In the gaze detection process according to the fifth embodiment shown in Figure 21, the processing from step ST17 onward is the same as that of the gaze detection process according to the first embodiment described above.

[0157] In step ST34, the second image display control unit 104 acquires the third image data stored in the NVM44. After the processing in step ST34 is completed, the gaze detection process proceeds to step ST35.

[0158] In step ST35, the second image display control unit 104 outputs the third image data acquired in step ST34 to the display 72. As a result, the second image 202, which is a patterned black image, is displayed on the display 72. After the processing in step ST35 is completed, the gaze detection process proceeds to step ST36.

[0159] In step ST36, the light source control unit 122 stops the irradiation of near-infrared light from the light source 82. After the processing in step ST36 is completed, the gaze detection process proceeds to step ST37.

[0160] In step ST37, the gaze detection processing unit 106 acquires gaze data obtained by the gaze sensor 86 (i.e., gaze data when a patterned black image is displayed). After the processing in step ST37 is completed, the gaze detection process proceeds to step ST38.

[0161] In step ST38, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the gaze data acquired in step ST37. After the processing in step ST38 is completed, the gaze detection process proceeds to step ST17 (see Figure 10).

[0162] As described above, in the fifth embodiment, if no object 304 is present, the processor 42 outputs a second image data showing a second image 202, which is a black image, to the display 72 while irradiating it with near-infrared light from the light source 82. Therefore, the line of sight position can be detected during the period when the black image shown by the second image data is displayed on the display 72.

[0163] On the other hand, if an object 304 is present, the processor 42 stops irradiating near-infrared light from the light source 82 and outputs a third image data, which is a patterned black image (second image 202), to the display 72. Therefore, even when an object 304 is present, the line of sight position can be detected based on the patterned black image.

[0164] Furthermore, the third image data is image data that includes a pattern 206 for detecting the gaze position relative to the display 72. Therefore, even if the second image 202, which includes a black area 202A, is displayed on the display 72, the gaze position can be detected based on the pattern 206.

[0165] [Sixth Embodiment] Next, a sixth embodiment will be described.

[0166] Figures 22 to 25 show the sixth embodiment. The sixth embodiment is modified from the fifth embodiment as follows. In the example shown in the sixth embodiment, object 304 is located at the position through which near-infrared light passes.

[0167] Figure 22 shows, as an example, a second image 202, which is a patterned black image, displayed on the display 72, while the user is blinking (i.e., closing their eyes 302). As shown in Figure 22 as an example, the processor 42 operates as a blink detection unit 132 and a memory control unit 134.

[0168] The blink detection unit 132 acquires gaze data obtained from the gaze sensor 86. Based on the acquired gaze data, the blink detection unit 132 determines whether or not the user is blinking (i.e., whether or not the eyes 302 are closed). The determination of whether or not the user is blinking is achieved, for example, by performing image processing such as AI-based image recognition and / or pattern matching.

[0169] If the blink detection unit 132 determines that the user is blinking, the memory control unit 134 stores the gaze data acquired by the blink detection unit 132 (i.e., gaze data when the eyes 302 are closed) in the NVM 44. When the second image 202, which is a patterned black image, is displayed on the display 72 and the user's eyes 302 are closed, multiple patterns 206 are projected onto the surface of the object 304.

[0170] Figure 23 shows, as an example, a second image 202, which is a patterned black image, displayed on the display 72, and the user is not blinking (i.e., eyes 302 are open). Figure 23 also shows, as an example, a case where gaze data is stored in the NVM 44 when eyes 302 are closed.

[0171] As an example, as shown in Figure 23, the processor 42 operates as a data determination unit 136. The data determination unit 136 determines whether or not gaze data for when the eyes 302 are closed is stored in the NVM 44.

[0172] If the blink detection unit 132 determines that the user is not blinking, and the data determination unit 136 determines that gaze data for when the eyes 302 are closed is stored in the NVM 44, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the degree of difference between the gaze data for when the eyes 302 are closed stored in the NVM 44 and the gaze data for when the eyes 302 are open acquired by the blink detection unit 132.

[0173] For example, the gaze detection processing unit 106 calculates the difference between gaze data when the eye 302 is closed and gaze data when the eye 302 is open. By calculating the difference between gaze data when the eye 302 is closed and gaze data when the eye 302 is open, multiple patterns 206 projected onto the surface of object 304 cancel each other out, and an image showing multiple patterns 206 projected onto the surface of the cornea 308 is obtained. Then, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the acquired difference. In other words, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on multiple patterns 206 projected onto the surface of the cornea 308.

[0174] Eye-line data acquired when object 304 is present and eye 302 is closed is an example of "first eye-line data" relating to the technology of this disclosure. Eye-line data acquired when object 304 is present and eye 302 is open is an example of "second eye-line data" relating to the technology of this disclosure.

[0175] Figure 24 shows, as an example, a case where the second image 202, which is a patterned black image, is displayed on the display 72, and the user is not blinking (i.e., eyes 302 are open), but gaze data for when eyes 302 are closed is not stored in the NVM 44.

[0176] As an example, as shown in Figure 24, if the blink determination unit 132 determines that the user is not blinking, and the data determination unit 136 determines that no gaze data for when the eyes 302 are closed is stored in the NVM 44, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the gaze data for when the eyes 302 are open, which was acquired by the blink determination unit 132.

[0177] Next, the operation of the imaging device 10 according to the sixth embodiment will be explained with reference to Figure 25. Figure 25 shows an example of the flow of the gaze detection process according to the sixth embodiment.

[0178] In the gaze detection process according to the sixth embodiment shown in Figure 25, steps ST40 to ST45 are executed in addition to the gaze detection process according to the fifth embodiment described above (see Figure 17). Step ST40 is executed after step ST36. In the gaze detection process according to the sixth embodiment shown in Figure 25, the processing before step ST36 is the same as in the gaze detection process according to the fifth embodiment described above.

[0179] In step ST40, the blink detection unit 132 acquires gaze data obtained by the gaze sensor 86. After the processing in step ST40 is completed, the gaze detection process proceeds to step ST41.

[0180] In step ST41, the blink determination unit 132 determines whether the user is blinking (i.e., whether the eyes 302 are closed) based on the gaze data acquired in step ST40. If the user is blinking in step ST41 (i.e., the eyes 302 are closed), the determination is affirmed, and the gaze detection process proceeds to step ST42. If the user is not blinking in step ST41 (i.e., the eyes 302 are open), the determination is denied, and the gaze detection process proceeds to step ST43.

[0181] In step ST42, the memory control unit 134 stores the gaze data acquired in step ST40 (i.e., gaze data when the eyes 302 are closed) in the NVM 44. After the processing in step ST42 is executed, the gaze detection process proceeds to step ST17 (see Figure 10). In the gaze detection process according to the sixth embodiment shown in Figure 25, the processing from step ST17 onward is the same as that of the gaze detection process according to the first embodiment described above.

[0182] In step ST43, the data determination unit 136 determines whether or not gaze data for when the eye 302 is closed is stored in the NVM 44. If, in step ST43, gaze data for when the eye 302 is closed is not stored in the NVM 44, the determination is denied and the gaze detection process proceeds to step ST44. If, in step ST43, gaze data for when the eye 302 is closed is stored in the NVM 44, the determination is affirmed and the gaze detection process proceeds to step ST45.

[0183] In step ST44, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the gaze data obtained in step ST40 when the eye 302 is open. After the processing in step ST45 is executed, the gaze detection process proceeds to step ST17 (see Figure 10).

[0184] In step ST45, the gaze detection processing unit 106 detects the gaze position relative to the display 72 based on the degree of difference between the gaze data stored in the NVM 44 when the eyes 302 are closed and the gaze data acquired in step ST40 when the eyes 302 are open. After the processing in step ST43 is executed, the gaze detection process moves to step ST17 (see Figure 10).

[0185] As described above, in the sixth embodiment, the processor 42 detects the line of sight position based on the degree of difference between the line of sight data acquired when the object 304 is present and the eye 302 is closed, and the line of sight data acquired when the object 304 is present and the eye 302 is open. Therefore, even when the object 304 is present, the line of sight position can be detected based on the visible light of the pattern 206 reflected by the cornea 308.

[0186] [Seventh Embodiment] Next, a seventh embodiment will be described.

[0187] Figure 26 shows an example of a first image 200 displayed on the display 72 in the seventh embodiment. The first image 200 displayed on the display 72 includes a first image 200 in which the line of sight of the subject appearing as an image in the first image 200 and the user do not coincide (i.e., they are not making eye contact), and a first image 200 in which the line of sight of the subject appearing as an image in the first image 200 and the user do coincide (i.e., they are making eye contact).

[0188] The frame 210 indicates the user's line of sight relative to the display 72, and the arrow 212 indicates the direction of the line of sight of the subject appearing as an image in the first image 200. If the arrow 212 is outside the frame 210, it means that the line of sight of the subject appearing as an image in the first image 200 and the user do not match. If the arrow 212 is within the frame 210, it means that the line of sight of the subject appearing as an image in the first image 200 and the user do match. The size of the frame 210 is set to a size that ensures sufficient detection accuracy in determining whether or not the line of sight matches.

[0189] In the seventh embodiment, the processing when the second image 202 is displayed is modified compared to the first embodiment as follows. As an example, as shown in Figure 27, the processor 42 operates as a gaze determination unit 142 and an image capture control unit 144.

[0190] The gaze determination unit 142 acquires the most recent first image data from the first image data stored in the image memory 32 and detects the first gaze direction of the subject appearing in the first image 200 indicated by the acquired first image data. The first gaze direction of the subject is detected, for example, by performing image processing on the first image 200. The first gaze direction is an example of the "first gaze direction" related to the technology of this disclosure.

[0191] Furthermore, the gaze determination unit 142 detects a second gaze direction corresponding to the gaze position detected by the gaze detection processing unit 106. The second gaze direction is an example of the "second gaze direction" according to the technology of this disclosure. The second gaze direction is identified, for example, based on a line 312 connecting the coordinates of the center 306A of the eyeball 306 and the coordinates of the center 310A of the pupil 310, as explained in Figure 3. The gaze determination unit 142 then determines whether the first gaze direction and the second gaze direction match (i.e., whether the gaze directions of the subject and the user match) based on the second gaze direction corresponding to the gaze position detected by the gaze detection processing unit 106 and the first gaze direction of the subject.

[0192] If the gaze determination unit 142 determines that the first gaze direction and the second gaze direction do not match, the imaging control unit 144 does not control the image sensor 24 to capture an image of the subject.

[0193] As an example, as shown in Figure 28, when the line of sight determination unit 142 determines that the first line of sight direction and the second line of sight direction coincide, the imaging control unit 144 outputs an imaging control signal to the image sensor control circuit 26, causing the image sensor 24 to image the subject (not shown) via the image sensor control circuit 26.

[0194] The processor 42 operates as a memory control unit 134. The memory control unit 134 acquires imaging data obtained when a subject is imaged by the image sensor 24. The imaging data is image data that includes a subject whose line of sight coincides with that of the user. The memory control unit 134 then stores the acquired imaging data in the NVM 44. As a result, imaging data indicating the recorded image to be recorded as data is stored in the NVM 44. The imaging data is an example of the "third image data" related to the technology of this disclosure.

[0195] Furthermore, the imaging device 10 has an operating mode called a gaze-matching imaging mode in which, when the gaze direction of the subject and the user match (i.e., their eyes are meeting), it captures an image of the subject and stores the image data representing the recorded image in the NVM44. The imaging device 10 then executes the gaze-matching imaging mode when it switches to the gaze-matching imaging mode.

[0196] Next, the operation of the imaging device 10 according to the seventh embodiment will be explained with reference to Figure 29. Figure 29 shows an example of the flow of the gaze detection process according to the seventh embodiment.

[0197] In the gaze detection process according to the seventh embodiment shown in Figure 29, steps ST50 to ST52 are executed in addition to the gaze detection process according to the first embodiment described above (see Figure 10). Step ST50 is executed after step ST16. In the gaze detection process according to the seventh embodiment shown in Figure 29, the processing before step ST16 is the same as in the gaze detection process according to the first embodiment described above.

[0198] In step ST50, the gaze determination unit 142 acquires the most recent first image data (i.e., the newest first image data) from the first image data stored in the image memory 32, and detects the first gaze direction of the subject appearing in the first image 200 indicated by the acquired first image data. Then, the gaze determination unit 142 determines whether the first gaze direction and the second gaze direction match, based on the second gaze direction corresponding to the gaze position detected by the gaze detection processing unit 106 and the first gaze direction of the subject.

[0199] In step ST50, if the first gaze direction and the second gaze direction match, the determination is affirmed, and the gaze detection process proceeds to step ST51. In step ST50, if the first gaze direction and the second gaze direction do not match, the determination is denied, and the gaze detection process proceeds to step ST17 (see Figure 10) without performing the process of causing the image sensor 24 to capture an image of the subject. In the gaze detection process according to the seventh embodiment shown in Figure 25, the processing from step ST17 onward is the same as the gaze detection process according to the first embodiment described above.

[0200] In step ST51, the imaging control unit 144 instructs the image sensor 24 to capture an image of the subject. After the processing in step ST51 is completed, the gaze detection process proceeds to step ST52.

[0201] In step ST52, the memory control unit 134 acquires the imaging data obtained when the subject is imaged by the image sensor 24 in step ST51. The memory control unit 134 then stores the acquired imaging data in the NVM 44. After the processing in step ST52 is executed, the gaze detection process proceeds to step ST17 (see Figure 10).

[0202] As described above, in the seventh embodiment, the processor 42 detects a first line of sight direction of the subject appearing in the first image 200 shown by the first image data. The processor 42 also detects a second line of sight direction corresponding to the line of sight position. Then, based on the first and second line of sight directions, the processor 42 acquires imaging data obtained by imaging with the image sensor 24. Therefore, imaging data corresponding to the first and second line of sight directions can be acquired.

[0203] Furthermore, the processor 42 acquires imaging data, for example, when the first line of sight direction and the second line of sight direction coincide. Therefore, imaging data in which the first line of sight direction and the second line of sight direction coincide can be acquired.

[0204] [Eighth Embodiment] Next, the eighth embodiment will be described.

[0205] Figures 30 to 32 show the eighth embodiment. The eighth embodiment is modified from the first embodiment as follows. As an example, as shown in Figure 30, the processor 42 operates as an attitude determination unit 152 and an orientation information generation unit 154.

[0206] The posture determination unit 152 acquires acceleration data from the acceleration sensor 22. Based on the acceleration data, the posture determination unit 152 determines whether the posture of the imaging device 10 is prone or supine. The posture determination unit 152 determines that the posture of the imaging device 10 is prone or supine if the direction of the acceleration indicated by the acceleration data coincides with the roll axis direction of the imaging device 10. The acceleration data is an example of "acceleration data" related to the technology of this disclosure.

[0207] The orientation information generation unit 154 does not generate orientation information regarding the orientation of the imaging device 10 if the orientation determination unit 152 determines that the orientation of the imaging device 10 is not prone or supine. Orientation information indicates whether the orientation of the imaging device 10 is horizontal or vertical. The orientation of the imaging device 10 is horizontal when, for example, the horizontal direction of the imaging device 10 (i.e., the pitch axis direction) is parallel to the horizontal direction of the user's eye 302, and the orientation of the imaging device 10 is vertical when, for example, the vertical direction of the imaging device 10 (i.e., the yaw axis direction) is parallel to the horizontal direction of the user's eye 302.

[0208] The orientation information generation unit 154 generates orientation information regarding the orientation of the imaging device 10 when the posture determination unit 152 determines that the orientation of the imaging device 10 is prone or supine. Specifically, the orientation information generation unit 154 detects the direction of movement of the gaze position (i.e., the direction of the eye 302) and generates orientation information regarding the orientation of the imaging device 10 based on the detected direction of movement of the gaze position. For example, the orientation information generation unit 154 identifies the trajectories of multiple gaze positions based on multiple gaze data stored in the RAM 46, and detects the direction of movement of the gaze position based on the identified trajectories of multiple gaze positions.

[0209] As an example, as shown in Figure 31, when a user has the imaging device 10 capture an image of a subject, the user moves their line of sight horizontally (i.e., tilts their eyes 302 horizontally). When the direction of movement of the line of sight relative to the imaging device 10 is horizontal to the imaging device 10, the orientation information generation unit 154 generates horizontal orientation information indicating that the imaging device 10 is oriented horizontally. On the other hand, when the direction of movement of the line of sight relative to the imaging device 10 is vertical to the imaging device 10, the orientation information generation unit 154 generates vertical orientation information indicating that the imaging device 10 is oriented vertically. vertical Orientation information is an example of "orientation information" relating to the technology disclosed herein.

[0210] The horizontal and vertical orientation information is added to the image data obtained after the orientation information has been generated by the orientation information generation unit 154. When the imaging device 10 is oriented horizontally, the image data obtained is image data showing a horizontal image 214, and when the imaging device 10 is oriented vertically, the image data obtained is image data showing a vertical image 216.

[0211] The image data shown in Figure 31 may be the first image data showing the first image 200 described above, or it may be imaging data obtained when the gaze direction of the subject and the user coincide (see Figure 28). In addition, the image data shown in Figure 31 may be image data obtained by imaging with the image sensor 24, in addition to the first image data and imaging data. The image data shown in Figure 31 is an example of the "fourth image data" related to the technology of this disclosure.

[0212] Next, the operation of the imaging device 10 according to the eighth embodiment will be explained with reference to Figure 32. Figure 32 shows an example of the flow of the gaze detection process according to the eighth embodiment.

[0213] In the gaze detection process according to the eighth embodiment shown in Figure 32, steps ST60 to ST61 are executed in addition to the gaze detection process according to the first embodiment described above (see Figure 10). Step ST60 is executed after step ST16. In the gaze detection process according to the eighth embodiment shown in Figure 32, the processing before step ST16 is the same as in the gaze detection process according to the first embodiment described above.

[0214] In step ST60, the posture determination unit 152 acquires acceleration data from the acceleration sensor 22. Based on the acceleration data, the posture determination unit 152 determines whether the posture of the imaging device 10 is prone or supine. If the posture of the imaging device 10 is prone or supine in step ST60, the determination is affirmed, and the gaze detection process proceeds to step ST61. If the posture of the imaging device 10 is not prone or supine in step ST60, the determination is denied, and the gaze detection process proceeds to step ST17 (see Figure 10) without performing the process of generating orientation information for the imaging device 10. In the gaze detection process according to the eighth embodiment shown in Figure 32, the processing from step ST17 onward is the same as the gaze detection process according to the first embodiment described above.

[0215] In step ST61, the orientation information generation unit 154 detects the direction of movement of the gaze position (i.e., the orientation of the eye 302) and generates orientation information regarding the orientation of the imaging device 10 based on the detected direction of movement of the gaze position. After the processing in step ST61 is executed, the gaze detection process proceeds to step ST17 (see Figure 10).

[0216] As described above, in the eighth embodiment, the processor 42 generates orientation information regarding the orientation of the imaging device 10 based on the acceleration data from the acceleration sensor 22 and the line of sight position. Therefore, for example, even if the imaging device 10 is lying face down or face up, orientation information corresponding to the line of sight can be obtained.

[0217] Furthermore, the processor 42 adds orientation information to the image data obtained by the image sensor 24. Therefore, even if the imaging device 10 is lying face down or face up, an image with an orientation corresponding to the orientation of the imaging device 10 can be obtained.

[0218] [Ninth Embodiment] Next, a ninth embodiment will be described.

[0219] Figures 33 to 35 show the ninth embodiment. The ninth embodiment is modified from the first embodiment as follows. As an example, as shown in Figure 33, the processor 42 operates as a gaze detection target determination unit 162, a gaze position movement direction determination unit 164, and a touch panel control unit 166.

[0220] The gaze detection target determination unit 162 determines which of the left and right eyes is the target of gaze detection based on the gaze data. The gaze detection target determination unit 162 determines which of the left and right eyes is the target of gaze detection by performing image processing on the gaze data, for example, AI-based image recognition processing and / or pattern matching. The process of determining which of the left and right eyes is the target of gaze detection is the first determination process, and the first determination process is an example of the "first determination process" related to the technology of this disclosure.

[0221] The gaze position movement direction determination unit 164 determines the orientation of the left or right eye that has been determined to be the target of gaze detection based on the gaze position. Specifically, the gaze position movement direction determination unit 164 detects the direction of movement of the gaze position (i.e., the orientation of the eye 302) and determines whether the detected direction of movement of the gaze position is in the horizontal or vertical direction of the imaging device 10. For example, the gaze position movement direction determination unit 164 identifies the trajectories of multiple gaze positions based on multiple gaze data stored in the RAM 46 and detects the direction of movement of the gaze position based on the identified trajectories of multiple gaze positions. The orientation of the eye 302 is detected by detecting the direction of movement of the gaze position. The process of determining the orientation of the left or right eye that has been determined to be the target of gaze detection based on the gaze position is a second determination process, and the second determination process is an example of a "second determination process" related to the technology of this disclosure.

[0222] The touch panel control unit 166 sets the effective and / or invalid areas of the touch panel 34A based on the determination result by the gaze detection target determination unit 162 and the determination result by the gaze position movement direction determination unit 164. Specifically, the touch panel control unit 166 sets the effective and / or invalid areas of the touch panel 34A by outputting area control signals indicating the effective and / or invalid areas of the touch panel 34A to the touch panel display control circuit 36.

[0223] As an example, as shown in Figure 34, the touch panel control unit 166 sets the left half of the touch panel 34A as an invalid area and the right half of the touch panel 34A as an active area when the gaze detection target is the right eye and the direction of movement of the gaze position is in the horizontal direction of the imaging device 10. The touch panel control unit 166 sets the entire area of ​​the touch panel 34A as an active area when the gaze detection target is the right eye and the direction of movement of the gaze position is in the vertical direction of the imaging device 10.

[0224] Furthermore, if the gaze detection target is the left eye and the direction of movement of the gaze position is in the horizontal direction of the imaging device 10, the touch panel control unit 166 sets the left half of the touch panel 34A as an active area and the right half of the touch panel 34A as an inactive area. If the gaze detection target is the left eye and the direction of movement of the gaze position is in the vertical direction of the imaging device 10, the touch panel control unit 166 sets the entire area of ​​the touch panel 34A as an inactive area. The touch panel 34A is used, for example, to display and move a focus frame for adjusting the focus.

[0225] Next, the operation of the imaging device 10 according to the ninth embodiment will be explained with reference to Figure 35. Figure 35 shows an example of the flow of the gaze detection process according to the ninth embodiment.

[0226] In the gaze detection process according to the ninth embodiment shown in Figure 35, steps ST70 to ST76 are executed between the processing of step ST16 and step ST17, compared to the gaze detection process according to the first embodiment described above (see Figure 10). In the gaze detection process according to the ninth embodiment shown in Figure 35, the processing before step ST16 is the same as that of the gaze detection process according to the first embodiment described above.

[0227] In step ST70, the gaze detection target determination unit 162 determines whether the left eye or the right eye is the target of gaze detection based on the gaze data acquired in step ST15. If the right eye is the target of gaze detection in step ST70, the gaze detection process proceeds to step ST71. If the left eye is the target of gaze detection in step ST70, the gaze detection process proceeds to step ST74.

[0228] In step ST71, the gaze position movement direction determination unit 164 determines whether the direction of movement of the gaze position (i.e., the orientation of the eye 302) is in the horizontal or vertical direction of the imaging device 10. If, in step ST71, the direction of movement of the gaze position is in the horizontal direction of the imaging device 10, the gaze detection process proceeds to step ST72. If, in step ST71, the direction of movement of the gaze position is in the vertical direction of the imaging device 10, the gaze detection process proceeds to step ST73.

[0229] In step ST72, the touch panel control unit 166 sets the left half of the touch panel 34A as an invalid area and the right half of the touch panel 34A as an active area. After the processing in step ST72 is executed, the gaze detection process proceeds to step ST17 (see Figure 10). In the gaze detection process according to the ninth embodiment shown in Figure 35, the processing from step ST17 onward is the same as that of the gaze detection process according to the first embodiment described above.

[0230] In step ST73, the touch panel control unit 166 sets the entire area of ​​the touch panel 34A as the active area. After the processing in step ST73 is completed, the gaze detection process proceeds to step ST17 (see Figure 10).

[0231] In step ST74, the gaze position movement direction determination unit 164 determines whether the direction of movement of the gaze position (i.e., the orientation of the eye 302) is in the horizontal or vertical direction of the imaging device 10. If, in step ST74, the direction of movement of the gaze position is in the horizontal direction of the imaging device 10, the gaze detection process proceeds to step ST75. If, in step ST74, the direction of movement of the gaze position is in the vertical direction of the imaging device 10, the gaze detection process proceeds to step ST76.

[0232] In step ST75, the touch panel control unit 166 sets the left half of the touch panel 34A as an active area and the right half of the touch panel 34A as an inactive area. After the processing in step ST75 is completed, the gaze detection process proceeds to step ST17 (see Figure 10).

[0233] In step ST76, the touch panel control unit 166 sets the entire area of ​​the touch panel 34A as an invalid area. After the processing in step ST76 is executed, the gaze detection process proceeds to step ST17 (see Figure 10).

[0234] As described above, in the ninth embodiment, the processor 42 performs a first determination process to determine which of the left and right eyes is the target of gaze detection based on gaze data, and a second determination process to determine the orientation of the left or right eye that was determined to be the target of gaze detection in the first determination process, based on the gaze position. Then, the processor 42 sets the effective area and / or invalid area of ​​the touch panel 34A based on the first determination result from the first determination process and the second determination result from the second determination process.Therefore, for example, compared to the case where the entire surface of the touch panel 34A is kept set as the effective area, malfunctions of the imaging device 10 caused by the user's nose (not shown) touching the touch panel 34A can be suppressed.

[0235] [Tenth Embodiment] Next, the tenth embodiment will be described.

[0236] Figures 36 to 38 show a tenth embodiment. The tenth embodiment is modified from the first embodiment as follows. As an example, as shown in Figure 36, the processor 42 operates as a pupil size detection unit 172 and a brightness setting unit 174.

[0237] The pupil size detection unit 172 detects the size of the pupil 310 of the eye 302 based on the gaze data. The size of the pupil 310 may be detected based on the area of ​​the pupil 310 shown in the image indicated by the gaze data, or it may be detected based on the diameter of the pupil 310 shown in the image indicated by the gaze data. The size of the pupil 310 changes depending on the brightness of the environment in which the imaging device 10 is used.

[0238] The brightness setting unit 174 sets the brightness of the display 72 when the first image 200 is displayed, based on the result of the pupil size detection unit 172 detecting the size of the pupil 310.

[0239] As an example, as shown in Figure 37, the brightness setting unit 174 brightens the brightness of the display 72 when the first image 200 is displayed as the size of the pupil 310 decreases. Conversely, the brightness setting unit 174 darkens the brightness of the display 72 when the first image 200 is displayed as the size of the pupil 310 increases. When the brightness of the display 72 is set by the brightness setting unit 174, the brightness of the display 72 when the first image 200 is displayed is adjusted to the brightness set by the brightness setting unit 174.

[0240] Next, the operation of the imaging device 10 according to the 10th embodiment will be explained with reference to Figure 38. Figure 38 shows an example of the flow of the gaze detection process according to the 10th embodiment.

[0241] In the gaze detection process according to the 10th embodiment shown in Figure 38, steps ST80 to ST81 are executed in addition to the gaze detection process according to the first embodiment described above (see Figure 10). Step ST80 is executed after step ST16. In the gaze detection process according to the 10th embodiment shown in Figure 38, the processing before step ST16 is the same as in the gaze detection process according to the first embodiment described above.

[0242] In step ST80, the pupil size detection unit 172 detects the size of the pupil 310 of the eye 302 based on the gaze data. After the processing in step ST80 is completed, the gaze detection process proceeds to step ST81.

[0243] In step ST81, the brightness setting unit 174 sets the brightness of the display 72 when the first image 200 is displayed, based on the result of the pupil size detection unit 172 detecting the size of the pupil 310. After the processing in step ST81 is executed, the gaze detection process moves to step ST17 (see Figure 10). In the gaze detection process according to the 10th embodiment shown in Figure 38, the processing from step ST17 onward is the same as the gaze detection process according to the first embodiment described above.

[0244] As described above, in the tenth embodiment, the processor 42 detects the size of the pupil 310 based on the gaze data and adjusts the brightness of the display 72 based on the result of detecting the size of the pupil 310. Therefore, for example, the brightness of the display 72 can be adjusted according to light adaptation and / or dark adaptation.

[0245] Although the first to tenth embodiments have been described above, the above embodiments and modifications can be combined with each other as long as no contradictions arise. Furthermore, if there are multiple overlapping steps when the above embodiments and modifications are combined, a priority order may be assigned to the multiple steps depending on various conditions.

[0246] Furthermore, although a digital camera is exemplified as the imaging device 10 in the above embodiment, the technology of this disclosure is not limited thereto, and for example, a terminal device that functions as the imaging device 10 (e.g., a tablet terminal, a smart device, or a wearable terminal) may be used. Also, the digital camera as the imaging device 10 may be any type of digital camera, such as a compact digital camera, a mirrorless SLR camera, or a digital SLR camera. The digital camera may be a digital camera with interchangeable lenses or a digital camera with a fixed lens. In addition, the technology of this disclosure may be applied to various information processing devices other than the imaging device 10 (e.g., a cell observation device, an ophthalmic observation device, or a surgical microscope).

[0247] Furthermore, in the above embodiment, the line of sight position of one eye is detected, but the line of sight positions of both eyes may also be detected.

[0248] Furthermore, although the above embodiments illustrate the use of a processor 42, at least one other CPU, at least one GPU, and / or at least one TPU may be used instead of, or in conjunction with, the processor 42.

[0249] Furthermore, although the above embodiments have described examples in which the gaze detection program 100 is stored in the NVM 44, the technology of this disclosure is not limited thereto. For example, the gaze detection program 100 may be stored in a portable, non-temporary, computer-readable storage medium such as an SSD or USB memory (hereinafter simply referred to as "non-temporary storage medium"). The gaze detection program 100 stored in the non-temporary storage medium is installed in the computer 20 of the imaging device 10, and the processor 42 executes processing according to the gaze detection program 100.

[0250] Alternatively, the gaze detection program 100 may be stored in a storage device such as another computer or server connected to the imaging device 10 via a network, and the gaze detection program 100 may be downloaded and installed on the computer 20 in response to a request from the imaging device 10.

[0251] Furthermore, it is not necessary to store the entire gaze detection program 100 in a storage device such as another computer or server connected to the imaging device 10, or in the NVM 44; it is acceptable to store only a portion of the gaze detection program 100.

[0252] Furthermore, although the imaging device 10 has a built-in computer 20, the technology of this disclosure is not limited to this, and for example, the computer 20 may be provided outside the imaging device 10.

[0253] Furthermore, while the above embodiments illustrate a computer 20 including a processor 42, an NVM 44, and RAM 46, the technology of this disclosure is not limited thereto, and devices including ASICs, FPGAs, and / or PLDs may be used instead of the computer 20. Alternatively, a combination of hardware and software configurations may be used instead of the computer 20.

[0254] Furthermore, the following types of processors can be used as hardware resources to perform the gaze detection processing described in each of the above embodiments. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for performing gaze detection processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to perform particular processing, such as FPGAs, PLDs, or ASICs. Each of these processors has built-in or connected memory, and each processor performs gaze detection processing by using this memory.

[0255] The hardware resources that perform gaze detection processing may consist of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources that perform gaze detection processing may consist of a single processor.

[0256] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs gaze detection processing. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform gaze detection processing, on a single IC chip, as exemplified by SoCs. Thus, gaze detection processing is realized using one or more of the above types of processors as hardware resources.

[0257] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits combining circuit elements such as semiconductor devices. Also, the gaze detection process described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0258] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0259] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0260] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. An information processing device equipped with a processor, The aforementioned processor, In the mode that performs gaze detection processing, First image data and second image data are acquired. Selectively output the first image data and the second image data to the display, During the period when the second image data is output to the display, gaze data is acquired. Based on the aforementioned gaze data, the gaze position relative to the display is detected. During the period when the first image data is output to the display, the gaze position is not detected. The first image data and the second image data are frame-by-frame image data in the mode, The first image data is image data obtained by capturing with the first image sensor. The second image data is image data showing an image for gaze detection that includes a grayscale region. Information processing device.

2. The information processing device further comprises a second image sensor, The gaze detection image includes an image region obtained by capturing an image with the second image sensor and the grayscale region. The processor changes the position of the image region and the achromatic region for each frame of the gaze detection image. The information processing apparatus according to claim 1.

3. The processor sets the position of the image region based on the gaze position for each frame of the gaze detection image. The information processing apparatus according to claim 2.

4. The achromatic region is a black region or a gray region. The information processing apparatus according to claim 1.

5. The information processing device further comprises a light source that illuminates the eye which is the target of gaze detection, The aforementioned processor, If there is no object in the position through which the light passes, the second image data is output to the display while the light is irradiated from the light source. If the object is present, the irradiation of light from the light source is stopped, and the second image data is output to the display. The information processing apparatus according to claim 1.

6. The second image data is image data that includes a pattern for detecting the gaze position relative to the display. The information processing apparatus according to claim 5.

7. The processor detects the gaze position based on the degree of difference between the first gaze data, which is obtained as gaze data when the object is present and the eyes are closed, and the second gaze data, which is obtained as gaze data when the object is present and the eyes are open. The information processing apparatus according to claim 5.

8. The information processing device further comprises a third image sensor, The processor detects the first line of sight direction of the subject appearing in the first image shown by the first image data, Based on the second line of sight direction corresponding to the aforementioned line of sight position and the first line of sight direction, the third image data obtained by imaging with the third image sensor is acquired. The information processing apparatus according to claim 1.

9. The processor acquires the third image data when the first line of sight direction and the second line of sight direction coincide. The information processing apparatus according to claim 8.

10. The first number of frames, which is the number of frames of the first image data output to the display per unit time, is greater than the second number of frames, which is the number of frames of the second image data output to the display per unit time. The information processing apparatus according to claim 1.

11. The processor sets the ratio of the number of first frames and the number of second frames per unit time according to the reliability of the detection result of the gaze position. The information processing apparatus according to claim 10.

12. The processor sets the brightness of the third image shown by the second image data according to the brightness of the second image shown by the first image data. The information processing apparatus according to claim 1.

13. The information processing device further comprises an acceleration sensor, The processor generates orientation information relating to the orientation of the information processing device based on the acceleration data from the acceleration sensor and the line of sight. The information processing apparatus according to claim 1.

14. The information processing device further comprises a fourth image sensor, The processor adds the orientation information to the fourth image data obtained by imaging with the fourth image sensor. The information processing apparatus according to claim 13.

15. The information processing device further comprises a touch panel, The aforementioned processor, A first determination process is performed to determine whether the left eye or the right eye is the target of gaze detection based on the gaze data, and a second determination process is performed to determine the orientation of the left eye or the right eye that was determined to be the target of gaze detection in the first determination process, based on the gaze position. Based on the first determination result from the first determination process and the second determination result from the second determination process, the effective area and / or invalid area of ​​the touch panel are set. The information processing apparatus according to claim 1.

16. The processor is Based on the aforementioned gaze data, the pupil size is detected. Based on the results of detecting the pupil size, adjust the brightness of the display. The information processing apparatus according to claim 1.

17. The information processing device is an imaging device. The information processing apparatus according to claim 1.

18. In a mode that performs gaze detection processing, To acquire the first image data and the second image data, Selectively outputting the first image data and the second image data to the display, Acquiring gaze data during the period when the second image data is output to the display, and Based on the aforementioned gaze data, the gaze position relative to the display is detected. Equipped with, During the period when the first image data is output to the display, the gaze position is not detected. The first image data and the second image data are frame-by-frame image data in the mode, The first image data is image data obtained by capturing with the first image sensor. The second image data is image data showing an image for gaze detection that includes a grayscale region. Information processing methods.

19. A program for causing a computer to perform a process, The aforementioned process is, In the mode that performs gaze detection processing, To acquire the first image data and the second image data, Selectively outputting the first image data and the second image data to the display, Acquiring gaze data during the period when the second image data is output to the display, and Based on the aforementioned gaze data, the gaze position relative to the display is detected. Equipped with, During the period when the first image data is output to the display, the gaze position is not detected. The first image data and the second image data are frame-by-frame image data in the mode, The first image data is image data obtained by capturing with the first image sensor. The second image data is image data showing an image for gaze detection that includes a grayscale region. program.

Images