Information processing apparatus

Abstract

To provide a technology capable of processing gaze information so that an action intended by a user is performed.SOLUTION: The information processing apparatus comprises: first acquisition means for acquiring information indicating a first line of sight position of a user; second acquisition means for acquiring size information, which is information on the size of an object perceived by the user, for one or more objects existing in the direction to which the user's line of sight is directed based on the information indicating the first line of sight position; and third acquisition means for acquiring information indicating the second line of sight position by reducing changes in the information indicating the first line of sight position by a reduction degree based on the size information.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an information processing device, and more particularly to a technique for processing line-of-sight information. [Background technology]

[0002] A technology has been proposed that detects the gaze position of a user who is looking within the display area of ​​an imaging device equipped with an electronic viewfinder or within the display area of ​​a head mounted display device (HMD) and displays an item at the gaze position. In the imaging device or HMD, it is also possible to display an image (image of real space) acquired by an imaging element that receives a light beam that has passed through an imaging optical system. It is also possible to detect and recognize various types of subjects (real objects) from the image acquired by the imaging element. In the HMD, it is also possible to display virtual objects, for example, to display virtual objects so as to place the virtual objects in real space. It is also possible to select a subject in real space or a virtual object using the detected gaze position.

[0003] The human eye has a characteristic that involuntary and minute eye movements (fixational eye movement) always occur even when staring at one point, and Patent Documents 1 and 2 disclose technologies that take fixational eye movement into consideration. Patent Document 1 discloses a technology that reduces changes in gaze information to a greater extent when the focal length for capturing the image viewed by the user is short than when the focal length is long. Patent Document 2 discloses a technology that expands an area corresponding to an index (an area that functions as an gaze switch) when the index is gazed at for a predetermined period of time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-183552 A [Patent Document 2] Japanese Patent Application Publication No. 7-283986 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the techniques disclosed in Patent Documents 1 and 2, even if a user looks at an object (a real object or a virtual object), the object may not perform the action intended by the user.

[0006] An object of the present invention is to provide a technique capable of processing gaze information so as to perform an action intended by a user. [Means for solving the problem]

[0007] A first aspect of the present invention is an information processing device characterized by having a first acquisition means for acquiring information indicating a first gaze position of a user, a second acquisition means for acquiring size information, which is information on the size of one or more objects present in the direction of the user's gaze based on the information indicating the first gaze position, as perceived by the user, and a third acquisition means for acquiring information indicating a second gaze position by reducing the change in the information indicating the first gaze position by a degree of reduction based on the size information.

[0008] A second aspect of the present invention is characterized in that the display device includes a first acquisition means for acquiring information indicating a first gaze position of a user in a displayed image, a second acquisition means for acquiring magnification information which is information on a display magnification of the image, and a third acquisition means for acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a reduction degree based on the magnification information. The information processing device has the following characteristics.

[0009] A third aspect of the present invention is an information processing device having a control means for controlling the display of an indicator indicating a user's gaze position, and characterized in that when the user is looking at an object of a first size, the fluctuation in the position of the indicator is smaller than when the user is looking at an object of a second size that is larger than the first size.

[0010] A fourth aspect of the present invention is an information processing device having a control means for controlling to display an indicator indicating a user's gaze position on a displayed image, and characterized in that when the display magnification of the image is a first magnification, the fluctuation in the position of the indicator is smaller than when the display magnification is a second magnification higher than the first magnification.

[0011] A fifth aspect of the present invention is an information processing method comprising the steps of: acquiring information indicating a first gaze position of a user; acquiring size information, which is information on the size of one or more objects present in the direction of the user's gaze based on the information indicating the first gaze position, as perceived by the user; and acquiring information indicating a second gaze position by reducing the change in the information indicating the first gaze position by a degree of reduction based on the size information.

[0012] A sixth aspect of the present invention is an information processing method comprising the steps of acquiring information indicating a first gaze position of a user in a displayed image, acquiring magnification information which is information on the display magnification of the image, and acquiring information indicating a second gaze position by reducing the change in the information indicating the first gaze position by a degree of reduction based on the magnification information.

[0013] A seventh aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device described above. An eighth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device described above. Effect of the Invention

[0014] According to the present invention, it is possible to process gaze information so as to perform an action intended by a user. [Brief description of the drawings]

[0015] [Figure 1] FIG. [Diagram 2] FIG. [Diagram 3] FIG. 2 is a block diagram of the camera. [Figure 4] FIG. 1 is an external view of an HMD. [Diagram 5] FIG. 2 is a block diagram of an HMD. [Figure 6] FIG. [Figure 7] FIG. 2 is a diagram showing an optical system for detecting a line of sight. [Figure 8] FIG. 2 is a diagram showing an eye image and its luminance distribution. [Figure 9] 13 is a flowchart of a gaze detection process. [Figure 10] 13 is a flowchart of a line-of-sight change reduction process. [Figure 11] 13 is a flowchart of an allowable range determination process. [Figure 12] FIG. [Figure 13] 13 is a flowchart of a reduction degree determination process. [Figure 14] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <<Embodiment 1>> Hereinafter, a first embodiment of the present invention will be described with reference to the accompanying drawings.

[0017] <Configuration Description> 1(A) and 1(B) show the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to the first embodiment. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a rear perspective view. As shown in FIG. 1(A), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operation member that accepts a photographing operation from a user (photographer), is arranged on the camera housing 1B. As shown in FIG. 1(B), an eyepiece lens 12 (eyepiece optical system) is arranged on the back of the camera housing 1B, through which the user peers into a display device 10 (display panel) (described later) contained in the camera housing 1B. The eyepiece optical system may include a plurality of lenses. On the back of the camera housing 1B, operation members 41 to 43 that accept various operations from the user are also arranged. For example, the operation member 41 is a touch panel that accepts touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-way key that can be pressed in each of four directions. The operation member 41 (touch panel) includes a display panel such as a liquid crystal panel, and has a function of displaying images on the display panel.

[0018] FIG. 2 is a cross-sectional view of the camera 1 taken along the YZ plane defined by the Y axis and Z axis shown in FIG.

[0019] The photographing lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture drive section 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. For simplicity, two lenses 101 and 102 are shown, but in reality, more than two lenses are included in the photographing lens unit 1A. The lens drive member 114 includes a drive gear, and the photocoupler 115 detects the rotation of the pulse plate 116 that is interlocked with the lens drive member 114 and transmits it to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (information on the lens drive amount), and moves the lens 101 to change the focus position. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B.

[0020] The photographing lens unit 1A also includes an acceleration sensor 119, a zoom driving section 120, and a lens CPU 121. The acceleration sensor 119 detects the acceleration of the camera 1 and is used to detect the movement of the camera 1, such as panning. The zoom driving section 120 drives one or both of the lenses 101 and 102 so as to change the width of the angle of view captured in response to a zoom operation. The zoom operation may involve turning an operation member provided on a lens barrel (not shown), or may involve operation on an operation member of the camera housing 1B. When an operation on an operation member of the camera housing 1B is performed as a zoom operation, a signal corresponding to the zoom operation is sent from the camera housing 1B to the photographing lens unit 1A, and the zoom driving section 120 drives the lenses 101 and 102 in response to the signal. The lens CPU 121 is a central processing section of a microcomputer, and controls the entire photographing lens unit 1A.

[0021] The camera housing 1B contains an image sensor 2, a camera CPU 3, a memory section 4, an acceleration sensor 6, a display device 10, and a display device drive circuit 11. The image sensor 2 is disposed on a planned imaging surface of the photographing lens unit 1A. The camera CPU 3 is a central processing section of a microcomputer, and controls the entire camera 1. The memory section 4 stores the information stored in the memory section 4 by the image sensor 2. The acceleration sensor 6 detects the acceleration of the camera 1 and is used to detect the movement of the camera 1, such as panning. For the detection of panning, either the acceleration sensor 119 in the photographing lens unit 1A or the acceleration sensor 6 in the camera housing 1B or both may be used. The display device 10 includes a display element such as a liquid crystal element, and displays various images such as captured images (subject images) on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. A user can view an image (for example, an image of real space captured by the image capture element 2) displayed on the display surface of the display device 10 through the eyepiece 12.

[0022] The camera housing 1B also includes light sources 13a and 13b, a light splitter 15, a light receiving lens 16, and an eye imaging element 17. The light sources 13a and 13b are light sources for illuminating the user's eyeball 14. The light sources 13a and 13b have been used in devices such as single-lens reflex cameras to detect the line of sight (the direction of the line of sight; the direction in which the user is looking) from the relationship between the reflected image (corneal reflection image; Purkinje image) of light reflected by the cornea and the pupil. For example, the light sources 13a and 13b are infrared light-emitting diodes that emit infrared light that is insensitive to the user, and are arranged around the eyepiece 12. The optical image of the illuminated eyeball 14 (eyeball image; an image formed by the light emitted from the light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the light splitter 15. The eyeball image is then formed by light receiving lens 16 on eye imaging element 17, which is a two-dimensional array of photoelectric elements such as CCD or CMOS. Light receiving lens 16 positions the pupil of eyeball 14 and eye imaging element 17 in a conjugate imaging relationship. The line of sight direction of eyeball 14 is detected from the positional relationship between the corneal reflection image in the eyeball image formed on eye imaging element 17 and the pupil, using a predetermined algorithm described later.

[0023] FIG. 3 is a block diagram showing the electrical configuration within the camera 1. As shown in FIG.

[0024] Connected to the camera CPU3 are a digital interface circuit 201, a photometry circuit 202, an auto focus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, and a light source drive circuit 205. Also connected to the camera CPU3 are an acceleration sensor 6, a memory unit 4, an image processing unit 208, and an attitude detection unit 207. Operation members 41-43 are also connected to the camera CPU3, and when the operation members 41-43 are operated, operation signals corresponding to the operation are sent from the operation members 41-43 to the camera CPU3, and the camera CPU3 performs control corresponding to the operation signals.

[0025] In addition, the camera CPU 3 sends a signal to the lens CPU 121 via the mount contact 117. In response to the signal from the camera CPU 3, the lens CPU 121 controls the focus adjustment circuit 118, the aperture control circuit 206, the acceleration sensor 119, and the zoom drive unit 120, which are arranged in the photographing lens unit 1A. The aperture control circuit 206 is included in the aperture drive unit 112.

[0026] The memory unit 4 has a function of storing image pickup signals from the image pickup element 2 and the eye image pickup element 17. The memory unit 4 is connected to a recording medium 50, and the camera CPU 3 converts the image pickup signal of the image pickup element 2 stored in the memory unit 4 into a recording signal and sends the recording signal to the recording medium 50.

[0027] The attitude detection unit 207 detects attitude information of the camera 1 held by the user. For example, the attitude detection unit 207 detects a change in attitude between a normal position in which the image sensor 2 is horizontal to the subject and a vertical position in which the image sensor 2 is vertical to the subject.

[0028] The digital interface circuit 201 performs A / D conversion on the output of the eye image pickup element 17 (CCD-EYE) when an eyeball image is formed on the eye image pickup element 17 (eye image obtained by picking up an image of the eye), and The result of the above is transmitted to the camera CPU 3. The camera CPU 3 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm described later, and calculates the user's viewpoint (gaze position; position where the gaze is directed; position where the user is looking) on ​​the display surface of the display device 10 from the positions of the feature points.

[0029] The photometry circuit 202 processes the signal obtained from the image sensor 2, which also functions as a photometry sensor, specifically the luminance signal corresponding to the brightness of the field, such as amplification, logarithmic compression, and A / D conversion, and sends the result as field luminance information to the camera CPU 3. Furthermore, the photometry circuit 202 detects the area of ​​a specified subject based on the field luminance information, and sends the result to the camera CPU 3 as subject area information.

[0030] The auto focus detection circuit 203 A / D converts signal voltages from a plurality of detection elements (a plurality of pixels) used for phase difference detection included in the image sensor 2, and sends the converted signal to the camera CPU 3. The camera CPU 3 calculates the distance (amount of focus deviation) to the subject corresponding to each focus detection point from the signals of the plurality of detection elements. This is a well-known technique known as image plane phase difference AF (autofocus using phase difference signals). In the first embodiment, as an example, it is assumed that there are focus detection points in the field of view within the viewfinder (the field of view when looking through the viewfinder), specifically, at each of 180 points on the image plane corresponding to the 180 points on the display surface of the display device 10. The camera CPU 3 can grasp the amount of focus deviation for each of the 180 focus detection points.

[0031] Switches SW1 and SW2 are connected to the signal input circuit 204. Switch SW1 is a switch that is turned ON by the first stroke of the release button 5, and is used to start various processes (operations) of the camera 1, such as photometry, distance measurement, focus detection processing, and gaze detection processing. Switch SW2 is a switch that is turned ON by the second stroke of the release button 5, and is used to start a photographing operation. ON signals from switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the camera CPU 3.

[0032] The light source drive circuit 205 drives the light sources 13a and 13b.

[0033] The image processing unit 208 acquires various data (information) by performing various processes based on image data obtained by the imaging element 2. The image processing unit 208 may or may not be a dedicated hardware circuit such as an ASIC designed to realize a specific function. The function of the image processing unit 208 may be realized by a processor such as a DSP executing software.

[0034] The processes performed by the image processing unit 208 include, for example, pre-processing, color interpolation, correction, detection, data processing, and evaluation value calculation. The pre-processing includes signal amplification, reference level adjustment, and defective pixel correction. The color interpolation process is a process for obtaining values ​​of color components not included in the image data, and is also called demosaic processing. The correction process includes white balance adjustment, a process for correcting the luminance of the image, a process for correcting the optical aberration of the photographing lens unit 1A, and a process for correcting the color of the image. The detection process includes a process for detecting and tracking a characteristic region (e.g., a face region, a human body region, an animal region, or an automobile region), and a process for recognizing a person. The image processing unit 208 can also identify the subject (real object) that the user is looking at based on the result of the gaze detection process. The image processing unit 208 can also obtain motion characteristic information indicating the movement of the subject by analyzing the difference in image data along a time series. The data processing process includes a scaling process, an encoding process, a decoding process, and a header information generation process. The evaluation value calculation process includes a process of calculating an evaluation value used in AF (for example, image plane phase difference AF or contrast AF) and a process of calculating an evaluation value used in automatic exposure control. Note that these are examples of processes that can be performed by the image processing unit 208. There is no limitation on the processing performed by the processing unit 208. Furthermore, a part or all of the processing performed by the image processing unit 208 may be performed by the camera CPU 3.

[0035] FIG. 4 is a front perspective view showing the appearance of a head mounted display (HMD) 400 according to the first embodiment. When using the HMD 400, the user wears the HMD 400 on the head. The HMD 400 has various operation members such as a power switch and various setting buttons. The HMD 400 also has an image processing unit that performs various processes such as generating virtual objects and synthesizing images. The head mounting member 401 is a member for stably fixing the HMD 400 to the user's head. The head mounting member 401 may be a member that fastens the user's head, a member that is hung on the user's ears, or a member that fixes the HMD 400 to the user's head in another manner.

[0036] The HMD400 is a video see-through HMD that presents an image of real space to the user. The HMD400 can also present an image of a virtual object to the user. For example, the HMD400 has a liquid crystal panel arranged in front of the eyes of a user wearing the HMD400, a driver circuit that controls the liquid crystal panel, and a memory that holds an image to be displayed, and displays an image on the liquid crystal panel. An image may be directly projected onto the retina of the user using a laser. The HMD400 may present an image of a virtual space to the user. The HMD400 may be an optical see-through HMD that allows the user to view a real space other than an image. The virtual object may be placed in real space or in virtual space.

[0037] The HMD 400 includes an imaging unit 500. The imaging unit 500 is a device (camera unit) that captures an image of real space (the environment surrounding the user). For example, the imaging unit 500 is disposed on the front side of the HMD 400, and captures an image in front of (in front of) the HMD 400 (user).

[0038] 5 is a block diagram showing the configuration of the HMD 400. The HMD 400 has an imaging unit 500, a line of sight detection unit 501, a movement detection unit 502, a rotation detection unit 503, an information processing unit 504, an image processing unit 505, and a display unit 506. The information processing unit 504 has a captured image acquisition unit 507, a virtual object holding unit 508, a virtual object generation unit 509, a line of sight information acquisition unit 510, an HMD movement amount calculation unit 511, an HMD rotation amount calculation unit 512, and an HMD movement determination unit 513. The image processing unit 505 has a virtual object selection unit 514, a virtual object control unit 515, and a display image generation unit 516.

[0039] The imaging unit 500 has an optical system, an image sensor, a driver circuit for controlling the image sensor, an A / D conversion circuit for performing A / D conversion of a signal obtained by the image sensor, and a development circuit for developing the signal obtained by the A / D conversion circuit as an image, and captures a real space. Image data (image of the real space) captured by the imaging unit 500 is acquired by a captured image acquisition unit 507 of the information processing unit 504 and transmitted to a display image generation unit 516 of the image processing unit 505. The display image generation unit 516 performs a synthesis process for synthesizing a virtual object with the image of the real space, and generates a display image. In addition, the captured image acquisition unit 507 sends the image of the real space to a virtual object generation unit 509, and causes the virtual object generation unit 509 to generate a virtual object.

[0040] The gaze detection unit 501 is a device for detecting the gaze direction of the user, and for example, it detects the gaze direction from the relationship between the corneal reflection image and the pupil by irradiating the user's eyeball with infrared light using an infrared light emitting diode. Gaze information (gaze detection result; information about the gaze) obtained by the gaze detection unit 501 is acquired by the gaze information acquisition unit 510 of the information processing unit 504 and transmitted to the virtual object control unit 515 of the image processing unit 505. The virtual object control unit 515 The virtual object control unit 15 controls the virtual object and transmits the virtual object to the display image generation unit 516 of the image processing unit 505. As described above, the display image generation unit 516 performs a synthesis process for synthesizing the virtual object with an image in real space, and generates a display image.

[0041] The movement detection unit 502 is a device for detecting the movement of the HMD 400. For example, the movement detection unit 502 may detect the movement of the HMD 400 using position information from a global positioning system (GPS), or may detect the movement of the HMD 400 using an acceleration sensor. Movement information (movement detection result) obtained by the movement detection unit 502 is sent to an HMD movement amount calculation unit 511 of the information processing unit 504. The HMD movement amount calculation unit 511 calculates the movement amount of the HMD 400 based on the movement information obtained by the movement detection unit 502, and sends it to an HMD motion determination unit 513.

[0042] The rotation detection unit 503 is a device for detecting the rotation of the HMD 400. For example, the rotation detection unit 503 may detect the rotation of the HMD 400 using a rotation displacement sensor. Rotation information (rotation detection result) obtained by the rotation detection unit 503 is sent to an HMD rotation amount calculation unit 512 of the information processing unit 504. The HMD rotation amount calculation unit 512 calculates the amount of rotation of the HMD 400 based on the rotation information obtained by the rotation detection unit 503, and sends it to an HMD movement determination unit 513.

[0043] The HMD motion determination unit 513 determines the motion of the HMD 400 based on the amount of movement (direction and magnitude of movement) of the HMD 400 sent from the HMD movement amount calculation unit 511 and the amount of rotation (direction and magnitude of rotation) of the HMD 400 sent from the HMD rotation amount calculation unit 512. The motion information (motion determination result) obtained by the HMD motion determination unit 513 is sent to a virtual object control unit 515 of the image processing unit 505 and is used to control the virtual object.

[0044] Virtual object holding unit 508 holds data of a virtual object (information on shape, position, and orientation) and data of a virtual light source that irradiates light onto the virtual object. When a virtual object (data of a virtual object) held by virtual object holding unit 508 is selected by virtual object selection unit 514, the selected virtual object is sent to virtual object control unit 515. In addition, virtual object holding unit 508 sends a virtual object to virtual object generation unit 509, and causes virtual object generation unit 509 to generate another virtual object.

[0045] The virtual object generation unit 509 generates a virtual object based on the image (image in real space) sent from the captured image acquisition unit 507 and the virtual object sent from the virtual object holding unit 508. When the virtual object generated by the virtual object generation unit 509 is selected by the virtual object selection unit 514, the selected virtual object is sent to the virtual object control unit 515.

[0046] The virtual object selection unit 514 selects a virtual object to be displayed from a plurality of virtual objects including virtual objects held by the virtual object holding unit 508 and virtual objects generated by the virtual object generation unit 509. The virtual object to be displayed is a virtual object to be provided (displayed) to the user. The virtual object selection unit 514 then sends the virtual object to be displayed to the virtual object control unit 515. For example, image data of the virtual object, position data of the virtual object, and order data indicating the overlap order of the plurality of virtual objects are sent to the virtual object control unit 515 as data of the virtual object to be displayed.

[0047] The virtual object control unit 515 selects the virtual object sent from the virtual object selection unit 514 based on the line-of-sight information sent from the line-of-sight information acquisition unit 510 and the HMD movement determination unit 513. The image is then sent to the display image generating unit 516, based on the motion information sent from the image generating unit 516.

[0048] The display image generation unit 516 performs a synthesis process of synthesizing the image (image in real space) sent from the captured image acquisition unit 507 with the virtual object sent from the virtual object control unit 515, thereby generating a display image. Then, the display image generation unit 516 sends the display image to the display unit 506. The display unit 506 displays the display image sent from the display image generation unit 516.

[0049] In addition to the above-mentioned processes, the image processing unit 505 may perform one or more of pre-processing, color interpolation, correction, detection, and data processing. The pre-processing includes signal amplification, reference level adjustment, and defective pixel correction. The color interpolation process is a process of obtaining values ​​of color components not included in the image data, and is also called demosaicing. The correction process includes white balance adjustment, a process of correcting the luminance of an image, a process of correcting the optical aberration of the imaging unit 500, and a process of correcting the color of an image. The detection process includes a process of detecting and tracking a feature region (e.g., a face region, a human body region, an animal region, or an automobile region), and a process of recognizing a person. The image processing unit 505 can also identify an object (a real object or a virtual object) that the user is looking at based on the line of sight information. The image processing unit 505 can also obtain motion characteristic information indicating the movement of an object by analyzing the difference in image data along a time series. The data processing process includes a scaling process, an encoding process, a decoding process, and a header information generation process. Note that these are merely examples of processes that the image processing unit 505 can perform, and do not limit the processes that the image processing unit 505 can perform.

[0050] The devices to which the present invention can be applied are not limited to cameras and HMDs. For example, the present invention can be applied to various information processing devices such as smartphones, tablet terminals, and personal computers. Hereinafter, the example of the camera 1 will be mainly described, and supplementary explanations of the example of the HMD 400 will be provided as appropriate.

[0051] FIG. 6 is a diagram showing the field of view in the finder when the display device 10 of the camera 1 is in operation (when an image is displayed). In the case of the HMD 400, FIG. 6 can also be interpreted as a diagram showing the display surface of the display unit 506 in the state in which the display unit 506 is in operation. As shown in FIG. 6, the field of view in the finder includes a field of view mask 300, and in the display area surrounded by the field of view mask 300, an image of real space captured by the imaging element 2 is displayed. Furthermore, items 6201 and 6203 are displayed. The items 6201 and 6203 are superimposed on the image of real space. The camera 1 can detect the line of sight of the user looking through the finder, and can detect the user's line of sight position in the field of view in the finder (display surface of the display device 10). In FIG. 6, the item 6201 is an item (indicator; icon) indicating the user's line of sight position. The camera 1 can also detect the subject area the user is looking at based on the user's line of sight position. The area 6202 is a face area detected as the subject area the user is looking at. The item 6203 is an item (indicator; icon) that indicates the face region 6202 .

[0052] <Description of gaze detection process> The gaze detection process (gaze detection method) will be described with reference to Figs. 7, 8(A), 8(B), and 9. Fig. 7 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for detecting the gaze. As shown in Fig. 7, light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light receiving lens 16, and illuminate user's eyeball 14. A part of the light emitted from light sources 13a and 13b and reflected by eyeball 14 is collected on eye imaging element 17 by light receiving lens 16. Fig. 8(A) is a schematic diagram of an eye image (optical image of the eye projected on eye imaging element 17) captured by eye imaging element 17, and Fig. 8(B) is a diagram showing the output intensity of eye imaging element 17. In the case of HMD 400, the optical system in Fig. 7 is included in imaging unit 500, and the eye image in Fig. 8(A) is obtained by imaging unit 500. Fig. 9 is a flowchart of the gaze detection process. The gaze detection in Fig. 9 In the case of the camera 1, the camera CPU 3 performs the gaze detection process, and in the case of the HMD 400, the gaze detection unit 501 performs the gaze detection process.

[0053] 9 starts, in step S1, the camera CPU 3 drives the light sources 13a and 13b using the light source drive circuit 205 so as to emit infrared light toward the user's eyeball 14. An optical image of the user's eye illuminated by the infrared light passes through the light receiving lens 16 and is formed on the eye imaging element 17, which then performs photoelectric conversion. This provides an electrical signal of the eye image that can be processed.

[0054] In step S2, the camera CPU 3 acquires an eye image (image data, image signal) from the eye imaging element 17 via the digital interface circuit 201.

[0055] In step S3, the camera CPU 3 detects the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S2.

[0056] The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 142 are collected by the light receiving lens 16 and focused on the eye imaging element 17 to become corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from the ends a and b of the pupil 141 are also focused on the eye imaging element 17 to become pupil end images a' and b' in the eye image.

[0057] FIG. 8(B) shows luminance information (luminance distribution) of the region α in the eye image of FIG. 8(A). In FIG. 8(B), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. The coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe, and the coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb. As shown in FIG. 8(B), an extremely high level of luminance is obtained at the coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from the coordinates Xa to the coordinates Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the eye imaging element 17), an extremely low level of luminance is obtained except for the coordinates Xd, Xe. A luminance intermediate between the above two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). For example, a luminance intermediate between the above two types of luminance is obtained in a region where the X coordinate (coordinate in the X-axis direction) is smaller than coordinate Xa and a region where the X coordinate is larger than coordinate Xb.

[0058] From the luminance distribution as shown in FIG. 8B, the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. For example, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of the corneal reflection images, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of the pupil edge images a' and b'. When the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by forming an image of the light beam from the pupil center c on the eye imaging element 17 can be expressed as Xc≒(Xa+Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinate of the pupil center image c' can be estimated.

[0059] In step S4, the camera CPU 3 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between the corneal reflection images Pd', Pe'.

[0060] In step S5, the camera CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16. This approximately coincides with the X coordinate of the curvature center O. For this reason, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, then the rotation angle θx of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following formula 1. The rotation angle θy of the eyeball 140 in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to that for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0061] In step S6, the camera CPU 3 reads out parameters necessary for gaze detection from the memory unit 4. For example, the camera CPU 3 reads out parameters m, Ax, Bx, Ay, and By. The parameter m is a constant determined by the configuration of the optical system for performing the gaze detection process, and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c on the display device 10. The parameter m is determined in advance and stored in the memory unit 122. The parameters Ax, Bx, Ay, and By are gaze correction parameters for correcting individual differences in the gaze, and are acquired by performing calibration of the gaze detection. The gaze correction parameters Ax, Bx, Ay, and By are stored in the memory unit 4 before the gaze detection process starts.

[0062] In step S7, the camera CPU 3 estimates the user's viewpoint (gaze position) on the display device 10 using the rotation angles θx, θy calculated in step S5 and the parameters m, Ax, Bx, Ay, and By read in step S6. If the viewpoint coordinates (Hx, Hy) correspond to the pupil center c, the viewpoint coordinates (Hx, Hy) can be calculated by the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0063] In step S8, the camera CPU 3 stores the coordinates (Hx, Hy) of the viewpoint and the detection time of the viewpoint in the memory unit 4, and ends the gaze detection process. The detection time of the viewpoint may be, for example, the calculation time of the coordinates (Hx, Hy) or the capture time of the eye image.

[0064] The gaze detection method is not limited to the above method, and may be any method that acquires gaze information from an eye image. The gaze may be detected by a method that does not use an eye image, such as a method that detects an electrooculography without using an eye image and detects the gaze based on the electrooculography. As the gaze information, information indicating the gaze direction (direction of the gaze) may be obtained instead of information indicating the viewpoint. For example, processing may be performed to obtain a rotation angle (Ax×θx+Bx or Ay×θy+By) without obtaining the coordinates (Hx, Hy) of the viewpoint.

[0065] <Description of line of sight change reduction processing> One of the causes of errors in gaze detection is physiological eye movement. For example, even when a person intends to look at one point, the human eye moves involuntarily and slightly. This type of involuntary movement is called fixational eye movement. Since changes in gaze caused by fixational eye movement are not intended by the user, fixational eye movement is a cause of errors in gaze detection. Therefore, in the first embodiment, a gaze change reduction process is performed to reduce changes in gaze information (detected gaze) caused by fixational eye movement.

[0066] However, changes in gaze information (detected gaze) due to fixational eye movement do not always need to be reduced to the same degree. In other words, gaze change reduction processing does not always need to be performed with the same processing strength. Fixational eye movement is not the only physiological eye movement in humans. When tracking a slow-moving object (real object or virtual object) with the eyes, humans perform eye movement (a type of voluntary movement) that moves the gaze smoothly. Also, When tracking a fast-moving object with the eyes, humans tend to perform jumping movements that change their gaze suddenly. When gaze change reduction processing is performed, changes in the detected gaze (gaze information) become slower compared to actual changes in the gaze (eye movement). In other words, the ability of the detected gaze to track an object becomes poor. Therefore, if the processing strength of the gaze change reduction processing is strong (if the degree of reduction in changes in gaze information is strong), the gaze information will no longer accurately represent voluntary movements, jumping movements, and the like.

[0067] On the other hand, if the processing strength of the line-of-sight change reduction process is weak (if the degree of reduction in the change in line-of-sight information is weak), the line-of-sight information is not stable, and convenience is reduced. For example, when selecting an object with the line of sight, it becomes difficult to focus the line of sight on the object, making it difficult to select the object. In particular, when the object is small, it becomes difficult to select the object. Furthermore, when multiple objects are crowded together, a different object may be selected.

[0068] Furthermore, since the degree of fixational eye movement depends on the user, it is necessary to change the processing strength of the gaze change reduction process (the degree to which changes in gaze information are reduced) according to the degree of variation in the user's gaze.

[0069] Therefore, in the first embodiment, the processing strength of the line-of-sight change reduction process (the degree of reduction in the change in line-of-sight information) is changed in consideration of the viewing situation of the object and the degree of fixational eye movement.

[0070] Fig. 10 is a flowchart of the line-of-sight change reduction process. The line-of-sight change reduction process in Fig. 10 is assumed to start, for example, when the user looks into the finder of camera 1. In the case of HMD 400, it is assumed to start when the user wears HMD 400. The line-of-sight change reduction process in Fig. 10 is performed by camera CPU 3 in the case of camera 1, and line-of-sight detection process is performed by line-of-sight information acquisition unit 510 in the case of HMD 400.

[0071] In step S101, the camera CPU 3 acquires information indicating the user's gaze position by the gaze detection process of Fig. 9. The gaze position acquired in this step may be hereinafter referred to as the first gaze position. In the case of the HMD 400, the gaze information acquisition unit 510 acquires the first gaze position by having the gaze detection unit 501 perform the gaze detection process of Fig. 9. Note that the information indicating the gaze position is not limited to the coordinate position where the user is looking, and may be the direction in which the user is looking.

[0072] In step S102, the camera CPU 3 calculates the center and variation of the multiple first gaze positions corresponding to the multiple first gaze positions, respectively, based on the multiple first gaze positions acquired at different times up to the present. For example, the average value or median value of the coordinates (Hx, Hy) of the multiple gaze positions is calculated as the coordinate of the center of the multiple gaze positions, and the variance, standard deviation, or other statistical index of the coordinates (Hx, Hy) of the multiple gaze positions is calculated as the variation. The calculated variation may be the variation of the gaze, and the variation of the gaze direction may be calculated instead of the variation of the gaze position itself (coordinates). The acquisition period (first period) of the multiple first gaze positions used to calculate the center of the first gaze position and the acquisition period (second period) of the multiple first gaze positions used to calculate the variation of the first gaze position may be the same or different.

[0073] In step S103, the camera CPU 3 performs an allowable range determination process to determine an allowable range of variation in gaze position. For example, the size of the allowable range corresponds to the size of the range of gaze positions in which it is determined that the user is looking at an object (real object or virtual object). When an item showing the object the user is looking at is displayed, the size of the allowable range may correspond to the size of the range of gaze positions in which the item is displayed. In the case of selecting an object that a user is looking at, the size of the allowable range may correspond to the size of the range of gaze positions for selecting the object. The allowable range determination process will be described in detail with reference to FIG.

[0074] In step S104, the camera CPU 3 performs a reduction degree determination process to determine a reduction degree (reduction rate) of the change in the first gaze position. By reducing the change between the first gaze position acquired in step S101 of the previous time or earlier and the first gaze position acquired in step S101 of the current time, the change in the gaze position after the gaze change reduction process becomes insensitive to the change in the actual gaze position. Therefore, the reduction degree determination process can also be interpreted as a process to determine the sensitivity (or insensitivity) of the change in the gaze position after the gaze change reduction process to the change in the first gaze position, which is the actual gaze position value. The gaze position after the gaze change reduction process may be called the second gaze position. In the first embodiment, the reduction degree is determined based on the variation in the gaze position acquired in step S102 and the allowable range acquired in step S103. The reduction degree determination process will be described in detail with reference to Figs. 13(A) to 13(C).

[0075] In step S105, the camera CPU 3 acquires information indicating the second gaze position by changing the first gaze position so as to reduce the change in the first gaze position by the reduction degree obtained in step S104. The specific processing in step S105 is not particularly limited, and may be, for example, a low-pass filter processing that attenuates a component of a specific frequency band higher than a cutoff frequency in the change in gaze information. The processing in step S105 may be a first smoothing processing that determines gaze information representative of a plurality of gaze information acquired at different times up to the present (for example, average gaze information). The acquisition period (third acquisition period) of the plurality of gaze positions used in the first smoothing processing may be the same as the first period or the second period described above, or may be different. The processing in step S105 may be a second smoothing processing that excludes the change in gaze information when the change is smaller than a threshold value, or a third smoothing processing that excludes the change in gaze information in a time shorter than the threshold value. The processing in step S105 may be a combination of two or more of these multiple processings.

[0076] In step S106, the camera CPU 3 determines whether or not to stop the line-of-sight change reduction process. If it is determined to stop the line-of-sight change reduction process, the process ends, and if not, the process proceeds to step S101. For example, the line-of-sight change reduction process ends when the user takes his / her eye off the viewfinder of the camera 1. In the case of the HMD 400, the line-of-sight change reduction process ends when the user removes (flips up) the HMD 400.

[0077] FIG. 11 is a flowchart of the allowable range determination process performed in step S103 of FIG.

[0078] In step S201, the camera CPU 3 selects one or more objects that exist in the direction in which the user's gaze is directed based on the gaze information. Then, the camera CPU 3 acquires size information, which is information on the size of the selected one or more objects as perceived by the user. For example, in the case of the camera 1, an object near the center of the multiple gaze positions obtained in step S102 is selected from the objects detected by the image processing unit 208. In the case of the HMD 400, an object near the center of the multiple gaze positions is selected from the real objects detected by the image processing unit 505. In the case of the video see-through method, this real object is an object detected from an image capturing the real space as in the case of the camera 1. In addition, in the case of the optical see-through method, even if the object viewed by the user is an object in the real space, it is preferable to select the object viewed by the user by capturing an image of the real space. An object near the center of the multiple gaze positions may be selected from the virtual objects generated by the virtual object generating unit 509. Real object Either a real object or a virtual object may be selected, or both may be selected. The size information is, for example, information on a representative size such as an average size, a median size, a maximum size, or a minimum size of one or more selected objects.

[0079] In step S202, the camera CPU 3 determines the allowable range of the variation in the gaze position according to the size information acquired in step S201. For example, the camera CPU 3 determines whether the representative size indicated by the size information corresponds to size S, size M, or size L. Size S corresponds to a small size range, and corresponds to a case where the representative size is equal to or smaller than size threshold 1. Size M corresponds to a medium size range, and corresponds to a case where the representative size exceeds size threshold 1 and is equal to or smaller than size threshold 2. Size L corresponds to a large size range, and corresponds to a case where the representative size exceeds size threshold 2. When the representative size corresponds to size S, the camera CPU 3 determines the range from threshold 1 to threshold 2 as the allowable range, and when the representative size corresponds to size M, determines the range from threshold 2 to threshold 3 as the allowable range. Then, when the representative size corresponds to size L, the camera CPU 3 determines the range from threshold 3 to threshold 4 as the allowable range. The upper and lower limits of the allowable range are determined based on a statistical index (e.g., variance or standard deviation) of the variation in the gaze position. In this embodiment, the upper limit of the allowable range corresponding to size S and the lower limit of the allowable range corresponding to size M are equal (both are threshold 2), and the upper limit of the allowable range corresponding to size M and the lower limit of the allowable range corresponding to size L are equal (both are threshold 3). The method of determining the allowable range is not limited to this, and it is sufficient that the allowable range is determined so that, when the perceived size of the object indicated by the size information is a first size, a smaller allowable range is determined than when the perceived size is a second size larger than the first size. As described above, the allowable range may be determined by dividing the perceived size of the object into stages, or a determination method may be used in which a larger allowable range is determined as the perceived size increases. It is preferable that the allowable range monotonically increases as the perceived size increases.

[0080] A specific example of the process of step S201 will be described with reference to FIG. 12. FIG. 12 is a diagram showing an in-finder field of view 1201. An image of real space captured by the image sensor 2 is displayed in a display area 1202. Objects (real objects) 1204 to 1208 are displayed in the image of real space. Also, an item 1203 is displayed at the detected gaze position (a second gaze position which is the gaze position after the change has been reduced). Here, it is assumed that the display position of the item 1203 (the gaze position indicated by the item 1203) coincides with the center of the multiple gaze positions obtained in step S102.

[0081] In step S201, objects included in range 1209 whose distance from the position of item 1203 (the distance in the direction in which the user's field of vision widens) is shorter than a threshold value are selected. Only objects that are entirely included in range 1209 may be selected, or objects that are at least partially included in range 1209 may be selected. In the example of FIG. 12, it is assumed that objects 1204 to 1206 are determined to be included in range 1209, and objects 1207 and 1208 are determined not to be included in range 1209.

[0082] Next, information on the representative size of the objects included in the range 1209 is acquired as size information. The representative size is, for example, an average size, a median size, a maximum size, or a minimum size. In the example of FIG. 12, information on the average size, median size, maximum size, or minimum size of the objects 1204 to 1206 is acquired as size information. Even if the object recognized by the user is an object in real space, such as an optical see-through type HMD, the representative size of the object recognized by the user is acquired by photographing the real space, etc.

[0083] Fig. 13(A) is a flowchart of the reduction degree determination process performed in step S104 of Fig. 10. The line-of-sight change reduction process for changing line-of-sight information by performing the reduction degree determination process in Fig. 13(A) can also be interpreted as a process for changing line-of-sight information based on the size information obtained in step S201. Specifically, the line-of-sight change reduction process can also be interpreted as a process for changing line-of-sight information based on a comparison result between the line-of-sight position variation obtained in step S102 and the allowable range obtained in step S202.

[0084] In step S3001, the camera CPU 3 determines whether the variation in the gaze position obtained in step S102 is greater than the upper limit of the allowable range obtained in step S202. If the variation is greater than the upper limit, the process proceeds to step S3002, and if not, the process proceeds to step S3003.

[0085] In step S3002, the camera CPU 3 increases the degree of reduction of the change in the line of sight information (sets the degree of reduction to "strong"). By increasing the degree of reduction, the stability of the line of sight information is improved, and small objects can be easily selected.

[0086] In step S3003, the camera CPU 3 determines whether the variation in the gaze position obtained in step S102 is smaller than the lower limit of the allowable range obtained in step S202. If the variation is smaller than the lower limit, the process proceeds to step S3004, and if not, the process ends (without changing the reduction degree of the change in the gaze information).

[0087] In step S3004, the camera CPU 3 weakens the degree of reduction of changes in the line of sight information (sets the degree of reduction to "weak"). By weakening the degree of reduction, the line of sight information changes sensitively to changes in the actual line of sight, and the selected position can be changed quickly. For example, it becomes possible to quickly switch the selected part between multiple parts of a large object.

[0088] It is assumed that the process of step S105 (process of reducing the change in gaze information) is a low-pass filter process that attenuates components of a specific frequency band higher than a cutoff frequency in the change in gaze information. In this case, the degree of reduction of the change in gaze information can be strengthened by lowering the cutoff frequency, and the degree of reduction can be weakened by raising the cutoff frequency. It is assumed that the process of step S105 is a first smoothing process that determines gaze information (for example, average gaze information) that represents a plurality of gaze information acquired at different times up to the present. In this case, the degree of reduction of the change in gaze information can be strengthened by increasing the number of a plurality of gaze information used, and the degree of reduction can be weakened by decreasing the number of a plurality of gaze information used. It is assumed that the process of step S105 is a second smoothing process that excludes the change in gaze information if the change is smaller than a threshold value. In this case, the degree of reduction of the change in gaze information can be strengthened by increasing the threshold value, and the degree of reduction can be weakened by decreasing the threshold value. It is assumed that the process of step S105 is a third smoothing process that excludes the change in gaze information in a time shorter than the threshold value. In this case, by increasing the threshold value, the degree of reduction in the change in the line of sight information can be increased, and by decreasing the threshold value, the degree of reduction can be decreased.

[0089] In this embodiment, the default degree of reduction is set to "medium" between "strong" and "weak", but the number of stages of the degree of reduction may be more than three.

[0090] FIG. 13B is a flowchart of a first modified example of the reduction degree determination process performed in step S104 of FIG. 10. It is assumed that the camera CPU 3 acquires movement information, which is information on the movement of an object, from the image processing unit 208. In the case of the HMD 400, it is assumed that the line-of-sight information acquisition unit 510 acquires the movement information from the image processing unit 505. The reduction degree determination process of FIG. 13B The line-of-sight change reduction process for changing the line-of-sight information by performing a setting process can also be interpreted as a process for changing the line-of-sight information based on the size information and movement information obtained in step S201. Specifically, the line-of-sight change reduction process can also be interpreted as a process for changing the line-of-sight information based on the result of comparing the line-of-sight position variation obtained in step S102 with the allowable range obtained in step S202, and the movement information.

[0091] Steps S3101 to S3103 are the same as steps S3001 to S3003 in Fig. 13(A), respectively. If the variation in the gaze position obtained in step S102 is smaller than the lower limit, the process proceeds to step S3104.

[0092] In step S3104, the camera CPU 3 acquires movement information, which is information on the movement of one or more objects selected in step S201, and determines whether the one or more objects are moving based on the movement information. For example, the camera CPU 3 determines whether a representative movement amount (movement amount per predetermined time; movement speed) of the selected one or more objects is greater than a threshold. The representative movement amount is, for example, an average movement amount, a median movement amount, a maximum movement amount, or a minimum movement amount. If the representative movement amount is greater than the threshold, the object is considered to be moving, and if the representative movement amount is equal to or less than the threshold, the object is considered to be stationary. As the movement of the object, a movement in the in-plane direction of the display area 1202 (the direction in which the user's field of view widens) or a movement in a three-dimensional direction including the depth direction may be detected.

[0093] If one or more objects selected in step S201 are moving, proceed to step S3105. In step S3105, camera CPU 3 weakens the degree of reduction in the change in line of sight information (sets the degree of reduction to "weak"). By weakening the degree of reduction only when one or more objects selected in step S201 are moving, it becomes possible to move line of sight items quickly in accordance with the movement of the line of sight to track a moving body (moving object). Line of sight items are items that indicate the line of sight position.

[0094] If one or more objects selected in step S201 are not moving (are stationary), the reduction degree determination process is terminated (without changing the reduction degree of the change in the line of sight information). Since it may not be desirable for the line of sight item to move quickly relative to a stationary body (a stationary object), the reduction degree is not changed.

[0095] The threshold value for determining whether an object is moving may be changed depending on the shooting conditions or other settings. For example, when the AF mode is a mode (servo mode) for tracking a subject (real object), a value larger than that in the mode (OneShot mode) for shooting a stationary object may be used as the threshold value.

[0096] Fig. 13C is a flowchart of a second modified example of the reduction degree determination process performed in step S104 of Fig. 10. In the reduction degree determination process of Fig. 13C, the duration of the state of variation in gaze position is measured (counted).

[0097] In step S3201, the camera CPU 3 determines whether the variation in the gaze position obtained in step S102 is greater than the upper limit of the allowable range obtained in step S202. If the variation is greater than the upper limit, the process proceeds to step S3202; otherwise, the process proceeds to step S3205.

[0098] In step S3202, the camera CPU 3 judges whether or not the count value n1 corresponding to the duration of the state in which the variation in the gaze position is larger than the upper limit is larger than the threshold value N1. This judgment is made based on whether or not a first time has elapsed in the state in which the variation in the gaze position is larger than the upper limit. If the count value n1 is greater than the threshold value N1, the process proceeds to step S3203, otherwise the process proceeds to step S3204. The threshold value N1 is a positive integer equal to or greater than 0.

[0099] In step S3203, the camera CPU 3 increases the degree of reduction in the change in the line of sight information (sets the degree of reduction to "strong").

[0100] In step S3204, the camera CPU 3 increments the count value n1 by 1, and sets (initializes) a count value n2, which will be described later, to 0. Then, the reduction degree determination process ends.

[0101] In step S3205, camera CPU 3 determines whether the variation in gaze position obtained in step S102 is smaller than the lower limit of the allowable range obtained in step S202. If the variation is smaller than the lower limit, the process proceeds to step S3206; if not, the process ends (without changing the degree of reduction in the change in gaze information).

[0102] In step S3206, camera CPU 3 determines whether count value n2, which corresponds to the duration of the state in which the variation in gaze position is smaller than the lower limit, is greater than threshold value N2. This determination can also be interpreted as determining whether a second time has elapsed in a state in which the variation in gaze position is smaller than the lower limit. If count value n2 is greater than threshold value N2, proceed to step S3207; otherwise, proceed to step S3208. Threshold value N2 is a positive integer equal to or greater than 0.

[0103] In step S3207, the camera CPU 3 weakens the degree of reduction in the change in the line of sight information (sets the degree of reduction to "weak").

[0104] In step S3208, the camera CPU 3 increments the count value n2 by 1, and sets (initializes) a count value n1, which will be described later, to 0. Then, the reduction degree determination process ends.

[0105] In the reduction degree decision process of FIG. 13(C), frequent changes in the reduction degree of the change in the gaze information can be suppressed by taking into consideration the duration of the state of variation in the gaze position.

[0106] The thresholds N1 and N2 for the duration of the state of variation in gaze position may be the same or different values. Furthermore, each of the thresholds N1 and N2 may be switched according to the shooting conditions or other settings.

[0107] As described above, according to the first embodiment, the degree of reduction in the change in the gaze information is changed based on the size of an object present in the direction in which the gaze is directed (the size perceived by the user). This makes it possible to process the gaze information so that the user's intended action (for example, moving a gaze item or selecting an object) is performed, and it is possible to achieve both stability and agility in the gaze information. In FIG. 13, an example is described in which the result of comparing the variation in the gaze position with the allowable range is used as a method for determining the degree of reduction, but the method for determining the degree of reduction is not limited to this. For example, the degree of reduction may be determined based on the allowable range without using the variation in the gaze position, or the perceived size (representative size) of the object itself may be used instead of the allowable range. In the first embodiment, it is sufficient to make the variation in the second gaze position smaller when the user is looking at an object of a first size than when the user is looking at an object of a second size larger than the first size. As a result, as shown in FIG. 12, the item 1203 indicating the user's gaze position is displayed. When displaying the item 1203, the fluctuation in the display position of the item 1203 can be made smaller when the user is viewing a small object than when the user is viewing a large object.

[0108] <<Embodiment 2>> A second embodiment of the present invention will be described below. Note that, in the following, a description of the same points (such as configuration and processing) as in the first embodiment will be omitted, and differences from the first embodiment will be described. As the second embodiment, an example in which the reduction degree is changed according to the display magnification will be described. The display magnification can be set (switched) from a predetermined menu screen, for example. By changing the display magnification, as shown in FIG. 14, an image (display image) displayed in the display area in the field of view within the finder (display surface of the display device 10) is enlarged or reduced. In the case of the HMD 400, an image (display image) displayed on the display surface of the display unit 506 is enlarged or reduced.

[0109] In the second embodiment, the display image is interpreted as the object described in the first embodiment, and the line of sight change reduction process of Fig. 10 is performed. For example, in step S201 of Fig. 11, information on the size of the display image is acquired as size information. This size information can also be interpreted as information on the display magnification of the image (magnification information). Since the movement of the display image and the change in the display magnification are not frequently performed, the process of Fig. 13(A) can be more preferably performed as the reduction degree determination process than the processes of Figs. 13(B) and 13(C).

[0110] As described above, according to the second embodiment, the degree of reduction of the change in the gaze information is changed based on the display magnification of the image (the size of the displayed image). In this way, the gaze information can be processed so that the user's intended action (for example, the movement of the gaze item or the selection of an object) is performed, and both stability and agility of the gaze information can be achieved. As in the first embodiment, the method of determining the degree of reduction is not limited to the example using the comparison result between the variation in the gaze position and the allowable range. For example, the degree of reduction may be determined based on the allowable range without using the variation in the gaze position, or the magnitude of the display magnification itself may be used without using the allowable range. In the second embodiment, when the display magnification of the image is the first magnification, it is sufficient to make the fluctuation of the second gaze position smaller than when the display magnification is the second magnification higher than the first magnification. As a result, when the item 1203 indicating the user's gaze position is displayed as shown in FIG. 12, when the display magnification is small, the fluctuation of the display position of the item 1203 can be made smaller than when the display magnification is large.

[0111] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0112] For example, although the present invention has been described as being applied to both a camera and an HMD, the information processing device to which the present invention can be applied is not limited to these. For example, the present invention can also be applied to information processing devices such as smartphones, tablet terminals, personal computers, or servers.

[0113] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0114] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) A first acquisition means for acquiring information indicating a first gaze position of a user; A second acquisition means for acquiring size information, which is information on the size of one or more objects present in a direction in which the user's gaze is directed, as perceived by the user, based on information indicating the first gaze position; a third acquisition means for acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a degree of reduction based on the size information; 13. An information processing device comprising: (Configuration 2) When the size of the object indicated by the size information is a first size, the degree of reduction is greater than when the size of the object indicated by the size information is a second size that is larger than the first size. 2. The information processing device according to configuration 1. (Configuration 3) The one or more objects are one or more objects whose distance from a center of the plurality of first gaze positions acquired in a first period up to the present in a direction in which the user's visual field extends is shorter than a threshold value. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The third acquisition means acquires the information indicating the second gaze position based on a variation in the plurality of first gaze positions acquired during a second period up to the present and an allowable range according to the size information. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) the third acquisition means acquires the information indicating the second gaze position by reducing a change in the information indicating the first gaze position by a reduction degree based on the variation and the tolerance range; When the variation is the same, the degree of reduction is smaller when the size of the object indicated by the size information is a second size that is larger than the first size than when the size of the object indicated by the size information is a first size. 5. The information processing device according to configuration 4. (Configuration 6) The size information is information about an average size, a maximum size, or a minimum size of the one or more objects. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The second acquisition means acquires the size information and movement information that is information on the movement of the one or more objects; The third acquisition means acquires the information indicating the second gaze position by reducing a change in the information indicating the first gaze position by the reduction degree based on the size information and the movement information. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The third acquisition means is When the variation of the plurality of first gaze positions acquired during a second period up to the present is greater than an upper limit of an allowable range according to the size information, the degree of reduction is increased; When the variation is smaller than the lower limit of the allowable range, the degree of reduction is weakened. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The third acquisition means is If the variation is smaller than the lower limit but the one or more objects are not moving, the reduction degree is not changed; When the variation is smaller than the lower limit and the one or more objects are moving, the degree of reduction is weakened. 9. The information processing device according to configuration 8. (Configuration 10) The third acquisition means is When a first time has elapsed in a state in which the variation in the plurality of first gaze positions acquired during a second period up to the present is greater than an upper limit of a tolerance range according to the size information, the degree of reduction is strengthened; When a second time has elapsed in a state in which the variation is smaller than the lower limit of the allowable range, the degree of reduction is weakened. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 11) The third acquisition means acquires the information indicating the second gaze position by attenuating a component of a specific frequency band in a change in the information indicating the first gaze position. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) The third acquisition means is performing a low-pass filter process for attenuating a component of a specific frequency band higher than a cutoff frequency in a change in the information indicating the first gaze position, thereby acquiring the information indicating the second gaze position; By lowering the cutoff frequency, the degree of reduction is strengthened, By increasing the cutoff frequency, the degree of reduction is weakened. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) The third acquisition means is performing a process of determining a gaze position that represents the plurality of first gaze positions acquired during a third period up to the present, thereby acquiring information indicating the second gaze position; Increasing the number of the plurality of first gaze positions to enhance the degree of reduction; The degree of reduction is weakened by reducing the number of the plurality of first gaze positions. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 14) The third acquisition means is performing a process of excluding the change in the information indicating the first gaze position when the change is smaller than a threshold value, thereby acquiring the information indicating the second gaze position; By increasing the threshold value, the degree of reduction is strengthened; By decreasing the threshold, the degree of reduction is weakened. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 15) The third acquisition means is performing a process of excluding a change in the information indicating the first gaze position in a time shorter than a threshold, thereby acquiring information indicating the second gaze position; By increasing the threshold value, the degree of reduction is increased; By shortening the threshold, the degree of reduction is weakened. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 16) The one or more objects may include a real object. 16. The information processing device according to any one of configurations 1 to 15. (Configuration 17) The one or more objects may include a virtual object. 17. The information processing device according to any one of configurations 1 to 16. (Configuration 18) a first acquisition means for acquiring information indicating a first gaze position of a user in a displayed image; A second acquisition means for acquiring magnification information which is information on a display magnification of the image; a third acquisition means for acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a reduction degree based on the magnification information; 13. An information processing device comprising: (Configuration 19) When the display magnification indicated by the magnification information is a first magnification, the degree of reduction is greater than when the display magnification indicated by the magnification information is a second magnification higher than the first magnification. 19. The information processing device according to configuration 18. (Configuration 20) A control means for controlling to display an indicator showing a user's line of sight, When the user is viewing an object of a first size, the indicator position varies less than when the user is viewing an object of a second size that is larger than the first size. 23. An information processing apparatus comprising: (Configuration 21) a control means for controlling to display an indicator showing a user's line of sight in a displayed image; When the display magnification of the image is a first magnification, the fluctuation in the position of the indicator is smaller than when the display magnification is a second magnification higher than the first magnification. 23. An information processing apparatus comprising: (Method 1) obtaining information indicative of a first gaze position of a user; acquiring size information, which is information on the size of one or more objects present in a direction in which the user's gaze is directed, as perceived by the user, based on information indicating the first gaze position; acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a degree of reduction based on the size information; 13. An information processing method comprising: (Method 2) obtaining information indicative of a first gaze position of a user on a displayed image; acquiring magnification information which is information on a display magnification of the image; acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a reduction degree based on the magnification information; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 21. (medium) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 21. [Explanation of symbols]

[0115] 1: Camera 3: Camera CPU 400:HMD 510: Gaze information acquisition unit

Claims

1. A first acquisition means for acquiring information indicating a first gaze position of a user; a second acquisition means for acquiring size information, which is information on the size of one or more objects present in a direction in which the user's gaze is directed, as perceived by the user, based on information indicating the first gaze position; a third acquisition means for acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a degree of reduction based on the size information; 13. An information processing device comprising:

2. When the size of the object indicated by the size information is a first size, the degree of reduction is greater than when the size of the object indicated by the size information is a second size that is larger than the first size.

2. The information processing apparatus according to claim 1,

3. The one or more objects are one or more objects whose distance from a center of the plurality of first gaze positions acquired in a first period up to the present in a direction in which the user's visual field extends is shorter than a threshold value.

2. The information processing apparatus according to claim 1,

4. The third acquisition means acquires the information indicating the second gaze position based on a variation in the plurality of first gaze positions acquired during a second period up to the present and an allowable range according to the size information.

2. The information processing apparatus according to claim 1,

5. the third acquisition means acquires the information indicating the second gaze position by reducing a change in the information indicating the first gaze position by a reduction degree based on the variation and the tolerance range; When the variation is the same, the degree of reduction is smaller when the size of the object indicated by the size information is a second size that is larger than the first size than when the size of the object indicated by the size information is a first size.

5. The information processing apparatus according to claim 4.

6. The size information is information about an average size, a maximum size, or a minimum size of the one or more objects.

2. The information processing apparatus according to claim 1,

7. The second acquisition means acquires the size information and movement information which is information on the movement of the one or more objects; The third acquisition means acquires the information indicating the second gaze position by reducing a change in the information indicating the first gaze position by the degree of reduction based on the size information and the movement information.

2. The information processing apparatus according to claim 1,

8. The third acquisition means is When a variation in the plurality of first gaze positions acquired during a second period up to the present is greater than an upper limit of an allowable range according to the size information, the degree of reduction is increased; When the variation is smaller than the lower limit of the allowable range, the degree of reduction is weakened.

2. The information processing apparatus according to claim 1,

9. The third acquisition means is If the variation is smaller than the lower limit but the one or more objects are not moving, the reduction degree is not changed; When the variation is smaller than the lower limit and the one or more objects are moving, the degree of reduction is weakened.

9. The information processing apparatus according to claim 8,

10. The third acquisition means is When a first time has elapsed in a state in which the variation in the plurality of first gaze positions acquired during a second period up to the present is greater than an upper limit of a tolerance range according to the size information, the degree of reduction is strengthened; When a second time has elapsed in a state in which the variation is smaller than the lower limit of the allowable range, the degree of reduction is weakened.

2. The information processing apparatus according to claim 1,

11. The third acquisition means acquires the information indicating the second gaze position by attenuating a component of a specific frequency band in a change in the information indicating the first gaze position.

2. The information processing apparatus according to claim 1,

12. The third acquisition means is performing a low-pass filter process for attenuating a component of a specific frequency band higher than a cutoff frequency in a change in the information indicating the first gaze position, thereby acquiring the information indicating the second gaze position; By lowering the cutoff frequency, the degree of reduction is strengthened, By increasing the cutoff frequency, the degree of reduction is weakened.

2. The information processing apparatus according to claim 1,

13. The third acquisition means is performing a process of determining a gaze position that represents the plurality of first gaze positions acquired during a third period up to the present, thereby acquiring information indicating the second gaze position; Increasing the number of the plurality of first gaze positions to enhance the degree of reduction; The degree of reduction is weakened by reducing the number of the plurality of first gaze positions.

2. The information processing apparatus according to claim 1,

14. The third acquisition means is performing a process of excluding a change in the information indicating the first gaze position when the change is smaller than a threshold value, thereby acquiring information indicating the second gaze position; By increasing the threshold value, the degree of reduction is strengthened; By decreasing the threshold value, the degree of reduction is weakened.

2. The information processing apparatus according to claim 1,

15. The third acquisition means is performing a process of excluding a change in the information indicating the first gaze position in a time period shorter than a threshold value, thereby acquiring the information indicating the second gaze position; By increasing the threshold value, the degree of reduction is increased; By shortening the threshold, the degree of reduction is weakened.

2. The information processing apparatus according to claim 1,

16. The one or more objects may include a real object.

2. The information processing apparatus according to claim 1,

17. The one or more objects may include a virtual object.

2. The information processing apparatus according to claim 1,

18. a first acquisition means for acquiring information indicating a first gaze position of a user in a displayed image; A second acquisition means for acquiring magnification information which is information on a display magnification of the image; a third acquisition means for acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a reduction degree based on the magnification information; 13. An information processing device comprising:

19. When the display magnification indicated by the magnification information is a first magnification, the degree of reduction is greater than when the display magnification indicated by the magnification information is a second magnification higher than the first magnification.

20. The information processing apparatus according to claim 18,

20. A control means for controlling to display an indicator showing a user's line of sight, When the user is viewing an object of a first size, the indicator varies less in position than when the user is viewing an object of a second size that is larger than the first size.

23. An information processing apparatus comprising:

21. a control means for controlling to display an indicator showing a user's line of sight in a displayed image; When the display magnification of the image is a first magnification, the fluctuation of the position of the indicator is smaller than when the display magnification is a second magnification higher than the first magnification.

23. An information processing apparatus comprising:

22. acquiring information indicative of a first gaze position of a user; acquiring size information, which is information on the size of one or more objects present in a direction in which the user's gaze is directed, as perceived by the user, based on information indicating the first gaze position; acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a degree of reduction based on the size information; 13. An information processing method comprising:

23. obtaining information indicative of a first gaze position of a user in a displayed image; acquiring magnification information which is information on a display magnification of the image; acquiring information indicating a second gaze position by reducing a change in the information indicating the first gaze position by a reduction degree based on the magnification information; 13. An information processing method comprising:

24. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 21.

25. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 21.

Images