Display device, viewfinder device, and imaging device

Abstract

To provide technique that may easily capture eyes of a user while suppressing large size or cost increase of a device.SOLUTION: A display device according to the present invention comprises: a display panel; a display optical system for visually recognizing the display panel; an imaging sensor that has a rectangular imaging surface for imaging eyes of a user visually recognizing the display panel; and an imaging optical system for forming an optical image of the eyes on the imaging surface, wherein an optical axis of the imaging optical system is not parallel to an optical axis of the display optical system and a straight line formed by projecting the optical axis of the display optical system to the imaging surface is almost parallel to a long side of the imaging surface.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a display device, a viewfinder device, and an imaging device. [Background technology]

[0002] A camera has been proposed that includes a configuration for an eye-gaze input function (such as an eyeball imaging sensor for imaging the user's eye) in the viewfinder. When a user looks into a camera viewfinder, the distance between the eyeball and the eyepiece lens that constitutes the viewfinder optical system (an optical system through which the user looks into the viewfinder to view a subject, etc.) is usually not constant. In consideration of this, an optical path splitting prism is provided as part of the viewfinder optical system for the eye-gaze input function, and part of the optical axis of the eyeball imaging optical system (an optical system for focusing an optical image of the eye on the imaging surface of the eyeball imaging sensor) is coaxial with part of the optical axis of the viewfinder optical system. Patent Document 1 discloses a video camera that detects the gaze position (the position where the user is looking). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 5-091394 Summary of the Invention [Problem to be solved by the invention]

[0004] Now, suppose that in the video camera shown in Patent Document 1, an attempt is made to increase the magnification of the viewfinder optical system or improve the optical performance of the viewfinder optical system while maintaining the eye-gaze input function. In that case, the presence of an optical path-splitting prism would require the viewfinder section to be enlarged and costs to be significantly increased. However, if the optical path-splitting prism is removed and the optical axis of the eyeball imaging optical system and the optical axis of the viewfinder optical system are made independent of each other, it is possible to prevent the viewfinder section from being enlarged and costs to be significantly increased.

[0005] However, if the optical path splitting prism is removed and the optical axis of the eye imaging optical system and the optical axis of the viewfinder optical system are made independent of each other, it becomes difficult for the eye imaging sensor to capture the user's eye. For example, as the user's eye moves away from the eyepiece, the position of the pupil center in the image of the eye captured by the eye imaging sensor changes, and the user's eye moves out of the imaging range of the eye imaging sensor.

[0006] The present invention aims to provide a technology that makes it easier to catch the user's eye while suppressing increases in the size and cost of the device. [Means for solving the problem]

[0007] The display device of the present invention comprises a display panel, a display optical system for viewing the display panel, an imaging sensor having a rectangular imaging surface for capturing an image of the eye of a user viewing the display panel, and an imaging optical system for forming an optical image of the eye on the imaging surface, wherein the optical axis of the imaging optical system is non-parallel to the optical axis of the display optical system, and a straight line formed by projecting the optical axis of the display optical system onto the imaging surface is approximately parallel to the long side of the imaging surface.

[0008] The finder device of the present invention comprises the display device and an eyepiece to which the eye is placed, and the display device further comprises a detection means for detecting the line of sight of the user based on an image of the eye captured by the image sensor.The imaging device of the present invention comprises a second image sensor for capturing an image of a subject, and the display device capable of displaying the image of the subject captured by the second image sensor. [Effects of the Invention]

[0009] According to the present invention, it is possible to prevent the device from becoming larger and more expensive, while making it easier to catch the user's eye. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. [Figure 2]FIG. 2 is a block diagram of a camera. [Figure 3] FIG. 2 is a cross-sectional view of the camera housing. [Figure 4] A cross-sectional view of the EVF unit. [Figure 5] FIG. 5 is a cross-sectional view showing a part of FIG. [Figure 6] FIG. 1 is a diagram showing the layout of the EVF panel and the gaze detection system. [Figure 7] FIG. 1 is a diagram showing the arrangement of a G3 lens and a line-of-sight detection system. [Figure 8] FIG. 10 is a diagram showing an image captured by a gaze sensor. [Figure 9] FIG. 10 is a diagram showing an image captured by a gaze sensor. [Figure 10] 10 is a flowchart of a gaze detection operation. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described.

[0012] The appearance of a camera 1 (digital still camera; interchangeable lens camera) that is an imaging device according to this embodiment will be described using FIGS. 1(a) and 1(b). FIGS. 1(a) and 1(b) show the appearance of the camera 1. The present invention is applicable to any electronic device capable of detecting a user's gaze. For example, a user may view information such as images or text displayed on a display device, or view an optical image through an eyepiece optical system (display optical system). These electronic devices may include, for example, mobile phones, game consoles, tablet devices, personal computers, watch-type or eyeglass-type information terminals, head-mounted displays, and binoculars. The user's gaze, in other words, is the user's gaze position on an image or the like displayed on a display device.

[0013] Fig. 1(a) is a front perspective view, and Fig. 1(b) is a rear perspective view. As shown in Fig. 1(a), camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operating member that accepts image capturing operations from a user (photographer), is disposed on camera housing 1B.

[0014] 1(a) is the most basic position (standard position), or the normal position. In this case, the optical axis of the optical system of the photographing lens unit 1A is the Z axis, and the vertically upward axis perpendicular to the Z axis is the Y axis. The right-handed axis perpendicular to both the Y axis and the Z axis is the X axis.

[0015] As shown in FIG. 1(b), an eyepiece window frame 121 (eyepiece portion) is disposed on the back of the camera housing 1B, through which the user looks into the EVF panel 6 (described below) contained within the camera housing 1B. The eyepiece window frame 121 holds an eyepiece window 13 and protrudes outward (toward the back) from the camera housing 1B. Operation members 41 to 43 that accept various operations from the user are also disposed on the back of the camera housing 1B. For example, operation member 41 is a touch panel that accepts touch operations, operation member 42 is an operation lever that can be pushed down in each direction, and operation member 43 is a four-way key that can be pressed in each of four directions. Operation member 41 (touch panel) is equipped with a display panel 411 such as a liquid crystal panel and has the function of displaying images on the display panel 411.

[0016] The internal configuration of the camera 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the internal configuration of the camera 1.

[0017] The image sensor 2 is an image sensor (image capture sensor) such as a CCD or CMOS sensor, and is used to capture an image of a subject. The image sensor 2 photoelectrically converts an optical image formed on the imaging surface of the image sensor 2 by the optical system of the photographing lens unit 1A, and outputs the resulting analog image signal to an A / D converter (not shown). The A / D converter A / D converts the analog image signal obtained by the image sensor 2 and outputs it as image data.

[0018] Photographing lens unit 1A is composed of an optical system including a zoom lens, focus lens, diaphragm, etc., and when attached to camera housing 1B, it guides light from a subject to image sensor 2 and forms an image of the subject on the imaging surface of image sensor 2. Aperture control unit 518, focus adjustment unit 519, and zoom control unit 520 each receive instruction signals from CPU 3 via mount contacts 117, and drive and control the diaphragm, focus lens, and zoom lens in accordance with the instruction signals.

[0019] The CPU 3 included in the camera housing 1B reads out a control program for each block included in the camera housing 1B from the ROM included in the memory unit 4, expands the program into the RAM included in the memory unit 4, and executes it. In this way, the CPU 3 controls the operation of each block included in the camera housing 1B. The CPU 3 is connected to a line-of-sight detection unit 201, a photometry unit 202, an autofocus detection unit 203, a signal input unit 204, a light source drive unit 205, an eye proximity detection unit 208, a distance calculation unit 209, a display device drive unit 521, and the like. The CPU 3 also transmits signals to an aperture control unit 518, a focus adjustment unit 519, and a zoom control unit 520, which are arranged in the photographing lens unit 1A, via the mount contacts 117. In this embodiment, the memory unit 4 has a function of storing image signals from the image sensor 2 and the line-of-sight image sensor 19.

[0020] The line-of-sight imaging sensor 19 is an imaging element such as a CCD or CMOS sensor, and is an eyeball imaging sensor for capturing an image of the eye of the user viewing the EVF panel 6.

[0021] The gaze detection unit 201 A / D converts the output (gaze sensor image) of the gaze image sensor 19, for example, the output of the gaze image sensor 19 in a state where an eyeball image (optical image of the eyeball) is formed near the gaze image sensor 19, and sends the result to the CPU 3. The CPU 3 extracts feature points required for gaze detection from the gaze sensor image according to a predetermined algorithm described later, and calculates the user's gaze (gaze position on the display surface of the EVF panel 6) from the positions of the feature points.

[0022] The photometry unit 202 amplifies, logarithmically compresses, and A / D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor, specifically the luminance signal corresponding to the brightness of the field, and sends the result to the CPU 3 as field luminance information.

[0023] The autofocus detection unit 203 A / D converts signal voltages from multiple detection elements (multiple pixels) included in the image sensor 2 and used for phase difference detection, and sends the converted signal to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are focus detection points at 180 locations on the image plane corresponding to the 180 locations on the field of view image (display surface of the EVF panel 6) in the viewfinder.

[0024] Switches SW1 and SW2 are connected to the signal input unit 204. Switch SW1 is a switch for starting the photometry, distance measurement, line of sight detection, etc. of the camera 1, and is turned on by the first stroke (e.g., half-press) of the release button 5. Switch SW2 is a switch for starting the photographing operation, and is turned on by the second stroke (e.g., full press) of the release button 5. ON signals from switches SW1 and SW2 are input to the signal input unit 204. and transmits it to the CPU 3. The signal input unit 204 also receives operation inputs from the operation member 41 (touch panel), operation member 42 (operation lever), and operation member 43 (four-way key) shown in FIG.

[0025] The infrared LED 14 is a light source that irradiates the user's eyeball with infrared light. The light source driving unit 205 drives the infrared LED 14 based on a signal (instruction) from the CPU 3. For example, the light source driving unit 205 drives the infrared LED 14 in accordance with the instruction from the CPU 3 so that the infrared LED 14 emits light at a predetermined emission intensity.

[0026] The image processing unit 206 performs various image processing on the image data stored in the RAM of the memory unit 4. For example, correction of pixel defects caused by the optical system or the image sensor, demosaicing, white balance correction, color interpolation, gamma processing, etc. are performed.

[0027] The recording / output unit 207 records data including image data on a recording medium such as a removable memory card, or outputs this data to an external device via an external interface.

[0028] The eyepiece detection sensor 50 is an infrared proximity sensor, a capacitance sensor, etc. The eyepiece detection unit 208 transmits the output of the eyepiece detection sensor 50 to the CPU 3. The CPU 3 determines whether the user's eye is placed in contact with the eyepiece window frame 121 (eyepiece unit) based on the output of the eyepiece detection sensor 50 (eyepiece detection unit 208) in accordance with a predetermined algorithm.

[0029] The distance calculation unit 209 calculates the distance from the viewfinder to the user's eye based on the coordinates of a corneal reflection image (an image formed by specular reflection of infrared light emitted from the infrared LED 14 on the cornea) in the image (gaze sensor image) captured by the gaze image sensor 19. For example, the distance calculation unit 209 calculates the distance from the final surface of the display optical system 12 (described below) through which the user views the EVF panel 6 to the eye. Then, the distance calculation unit 209 transmits the calculated distance to the CPU 3.

[0030] The display device driving unit 521 drives the display device 510 based on a signal from the CPU 3. For example, the display device driving unit 521 displays an image of a subject captured by the imaging element 2, various pieces of information, and the like on the display device 510. The display device 510 is the display panel 411 or the EVF panel 6.

[0031] The configuration of camera housing 1B will be described using Figure 3. Figure 3 is a cross-sectional view of camera housing 1B cut along the YZ plane formed by the Y axis and Z axis shown in Figure 1(a), and is a diagram that schematically shows the configuration of camera housing 1B. This is a cross-sectional view of camera housing 1B in the normal position as seen from the user's left hand side.

[0032] In the camera housing 1B, the shutter 7 and the image sensor 2 are aligned in the optical axis direction of the photographing lens unit 1A. A display panel 411 is provided on the back of the camera housing 1B. The display panel 411 displays menus and images for operating the camera 1 and viewing and editing images captured by the camera 1. The display panel 411 is configured from a backlit LCD panel, an organic EL panel, or the like. An EVF unit 1C (finder device; finder module) is provided on the top of the camera housing 1B, and includes an EVF panel 6, a display optical system 12, and an eye gaze detection system 20. The EVF panel 6 is capable of displaying a screen similar to that of the display panel 411 and is configured from a backlit LCD panel, an organic EL panel, or the like. The display optical system 12 and the eye gaze detection system 20 will be described in detail below. The EVF unit 1C may or may not be detachable from the camera housing 1B (it may be fixed as part of the camera housing 1B).

[0033] The configuration of the EVF unit 1C will be described with reference to Fig. 4. Fig. 4 is a cross-sectional view of the EVF unit 1C cut along the YZ plane, and is a diagram that schematically shows the configuration of the EVF unit 1C.

[0034] The EVF panel 6, the display optical system 12, and the eyepiece window 13 are aligned along a display optical system optical axis 23 which is the optical axis of the display optical system 12.

[0035] The display optical system 12 is disposed in front of the display surface of the EVF panel 6 and is typically composed of multiple lenses to magnify the EVF panel 6. In this embodiment, the display optical system 12 is composed of three lenses: a G1 lens 9, a G2 lens 10, and a G3 lens 11. The number of lenses constituting the display optical system 12 is not particularly limited and may be four or five, for example. The G1 lens 9, G2 lens 10, and G3 lens 11 are optical lenses that transmit visible light and are made of optical glass or transparent optical plastic manufactured by cutting and grinding or molding.

[0036] The eyepiece window 13, located further in front of the display optical system 12 (on the opposite side of the display optical system 12 from the EVF panel 6), is a transparent member that has a portion that transmits visible light. The image displayed on the EVF panel 6 is magnified by the display optical system 12 and is observed by the user through the transparent portion of the eyepiece window 13.

[0037] The lenses and eyepiece window 13 that make up the display optical system 12 are not necessarily limited to those having only optically effective shapes and surfaces (e.g., those having only transparent surfaces). For example, the lenses and eyepiece window 13 that make up the display optical system 12 may have positioning or reinforcing shapes, shapes that allow the worker to grip them during assembly, shapes that serve as adhesive surfaces when adhesively fixed, or hollowed-out shapes, and these parts do not have to be transparent. Furthermore, parts that do not require optical transparency (e.g., parts where light transmission is not desired) may have light-shielding surfaces painted or printed.

[0038] An infrared LED 14 and an infrared-transmitting window 16 are provided behind the eyepiece window 13 (on the EVF panel 6 side). The infrared-transmitting window 16 is a window that conceals the infrared LED 14 so that it cannot be seen from the outside, and is made of a resin that absorbs visible light and transmits infrared light.

[0039] A line-of-sight detection system 20 is also arranged behind the eyepiece window 13. The line-of-sight detection system 20 includes a diaphragm 17, a line-of-sight optical system 18, and a line-of-sight imaging sensor 19, which are aligned along a line-of-sight optical system optical axis 22, which is the optical axis of the line-of-sight optical system 18.

[0040] The aperture 17 is an aperture that narrows the light flux necessary to form an image of the user's eye (eyeball 21) on the line-of-sight imaging sensor 19. In this embodiment, in order to detect the light emitted from the infrared LED 14 and reflected by the eyeball 21, the aperture 17 is provided with a filter that absorbs visible light and transmits infrared light.

[0041] The line-of-sight optical system 18 is an optical system (eyeball imaging optical system) for forming an optical image of the eyeball 21 on the imaging surface of the line-of-sight imaging sensor 19, and is composed of optical lenses etc. Although one lens is shown as the line-of-sight optical system 18 in Fig. 4, the line-of-sight optical system 18 may be composed of multiple lenses.

[0042] The gaze imaging sensor 19 is an eyeball imaging sensor for imaging the eyeball 21, and outputs an image containing an infrared component (gaze sensor image; for example, an image of the user's eye). The imaging surface of the gaze imaging sensor 19 is rectangular, and the gaze sensor image is also rectangular. Gaze sensor image The details will be described later with reference to FIGS. 8(a) and 8(b).

[0043] In this embodiment, the aperture 17, the line-of-sight optical system 18, and the line-of-sight imaging sensor 19 of the line-of-sight detection system 20 are separate components, but the line-of-sight detection system 20 may also be a modular, compact camera in which these components are packaged together.

[0044] In this embodiment, the line-of-sight optical system optical axis 22 and the display optical system optical axis 23 are non-parallel, and intersect at an angle 24. Specifically, when the camera housing 1B is in the normal position (predetermined position), the line-of-sight detection system 20 is located below the Y axis in the EVF unit 1C. The line-of-sight optical system optical axis 22 is configured to face toward the display optical system optical axis 23 located above the Y axis (diagonally upward in the YZ plane).

[0045] A conventional configuration includes a configuration in which an optical path splitting mirror or an optical path splitting prism is disposed as part of the display optical system 12, thereby partially aligning the line-of-sight optical system optical axis 22 and the display optical system optical axis 23. However, such a configuration makes it significantly more difficult to improve the optical performance of the EVF unit 1C while maintaining its size, compared to a configuration that does not use an optical path splitting mirror or an optical path splitting prism. Furthermore, optical path splitting prisms and the like are generally expensive, which can lead to increased costs. In this embodiment, the line-of-sight optical system optical axis 22 and the display optical system optical axis 23 are not parallel to each other, and a configuration that does not use an optical path splitting mirror or an optical path splitting prism is adopted, thereby improving the optical performance of the EVF unit 1C while suppressing increases in size and cost.

[0046] In this embodiment, the line-of-sight optical system optical axis 22 and the display optical system optical axis 23 are assumed to be on the same YZ plane. However, these two optical axes do not have to be on the same YZ plane; for example, one of the optical axes may be offset in the X-axis direction. In other words, the two optical axes may be in a twisted relationship.

[0047] A preferred arrangement of the line-of-sight optical system optical axis 22 and the display optical system optical axis 23 will be described with reference to Fig. 5. Fig. 5 is a cross-sectional view showing a part of Fig. 4.

[0048] 5, the camera housing 1B is in the normal position, and the eyeball 21 of the user looking into the EVF unit 1C is on the display optical system optical axis 23. The eyeball 21 is covered by the user's eyelid 30, which is made up of an upper eyelid 27 and a lower eyelid 28. The infrared LED 14 and the infrared-transmitting window 16 are positioned so as to irradiate the eyeball 21 with infrared light from above and below the display optical system optical axis 23, respectively.

[0049] The line-of-sight optical system optical axis 22 (line-of-sight imaging sensor 19) is disposed in a direction that looks up at the eyeball 21 from below the display optical system optical axis 23 (display optical system 12; EVF panel 6), i.e., from the direction of the user's lower eyelid 28. The upper eyelid 27 is usually larger and thicker than the lower eyelid 28. Therefore, by disposing the line-of-sight optical system optical axis 22 so that it looks up at the eyeball 21 from the lower eyelid 28 side, it is easier to capture the eyeball 21 than by disposing the line-of-sight optical system optical axis 22 so that it looks down at the eyeball 21 from the upper eyelid 27 side. Specifically, when the line-of-sight imaging sensor 19 images the eyeball 21, it is possible to prevent the eyeball 21 from being blocked by the eyelid 30 (occurrence of vignetting). Similarly, it is also possible to prevent the eyelid 30 from blocking the image of the chief ray of the specular reflection component of infrared light emitted by the infrared LED 14 (corneal reflection image; Purkinje image; Purkinje image). The smaller the angle 24, that is, the closer the line of sight optical system optical axis 22 and the display optical system optical axis 23 are to being parallel or coincident with each other, the easier it is for the line of sight detection system 20 to capture the image of the eyeball 21, so it is preferable that the angle 24 be small.

[0050] Since the camera 1 can be held in various ways, the user's (eyeball 21) posture (relative posture) with respect to the posture of the camera housing 1B also varies. Therefore, in the posture (posture and positional relationship between the camera housing 1B, eyeball 21, and eyelid 30) that is assumed to be the most frequent, It is preferable to set the optical axis 22 of the line of sight optical system so that the viewer looks up at the line of sight optical system 21.

[0051] The arrangement of the gaze detection system 20 will be explained in more detail using Figures 6 and 7. In Figures 6 and 7, the gaze detection system 20, which is made up of multiple components, is shown as a single modular unit. Figure 6 schematically shows the arrangement of the EVF panel 6 and the gaze detection system 20 as seen from the eyeball 21.

[0052] As shown in FIG. 6, when the EVF panel 6 is viewed from the eyeball 21 on the display optical system optical axis 23, the shape of the display surface of the EVF panel 6 is a horizontally long rectangle in which the horizontal side 6a (a side substantially parallel to the horizontal direction (left-right direction)) is longer than the vertical side 6b (a side substantially parallel to the vertical direction (up-down direction)). The lengths of the horizontal side 6a and the vertical side 6b satisfy the following formula 1. In other words, three times the length of the horizontal side 6a (the long side of the display surface of the EVF panel 6) is four or more times the length of the vertical side 6b (the short side of the display surface of the EVF panel 6). For example, the aspect ratio of the display surface of the EVF panel 6 (the length of the horizontal side 6a:the length of the vertical side 6b) is approximately 4:3. 3 × {length of horizontal side 6a} ≥ 4 × {length of vertical side 6b} (Equation 1)

[0053] The line-of-sight detection system 20 is positioned so as to look up from below the EVF panel 6 having the display surface. This makes it possible to suppress the occurrence of vignetting, as described above. Furthermore, because the display surface of the EVF panel 6 is horizontally long (the vertical side 6b is short), the angle 24 between the line-of-sight optical system optical axis 22 and the display optical system optical axis 23 becomes small, thereby further suppressing the occurrence of vignetting. If the area of the display surface of the EVF panel 6 is constant, the larger the aspect ratio of the display surface (length of horizontal side 6a / length of vertical side 6b), the more significantly the occurrence of vignetting can be suppressed. However, the shape of the display surface of the EVF panel 6 is not limited to a shape that satisfies Expression 1.

[0054] FIG. 7 shows a schematic diagram of the arrangement of the G3 lens 11 and the line-of-sight detection system 20 of the display optical system 12 as viewed from the eyeball 21 (in a direction parallel to the optical axis 23 of the display optical system).

[0055] The G3 lens 11 is an optical lens that constitutes the display optical system 12. The G3 lens 11 has an optical surface required for magnifying the EVF panel 6 within a lens peripheral portion 111, which is a circular peripheral portion centered on the optical axis 23 of the display optical system. The light beam required for magnifying the EVF panel 6 does not necessarily pass through the entire optical surface of the G3 lens 11. If necessary, part or all of the lens peripheral portion 111 can be removed (cut out) using straight or curved lines. In FIG. 7 , the lower portion of the lens peripheral portion 111 is cut out by a straight notch line 112. The gaze detection system 20 (gaze imaging sensor 19) is positioned so that a portion of the gaze detection system 20 (gaze imaging sensor 19) is included in a cutout area 11b surrounded by the lower portion of the lens peripheral portion 111 and the cutout line 112. The cutout line 112 is approximately parallel to the horizontal side 6a of the EVF panel 6 ( FIG. 6 ). This makes it possible to position the gaze detection system 20 closer to the display optical system optical axis 23, and makes it possible to make the gaze detection system optical axis 22 and the display optical system optical axis 23 closer to being parallel (coincident). Note that the cutout area 11b may include the entire gaze detection system 20 (gaze imaging sensor 19), or the cutout line may include a straight portion that is approximately parallel to the horizontal side 6a of the EVF panel 6 and a portion that is not.

[0056] The lenses of the display optical system 12, including the G3 lens 11, are made of optical glass or transparent optical plastic manufactured by cutting and grinding or molding, and from the viewpoint of processing costs and ensuring optical performance, it is preferable that the notch line 112 is a simple straight line. However, the notch line 112 is not limited to a straight line. For example, the notch line 112 may be a curved line that includes a straight line in part, or the entire notch line 112 may be a curved line. A hole may be drilled (cut out) in the outer periphery 111 of the lens, and the line of sight detection system 20 may be arranged so that the optical axis 22 of the line of sight optical system passes through a cylindrical hole formed in the outer periphery 111 of the lens by the drilling. In this case, the cut out The notch line 112 has a circular or elliptical shape. Cutting or molding may be performed so that the notch line 112 has an arc shape.

[0057] Furthermore, a cutout region may be provided in only one lens of the display optical system 12, or cutout regions may be provided in multiple lenses. In Fig. 4, cutout regions 10b and 11b are provided in two of the three lenses (G2 lens 10 and G3 lens 11) that make up the display optical system 12. Then, the gaze detection system 20 is arranged so that a part of the gaze detection system 20 is included in the cutout regions 10b and 11b when viewed from a direction parallel to the display optical system optical axis 23.

[0058] A gaze detection method using gaze detection system 20 will be described using Figures 9(a), 9(b), and 10. Figure 9(a) is a schematic diagram showing an image (gaze sensor image) captured by gaze image sensor 19. Figure 9(a) shows the gaze sensor image obtained when an eyeball image is projected onto gaze image sensor 19. Figure 9(b) shows the output intensity of gaze image sensor 19 (luminance of gaze sensor image). Figure 9(b) shows luminance information (luminance distribution) of area α in Figure 9(a). In Figure 9(b), the horizontal direction of the gaze sensor image is the X-axis direction and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. Figure 10 shows a schematic flowchart of gaze detection operation.

[0059] Infrared light emitted from two infrared LEDs 14 aligned in the X-axis direction illuminates the cornea of the user's eyeball 21. At this time, as shown in FIG. 9( a), two corneal reflection images (two corneal reflection images corresponding to the two infrared LEDs 14) formed by part of the infrared light reflected from the corneal surface are formed near the line-of-sight imaging sensor 19 and become corneal reflection images Pd and Pe in the line-of-sight sensor image. Similarly, an optical image of the pupil of the eyeball 21 is formed near the line-of-sight imaging sensor 19 and becomes pupil image 212 in the line-of-sight sensor image. An optical image of the iris of the eyeball 21 is also formed near the line-of-sight imaging sensor 19 and becomes iris image 213 in the line-of-sight sensor image. Hereinafter, the two ends of the pupil image 212 in the X-axis direction will be referred to as pupil edge images 212a and 212b. The corneal reflection images Pd and Pe are captured within the pupil image 212 or iris image 213. In this embodiment, the optical axis 22 of the line of sight optical system looks up at the optical axis 23 of the display optical system from below the Y axis (the optical axis 22 of the line of sight optical system looks up at the eyeball 21 from the side of the lower eyelid 28), so the pupil image 212 and the iris image 213 appear distorted, tapering upward.

[0060] 10, when the gaze detection operation starts, the infrared LED 14 emits infrared light. For example, the infrared LED 14 emits infrared light toward the user's eyeball 21. An image of the user's eyeball illuminated by the infrared light passes through the gaze optical system 18, is formed on the gaze image sensor 19, and is photoelectrically converted. This provides an electrical signal of the gaze sensor image that can be processed.

[0061] In step S802, the line-of-sight detection unit 201 sends the image obtained from the line-of-sight image sensor 19 (line-of-sight sensor image) to the CPU 3.

[0062] In step S803, CPU 3 obtains the coordinates of the point corresponding to the corneal reflection image and the pupil center from the gaze sensor image obtained in step S802.

[0063] As shown in FIG. 9(b), an extremely high level of brightness is obtained at the coordinates Xd and Xe of the corneal reflection images Pd and Pe. In the area of the pupil image 212 (the area from the coordinate Xa corresponding to the pupil edge image 212a to the coordinate Xb corresponding to the pupil edge image 212b), an extremely low level of brightness is obtained except for the coordinates Xd and Xe. In the area of the iris image 213 outside the pupil image 212, a brightness intermediate between the two types of brightness mentioned above is obtained. From the brightness distribution shown in FIG. 9(b), it can be seen that the X coordinates Xd and Xe of the corneal reflection images Pd and Pe and the X coordinate Xa of the pupil edge images 212a and 212b , Xb can be obtained. In addition, the intermediate position between the X coordinates Xa and Xb can be calculated as the X coordinate Xc of the pupil center c.

[0064] In step S804, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position (relative position) of the eyeball 21 with respect to the line of sight optical system 18, and can be calculated using a function of the distance (Xd-Xe) between the corneal reflection images Pd and Pe.

[0065] In step S805, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 21 relative to the line-of-sight optical system optical axis 22. The X coordinate of the midpoint between the corneal reflection images Pd and Pe and the X coordinate of the corneal center of curvature approximately coincide. Therefore, if the standard distance from the corneal center of curvature to the pupil center is defined as distance Oc, the rotation angle θx of the eyeball 21 in the XZ plane can be approximately calculated using the following equation 2. The rotation angle θy of the eyeball 21 in the YZ plane can also be calculated in a similar manner. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 2)

[0066] In step S806, CPU 3 estimates the coordinates (Hx, Hy) of the user's gaze position (point of view; position where the user's gaze is fixed; position where the user is looking) on the display surface of EVF panel 6 using the rotation angles θx, θy calculated in step S805.

[0067] In step S807, the CPU 3 stores the estimated coordinates (Hx, Hy) of the gaze position in the memory unit 4, and ends the gaze detection operation.

[0068] The gaze sensor images will be described using Figures 8(a) and 8(b). The position of the eyeball 21 relative to the EVF unit 1C cannot be determined to be one position. For example, the position of the user's eyeball 21 looking into the EVF panel 6 changes depending on whether the user is wearing glasses, the user's physical condition, the user's hairstyle, whether the user is wearing a hat, and so on. Figure 4 shows eyeball 21a close to the EVF panel 6 and eyeball 21b far from the EVF panel 6 as eyeballs 21 positioned on the display optical system optical axis 23. Figures 8(a) and 8(b) show gaze sensor image 211, which is a superposition of the gaze sensor image for eyeball 21a and the gaze sensor image for eyeball 21b.

[0069] Fig. 8(a) shows an eye-gaze sensor image 211 according to this embodiment, and for comparison Fig. 8(b) shows a general eye-gaze sensor image 211. The eye-gaze optical system optical axis 22 and the eye-gaze optical system 18 are the same in Fig. 8(a) and Fig. 8(b), but the orientation of the eye-gaze image sensor 19 (the length and width of the imaging surface) is different.

[0070] Eyeball image 221a is an image formed by eyeball 21a near gaze image sensor 19, and eyeball image 221b is an image formed by eyeball 21b near gaze image sensor 19. Naturally, both eyeball image 221a and eyeball image 221b are not captured in an actual gaze sensor image; only one of eyeball image 221a and eyeball image 221b can be captured. Projection optical axis 223 is a straight line formed by projecting display optical system optical axis 23 onto gaze sensor image 211 (the imaging surface of gaze image sensor 19).

[0071] The imaging surface of the line-of-sight imaging sensor 19 is rectangular, and the line-of-sight sensor image is also rectangular. In FIGS. 8(a) and 8(b), the line-of-sight sensor image 211 is a rectangular image having long sides 211a and short sides 211b. The long side 211a is longer than the short side 211b. In FIG. 8(a) (this embodiment), the projection optical axis 223 is approximately parallel to the long side 211a of the line-of-sight sensor image 211 (the long side of the line-of-sight imaging sensor 19). On the other hand, in FIG. 8(b) (comparative example), the projection optical axis 223 is approximately parallel to the short side 211b of the line-of-sight sensor image 211 (approximately perpendicular to the long side 211a).

[0072] 8(a) and 8(b), in Fig. 8(b) (comparative example), eyeball image 221b is not included in gaze sensor image 211, but in Fig. 8(a) (present embodiment), eyeball image 221b is completely included in gaze sensor image 211. In other words, eyeball 21b cannot be captured in the comparative example, but eyeball 21b can be captured in this embodiment.

[0073] When the user looks into the EVF unit 1C, the user adjusts the position of the eyeball 21 so that the center of the eyeball approaches the display optical system optical axis 23. However, as described above, the position of the eyeball 21 on the display optical system optical axis 23 varies. According to this embodiment, the projection optical axis 223 is approximately parallel to the long side 211a of the gaze sensor image 211 (the long side of the gaze imaging sensor 19), so that the range on the display optical system optical axis 23 that can be detected by the gaze detection system 20 can be expanded. Furthermore, the eyeball 21 can be captured even if it is located far from the eyepiece window 13.

[0074] Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments. In the above embodiments, the present invention can be considered to be applied to an imaging device or a viewfinder device. The present invention can be applied to any device having a display panel in which the distance between the display panel and the eye viewing the display panel can be changed. For example, the present invention can be applied to a display device such as an HMD (head-mounted display). When the present invention is applied to an HMD, for example, at least one of a right-eye gaze detection system and a left-eye gaze detection system is provided in the HMD as the gaze detection system 20, and the HMD detects the gaze of a user wearing the HMD (at least one of the right and left eyes). The display panel of the HMD may also have a rectangular display surface, and may be oriented so that the long side of the display surface is approximately parallel to the up-down direction when the HMD is in the normal position. Even in such a case, the gaze detection system may be positioned so that the projection optical axis is approximately parallel to the long side of the gaze sensor image (the long side of the gaze image sensor). For example, when the HMD is in the normal position, the line-of-sight image capture sensor 19 is positioned horizontally to the left or right of the portrait-oriented EVF panel 6. The present invention also includes configurations that can be obtained by appropriately modifying or changing the configurations of the above-described embodiments within the scope of the gist of the present invention. [Explanation of symbols]

[0075] 1: Camera 1B: Camera body 1C: EVF unit 2: Image sensor 6: EVF panel 12: Display optical system 18: Line-of-sight optical system 19: Line-of-sight imaging sensor

Claims

1. A display panel; a display optical system for viewing the display panel; an image sensor having a rectangular imaging surface for capturing an image of a user's eye viewing the display panel; an imaging optical system for forming an optical image of the eye on the imaging surface; and an optical axis of the imaging optical system is not parallel to an optical axis of the display optical system; The straight line obtained by projecting the optical axis of the display optical system onto the imaging surface is approximately parallel to the long side of the imaging surface. A display device characterized by:

2. The image sensor is disposed in such a manner that when the display device is in a predetermined position, the image sensor is oriented so as to look up from below the optical axis of the display optical system.

2. The display device according to claim 1.

3. The image sensor is disposed in such a manner that when the display device is in the predetermined orientation, the image sensor is oriented so as to look up from below the display panel.

3. The display device according to claim 2.

4. The shape of the display surface of the display panel is a rectangle whose long sides are approximately parallel to the left-right direction when the display device is in the predetermined position.

4. The display device according to claim 3.

5. Three times the length of the long side of the display surface is four or more times the length of the short side of the display surface.

5. The display device according to claim 4.

6. the display optical system includes a lens having a shape obtained by removing a part of an outer periphery of a circle when viewed in a direction parallel to an optical axis of the display optical system, The image sensor is disposed so that, when viewed in a direction parallel to the optical axis of the display optical system, at least a portion of the image sensor is included in an area from which a portion of the outer periphery is removed.

6. The display device according to claim 1, wherein the display device is a display device having a plurality of light-emitting elements.

7. The lens has a shape in which a part of the outer periphery is cut out when viewed in a direction parallel to the optical axis of the display optical system.

7. The display device according to claim 6.

8. When viewed from a direction parallel to the optical axis of the display optical system, the lens has a shape such that a part of the outer periphery is cut out along a cutout line including a straight portion that is approximately parallel to a side of the display surface of the display panel.

8. The display device according to claim 7,

9. a detection means for detecting the line of sight of the user based on the image of the eye captured by the image sensor; Further having 9. The display device according to claim 1, wherein the display device is a display device having a plurality of light-emitting elements.

10. The display device according to claim 9 ; an eyepiece portion to which the eye is placed; A finder device comprising:

11. a second image sensor for capturing an image of a subject; the display device according to any one of claims 1 to 9, capable of displaying an image of the subject captured by the second image sensor; An imaging device comprising:

Images