Optical device and floating image display device

Abstract

An optical device according to the present disclosure generates light including first light and second light having focal depths different from each other. A floating image display device according to the present disclosure is provided with the optical device. The optical device comprises: a display device; and an optical element that, on the basis of light from the display device, generates image forming light including the first light and the second light having focal depths different from each other, and forms a floating image by one or both of the first light and the second light.

Description

Optical device and floating image display device

[0001] The present disclosure relates to an optical device and a floating image display device.

[0002] 2. Description of the Related Art Conventionally, a non-contact input device is known, for example, as described in Japanese Patent Application Laid-Open No. 2003-222999.

[0003] Japanese Patent Application Laid-Open No. 2020-67707

[0004] The optical device of the present disclosure generates light including first light and second light having different focal depths.

[0005] The floating image display device of the present disclosure comprises an optical device having the above-described configuration, and the optical device comprises: a display device; and an optical element that generates imaging light including the first light and the second light having different focal depths based on light from the display device, and forms a floating image using one or both of the first light and the second light.

[0006] The floating image display device of the present disclosure also includes: an optical device that generates imaging light including first light and second light having different focal depths, and forms a floating image using one or both of the first light and the second light; an imaging device that receives one or both of a first reflected light obtained by reflecting the first light by a first object located in a projection range of the imaging light, and a second reflected light obtained by reflecting the second light by a second object located in the projection range; and a control unit that controls the floating image based on one or both of individual differences of the first object and individual differences of the second object.

[0007] Objects, features, and advantages of the present disclosure will become clearer from the detailed description and drawings below.

[0023] Fig. 1 is a cross-sectional view of a main part of an aerial image display device according to an embodiment of the present disclosure.

[0024] Fig. 2 is a cross-sectional view of a main part of another example of the aerial image display device of Fig. 1.

[0025] Fig. 3 is a diagram illustrating the movement and position of fingers relative to an aerial image.

[0026] Fig. 4 is a front view illustrating a first edge-enhanced image generated by an image processing device.

[0027] Fig. 5 is a diagram illustrating the movement and position of fingers relative to an aerial image.

[0028] Fig. 6 is a front view illustrating a first edge-enhanced image generated by an image processing device.

[0029] Fig. 7 is a flowchart illustrating an example of the operation of the aerial image display device.

[0030] Fig. 8 is a front view illustrating an example of a captured image generated by an imaging unit.

[0031] Fig. 9 is a flowchart illustrating another example of the operation of the aerial image display device.

[0032] Fig. 10 is a cross-sectional view of a main part of another example of the aerial image display device of Fig. 11.

[0033] Fig. 11 is a cross-sectional view of a main part of an aerial image display device according to another embodiment of the present disclosure.

[0034] Fig. 12 is a plan view illustrating a display device included in the aerial image display device of Fig. 13.

[0035] Fig. 13 is a plan view illustrating another example of the display device included in the aerial image display device of Fig. 1.

[0036] Fig. 14 is a plan view illustrating another example of the display device included in the aerial image display device of Fig. 1.

[0008] In recent years, various non-contact input devices have been proposed that detect a user's gesture operation on an aerial image formed in space and execute a predetermined function based on the detected gesture operation. Patent Document 1 discloses a non-contact input device that includes an aerial image display device that forms an aerial image in space, a detection unit that detects the position of an object such as a user's finger, and a determination unit that determines the user's gesture operation based on the detection result by the detection unit.

[0009] Conventional non-contact operation detection devices have sometimes been unable to accurately detect a user's gesture operation.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this specification, a floating image may be referred to as an aerial image, and a floating image display device may be referred to as an aerial image display device. The figures referred to below show the main components of an optical device and an aerial image display device according to embodiments. The optical device and the aerial image display device according to embodiments may include well-known components such as a display device holding member, an optical device holding member, and a housing, which are not shown. The figures referred to below are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones. Furthermore, in some of the drawings, a Cartesian coordinate system XYZ is defined for convenience.

[0011] The optical device 3 of this embodiment is configured to generate light (also referred to as imaging light) L including a first light (also referred to as first imaging light) L1 and a second light (also referred to as second imaging light) L2, which have different focal depths. More specifically, the optical device 3 is configured to simultaneously generate the first light L1 and the second light L2, which have different focal depths. This configuration makes it possible to focus the light L at different positions in the optical axis direction and to form a floating image F (e.g., the aerial image R in FIG. 1 ). Furthermore, it is possible to capture and detect multiple objects at different positions in the optical axis direction of the light L.

[0012] As shown in FIG. 1 , the optical device 3 may be configured such that the first focal length DF1 of the first light L1 and the second focal length DF2 of the second light L2 are different. This configuration reliably enables focusing and forming a floating image F at different positions in the optical axis direction of the light L. It also reliably enables imaging and detecting multiple objects at different positions in the optical axis direction of the light L. The first focal length DF1 may be the focal length from the second concave mirror 33 or the focal length from the center of the second concave mirror 33. The same applies to the second focal length DF2. For example, the second focal length DF2 may be longer than the first focal length DF1. The difference between the first focal length DF1 and the second focal length DF2 may be due to a reflective optical element OD (e.g., the second concave mirror 33 in FIG. 1 ) having an anisotropic curvature of the reflective surface, as described below. For example, the first focal length DF1 is caused by a portion with a relatively large degree of curvature, and the second focal length DF2 is caused by a portion with a relatively small degree of curvature.

[0013] In the above configuration, the focus FP1 of the first light L1 may be outside the range of the focal depth of the second light L2 in the optical axis direction of the light L. In this case, it is more reliably possible to image and detect multiple objects at different positions in the optical axis direction of the light L. Furthermore, the focus FP2 of the second light L2 may be outside the range of the focal depth of the first light L1 in the optical axis direction of the light L. In this case, it is more reliably possible to image and detect multiple objects at different positions in the optical axis direction of the light L. Furthermore, the focal depth of the first light L1 and the focal depth of the second light L2 may be at positions that do not overlap in the optical axis direction of the light L. In this case, it is more reliably possible to image and detect multiple objects at different positions in the optical axis direction of the light L.

[0014] The optical device 3 may be configured to include a reflective optical element OD (e.g., the second concave mirror 33 in FIG. 1 ) having an anisotropic curvature of the reflective surface. The curvature of the reflective surface of this optical element OD varies depending on the direction within the reflective surface. For example, the curvature in the horizontal direction (X direction in FIG. 1 ) when the reflective surface 33 a of the second concave mirror 33 is viewed from the front may differ from the curvature in the vertical direction (Y direction in FIG. 1 ) when the reflective surface 33 a is viewed from the front. The lateral curvature of the reflective surface 33 a when viewed from the front may be greater than the longitudinal curvature of the reflective surface 33 a when viewed from the front. In this case, the distance between the reflective surface 33 a and the focal point of light rays distributed in the horizontal direction of the reflective surface 33 a is shorter than the distance between the reflective surface 33 a and the focal point of light rays distributed in the vertical direction of the reflective surface 33 a. Therefore, the focal depth of light rays distributed in the horizontal direction of the reflective surface 33 a is smaller than the focal depth of light rays distributed in the vertical direction of the reflective surface 33 a. This optical element OD may be a first concave mirror 31 and a convex mirror 32 .

[0015] The reflective optical element OD may be a single optical element, or may be a composite optical element OD that combines multiple optical elements, such as a composite optical element OD that combines an optical element that imparts curvature in the horizontal direction and an optical element that imparts curvature in the vertical direction.

[0016] The optical device 3 may be configured to include a transmissive optical element having anisotropic refractive index. The refractive index of this optical element varies depending on the direction in the plane at one or both of the light incident surface and the light exit surface. For example, the refractive index in the horizontal direction when the light incident surface of the optical element is viewed from the front may be different from the refractive index in the vertical direction when the light incident surface is viewed from the front. The refractive index in the horizontal direction when the light incident surface is viewed from the front may be greater than the refractive index in the vertical direction when the light incident surface is viewed from the front. In this case, the distance between the incident surface and the focal point of light rays distributed in the horizontal direction of the incident surface is shorter than the distance between the incident surface and the focal point of light rays distributed in the vertical direction of the incident surface. Therefore, the focal depth of light rays distributed in the horizontal direction of the incident surface is smaller than the focal depth of light rays distributed in the vertical direction of the incident surface. This optical element may be a flat plate type, a convex lens type, a concave lens type, a plano-convex lens type (one of the entrance surface and the exit surface is flat and the other is convex), a plano-concave lens type (one of the entrance surface and the exit surface is flat and the other is concave), a concave-convex lens type (one of the entrance surface and the exit surface is concave and the other is convex), or the like.

[0017] Furthermore, the transmissive optical element having anisotropic refractive index may be a single optical member, or a composite optical element made up of a combination of multiple optical members, such as an optical element in which an optical member that imparts a refractive index in the horizontal direction and an optical member that imparts a refractive index in the vertical direction are stacked together.

[0018] The optical element OD may form a floating image F by one or both of the first light L1 and the second light L2. That is, it is not necessary to form a floating image F by the first light L1 and not by the second light L2. It is also possible to form a floating image F by the second light L2 without forming a floating image F by the first light L1. It is also possible to form a floating image F by the first light L1 and the second light L2. It is also possible to form one floating image F by the first light L1 and another floating image F by the second light L2.

[0019] The floating image display device 1 may form a floating image F in the air, underwater, in a transparent medium such as glass or plastic, in outer space, etc. In the following description of the embodiment, an aerial image display device 1 that forms an aerial image R in the air will be described.

[0020] 1, the aerial image display device 1 of this embodiment includes a display device 2 and an optical device 3. The aerial image display device 1 may further include an image processing device 6 and a control device 7.

[0021] The display device 2 includes a display unit 4. The display unit 4 has a display surface 4 a and displays an image formed as an aerial image R on the display surface 4 a. In other words, the display unit 4 emits light (also referred to as image light) L0 that is formed as the aerial image R from the display surface 4 a.

[0022] The display unit 4 may be composed of a transmissive display panel or a self-luminous display panel. When the display unit 4 is composed of a transmissive display panel, the display unit 4 may include an illuminator such as a backlight. The backlight may be a direct-type backlight having multiple light sources arranged two-dimensionally on the back side of the transmissive display panel. The backlight may be an edge-lit backlight having multiple light sources arranged around the periphery of the transmissive display panel. The backlight may include a lens array, a light guide plate, a diffusion plate, etc. for uniformly illuminating the transmissive display panel. The backlight light source may be, for example, a light-emitting diode (LED), a semiconductor laser (LD), a cold cathode fluorescent lamp, a halogen lamp, a xenon lamp, etc. The backlight light source includes a visible light source. The backlight light source may include a visible light source and an infrared light source. When the display unit 4 is composed of a self-luminous display panel, the display unit 4 does not need to include an illuminator.

[0023] The transmissive display panel may be a liquid crystal panel. The liquid crystal panel may be a known liquid crystal panel. Known liquid crystal panels include, for example, liquid crystal panels of an IPS (In-Plane Switching) type, an FFS (Fringe Field Switching) type, a VA (Vertical Alignment) type, an ECB (Electrically Controlled Birefringence) type, etc. The transmissive display panel is not limited to a liquid crystal panel, and may be a MEMS (Micro Electro Mechanical Systems) shutter type display panel.

[0024] The self-luminous display panel is configured to include a plurality of self-luminous elements. The self-luminous elements include, for example, LEDs, organic electroluminescence (OEL), inorganic electroluminescence (IEL), and the like. The self-luminous display panel may be a transparent display panel (hereinafter also referred to as a transparent display) configured to include a transparent substrate and a plurality of self-luminous elements. The transparent substrate may be made of a transparent material such as glass. The plurality of light-emitting elements may be arranged in a matrix on the transparent substrate, or may be housed in a plurality of recesses formed in a matrix on the transparent substrate.

[0025] The optical device 3 includes an optical element OD (e.g., a second concave mirror 33), and the optical element OD includes a free-form reflecting mirror. The free-form reflecting mirror has a reflecting surface at least partially formed of a free-form surface. The optical device 3 generates imaging light L including first imaging light L1 and second imaging light L2 having different focal depths based on image light L0 emitted from the display surface 4a of the display unit 4. The optical device 3 forms a real aerial image R using the imaging light L including the first imaging light L1 and the second imaging light L2. The aerial image R is formed on an imaging plane P in space. The focal depth of the first imaging light L1 refers to the range in the optical axis direction in which a clear aerial image is formed by the first imaging light L1 near the focal point of the first imaging light L1. The same applies to the focal depth of the second imaging light L2. The optical device 3 may be configured so that the focal depth of the first imaging light L1 (hereinafter also referred to as the first focal depth) is smaller than the focal depth of the second imaging light L2 (hereinafter also referred to as the second focal depth). The imaging plane P of the aerial image R may be located within the first focal depth of the first imaging light L1.

[0026] The phrase "generating imaging light L including first imaging light L1 and second imaging light L2 having different focal depths based on image light L0" may mean, as described above, generating first imaging light L1 and second imaging light L2 having different focal depths by reflecting image light L0 using a reflective optical element OD. It may also mean generating first imaging light L1 and second imaging light L2 having different focal depths by transmitting image light L0 through a transmissive optical element. It may also mean reflecting image light L0 using the reflective optical element OD and transmitting image light L0 through a transmissive optical element, generating first imaging light L1 and second imaging light L2 having different focal depths.

[0027] For example, the first focal depth may be approximately 1 to 4 mm, in which case the aperture value (F-number) is equivalent to approximately 1.1 to 1.7. The second focal depth may be approximately 1 to 70 mm, in which case the aperture value (F-number) is equivalent to approximately 16. The second focal depth may be approximately 70 mm or more, in which case the aperture value (F-number) is equivalent to approximately 16 or more. Therefore, the second focal depth may be approximately 15 times or more the first focal depth, but is not limited to this range. Note that "to" means "through," and the same applies hereinafter.

[0028] The reason why the image light L0 is converted into the imaging light L including the first imaging light L1 and the second imaging light L2 having different focal depths is as follows: The second concave mirror 33 has a focal depth that is different between light whose imaging surface (first imaging surface described below) is in the X-axis direction (horizontal direction, the first direction described below) and light whose imaging surface (second imaging surface described below) is in the Y-axis direction (vertical direction, the second direction described below). This is because the curvature (curvature) of the reflecting surface of the second concave mirror 33 in the X-axis direction is different from the curvature (curvature) of the reflecting surface of the second concave mirror 33 in the Y-axis direction. In other words, the curvature (curvature) of the reflecting surface of the second concave mirror 33 in the X-axis direction is greater than the curvature (curvature) of the reflecting surface of the second concave mirror 33 in the Y-axis direction. The second concave mirror 33 having this configuration may be a second concave mirror 33 made of a free-form reflecting mirror. Similarly, the first concave mirror 31 and the convex mirror 32 may also be made of a free-form reflecting mirror.

[0029] The free-form surface that defines the reflecting surface of the free-form reflecting mirror may be an XY polynomial surface (also called an SPS XYP surface) defined by the following formulas (1) and (2). The XY polynomial surface is expanded into polynomials of up to the tenth degree that are added to the reference conic surface. Therefore, in formulas (1) and (2), the sum of m and n is 10 or less. In formula (1), z is the sag of the surface parallel to the z axis (optical axis), c is the vertex curvature, and r is the radial distance (i.e., r ２ = x ２ +y ２ ), k is the conic constant, and Cj is the monomial x ｍ y ｎ is the coefficient of

[0030] The optical device 3 does not have to include a spherical reflecting mirror or an aspherical reflecting mirror (e.g., a parabolic reflecting mirror, a hyperbolic reflecting mirror, etc.) whose reflecting surface has rotational symmetry. The optical device 3 may be composed of at least one free-form reflecting mirror.

[0031] The first image-forming light L1 is located within a predetermined image-forming plane (hereinafter also referred to as the first image-forming plane), and the second image-forming light L2 is located within a predetermined image-forming plane (hereinafter also referred to as the second image-forming plane). In other words, the first image-forming light L1 propagates within the first image-forming plane, and the second image-forming light L2 propagates within the second image-forming plane. The first image-forming plane intersects with the image-forming plane P in a first direction within the image-forming plane P of the aerial image R, and the second image-forming plane intersects with the image-forming plane P in a second direction within the image-forming plane P. In other words, the first image-forming plane is a plane whose intersection with the image-forming plane P extends in the first direction, and the second image-forming plane is a plane whose intersection with the image-forming plane P extends in the second direction. In the following, a case will be described in which the first direction is the X-axis direction and the second direction is the Y-axis direction, but the first direction is not limited to the X-axis direction, and the second direction is not limited to the Y-axis direction. The first direction and the second direction may be any direction as long as they are substantially perpendicular to the Z-axis direction and intersect with each other.

[0032] As shown in FIG. 1 , the optical device 3 may be composed of a first concave mirror 31, a convex mirror 32, and a second concave mirror 33. The first concave mirror 31 is located on the optical path of the image light L0 emitted from the display surface 4a. The first concave mirror 31 is located in front of the display surface 4a (i.e., at a position spaced apart from the display surface 4a in the emission direction of the image light L0). The first concave mirror 31 is configured to reflect the image light L0 emitted from the display surface 4a in a direction different from the direction toward the display unit 4. The convex mirror 32 is located on the optical path of the reflected light reflected by the first concave mirror 31. The convex mirror 32 is configured to reflect the reflected light reflected by the first concave mirror 31 in a direction different from the direction toward the first concave mirror 31. The second concave mirror 33 is located on the optical path of the reflected light reflected by the convex mirror 32. The second concave mirror 33 is configured to reflect the light reflected by the convex mirror 32 in a direction different from the direction toward the convex mirror 32, thereby forming an aerial image R. In the optical device 3, only one or two of the first concave mirror 31, the convex mirror 32, and the second concave mirror 33 may be free-form reflecting mirrors, or all of them may be free-form reflecting mirrors. The optical device 3 is only required to be configured so that the reflected light reflected by at least the second concave mirror 33 becomes imaging light L including first imaging light L1 and second imaging light L2 (having different focal depths).

[0033] Alternatively, the reflected light reflected by the first concave mirror 31 may be configured to become imaging light L including the first imaging light L1 and the second imaging light L2. Alternatively, the reflected light reflected by the convex mirror 32 may be configured to become imaging light L including the first imaging light L1 and the second imaging light L2. That is, the first concave mirror 31, the convex mirror 32, and the second concave mirror 33 may convert the image light L0 into imaging light L including the first imaging light L1 and the second imaging light L2.

[0034] Furthermore, the light reflected by the first concave mirror 31 and the light reflected by the first concave mirror 31 and the convex mirror 32 do not need to include imaging light having different focal depths.

[0035] As shown in FIG. 2 , the optical device 3 may be composed of a first concave mirror 31 and a second concave mirror 33. The first concave mirror 31 is located on the optical path of the image light L0 emitted from the display surface 4a. The first concave mirror 31 is located in front of the display surface 4a (in a direction away from the display surface 4a along the emission direction of the image light L0). The first concave mirror 31 is configured to reflect the image light L0 emitted from the display surface 4a in a direction different from the direction toward the display unit 4. The second concave mirror 33 is configured to reflect the reflected light reflected by the first concave mirror 31 in a direction different from the direction toward the first concave mirror 31, thereby forming an aerial image R. In the optical device 3 shown in FIG. 2 , only one of the first concave mirror 31 and the second concave mirror 33 may be a free-form reflecting mirror, or all of them may be free-form reflecting mirrors. The optical device 3 is configured so that the reflected light reflected by at least the second concave mirror 33 becomes the imaging light L including the first imaging light L1 and the second imaging light L2 (having different focal depths). The reflected light reflected by the first concave mirror 31 may or may not include the imaging light having different focal depths.

[0036] The display device 2 may include an imaging unit 5. The imaging unit 5 may capture an image of an object O located in front of the aerial image display device 1 via the optical device 3. The front of the aerial image display device 1 refers to a direction away from the optical device 3 along the emission direction of the imaging light L, and includes the projection range of the imaging light L from the optical device 3. The object O may be the eye 10a of the user 10 or the user's finger 10b.

[0037] The imaging unit 5 receives reflected light of the imaging light L reflected by an object O within the projection range of the imaging light L from the optical device 3, but may also receive reflected light of ambient light reflected by the object O. The imaging light L from the optical device 3 may also be white light (image light). In this case, the brightness of the reflected light of the imaging light L reflected by the object O can be increased. This white light may be a portion of the imaging light L. For example, a finger 10b, which is the object O, may be brought into contact with a portion of the imaging light L (the white light portion).

[0038] The display unit 4 may be a transparent display. In this case, the imaging unit 5 may be located behind the display unit 4 (at a position spaced apart from the display unit 4 on the opposite side to the emission direction of the image light L0), as shown in FIG. 1 . The imaging unit 5 may receive light that has passed through the display unit 4 (light reflected from the object O). The imaging unit 5 may be disposed so that its imaging surface (light receiving surface) faces the display unit 4 and is parallel to the display surface 4 a. The imaging unit 5 may be located substantially at the confocal position of the optical device 3. In this case, when the object O is in contact with the aerial image R (i.e., located near the imaging plane P), it is possible to accurately detect the position of the object O.

[0039] The image capturing unit 5 being located at the confocal position of the optical device 3 means that the optical path (also referred to as the first optical path) from the display unit 4 to the aerial image R via the optical device 3 and the optical path (also referred to as the second optical path) from the image capturing unit 5 to the aerial image R via the optical device 3 are the same or different, and the lengths of the first optical path and the second optical path are the same. In this case, the first optical path and the second optical path are located at the same positions in the optical device 3.

[0040] The imaging unit 5 includes an imaging element. The imaging element may be, for example, a CCD (Charge Coupled Device) imaging element or a CMOS (Complementary Metal Oxide Semiconductor) imaging element. The imaging unit 5 may be a camera (for example, a CCD camera) including multiple imaging elements and optical elements such as an objective lens. The imaging unit 5 may be a visible light camera or an infrared camera.

[0041] The imaging unit 5 captures an image of an object O located in front of the aerial image display device 1 and generates a captured image. The captured image generated by the imaging unit 5 may be a color image. Image data of the captured image (hereinafter also referred to as luminance distribution data) may include a luminance value for each pixel. The luminance value may include an R (red) component, a G (green) component, and a B (blue) component. The R component value, the G component value, and the B component value may each be 8-bit data. In other words, the R component value, the G component value, and the B component value may each be represented by a numerical value (also referred to as a gradation value) between 0 and 255. Hereinafter, the R component value, the G component value, and the B component value may be referred to as R, G, and B, respectively.

[0042] The image processing device 6 acquires, from the imaging unit 5, luminance distribution data of the captured image generated by the imaging unit 5. The image processing device 6 is configured to detect the position of an object O located in front of the aerial image display device 1 based on the acquired luminance distribution data. The image processing device 6 may be configured to track the position of the object O.

[0043] The control device 7 is connected to and controls each component of the aerial image display device 1. The components controlled by the control device 7 include the display device 2 and the image processing device 6.

[0044] The control device 7 may be configured to include one or more processors. The processor may include a general-purpose processor configured to load a specific program and execute a specific function, and a dedicated processor specialized for specific processing. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control device 7 may be a system-on-a-chip (SoC) or system-in-a-package (SiP) in which one or more processors are configured to work together. The control device 7 may include a memory unit that stores various information or programs for operating each component of the aerial image display device 1. The memory unit may be configured, for example, as a semiconductor memory. The memory unit may function as a work memory for the control device 7. The control device 7 may include a memory unit, for example, as a random access memory (RAM) or a read-only memory (ROM). The memory may store the number and positions of fingers 10b in contact with aerial image R detected by image processing device 6. Control device 7 may include image processing device 6. In other words, image processing device 6 may constitute a part of control device 7.

[0045] <Detection of Gesture Operation> Next, a description will be given of the operation of detecting a gesture operation in the aerial image display device 1. The image capture unit 5 receives reflected light M of the imaging light L reflected by an object located in front of the aerial image display device 1 (the projection range of the imaging light L from the optical device 3). The image capture unit 5 receives reflected light M1 of the first imaging light L1 reflected by a first object O1 located in the projection range of the imaging light L (hereinafter also referred to as first reflected light) and reflected light M2 of the second imaging light L2 reflected by a second object O2 located in the projection range of the imaging light L.

[0046] The first object O1 may be in contact with the aerial image R. The first object O1 may be the fingers 10b of the user 10 performing a gesture operation on the aerial image R. The imaging unit 5 generates a captured image (hereinafter also referred to as a first reflected image) based on the first reflected light M1. The first reflected image may include an image of the fingers 10b of the user 10. The first object O1 is not limited to the fingers 10b of the user 10. The first object O1 may be, for example, the tip of a rod-shaped object held by the user 10 (the end on the optical device 3 side).

[0047] The second object O2 may be located farther from the optical device 3 than the aerial image R. The second object O2 may be the face of the user 10 viewing the aerial image R, particularly the eye 10a of the user 10. The imaging unit 5 generates a captured image (hereinafter also referred to as a second reflected image) based on the second reflected light M2. The second reflected image may include an image of the face of the user 10, particularly an image of the eye 10a of the user 10.

[0048] The first reflected light M1 is reflected light reflected by the first object O1 that is in contact with the aerial image R (i.e., that is located near the imaging plane P), and therefore the first reflected light M1 may be brighter than the second reflected light M2. The influence of the first reflected light M1 may be dominant in the captured image generated by the imaging unit 5. The captured image generated by the imaging unit 5 may be substantially the first reflected image.

[0049] (First Example) A first example of the operation of detecting a gesture operation in the aerial image display device 1 will be described. The image processing device 6 acquires luminance distribution data of the captured image (color image) from the imaging unit 5. The image processing device 6 converts the acquired luminance distribution data to grayscale. Grayscale conversion refers to converting the luminance distribution data of the color image into luminance distribution data in which the luminance value of each pixel is represented by black gradations, white gradations, and gray gradations between black and white gradations. The luminance value of each pixel in the converted luminance distribution data may be 8-bit data. In other words, the luminance value of each pixel in the converted luminance distribution data may be represented by a numerical value between 0 and 255.

[0050] The image processing device 6 may perform grayscale conversion on the luminance distribution data acquired from the imaging unit 5 by calculating the average values ​​of the R component value, the G component value, and the B component value for each pixel of the captured image. The image processing device 6 may perform grayscale conversion on the luminance distribution data acquired from the imaging unit 5 by converting the R component value, the G component value, and the B component value for each pixel of the captured image into the Y component value in the CIE XYZ space. The image processing device 6 may perform grayscale conversion on the luminance distribution data acquired from the imaging unit 5 by converting the R component value, the G component value, and the B component value for each pixel of the captured image into the Y component value in the YCbCr color space. The grayscale conversion is not limited to the above example and may be performed by any known method.

[0051] The image processing device 6 may detect the number and positions of first objects (for example, fingers 10b) in contact with the aerial image R based on the grayscale converted brightness distribution data.

[0052] The captured image generated by the imaging unit 5 includes a first reflected image and a second reflected image. The first reflected image is a reflected image generated based on the first reflected light M1, and the second reflected image is a reflected image generated based on the second reflected light M2. Since the aerial image display device 1 is configured such that the imaging unit 5 is located at the confocal position of the optical device 3, the first reflected image may include a clear image of the first object and a slightly blurred image of the second object. The second reflected image may include a clear image of the first object and a clear image of the second object. The same applies to captured images that have been converted to grayscale.

[0053] The image processing device 6 performs edge enhancement processing in the width direction on the grayscale-converted captured image (grayscale-converted brightness distribution data) to generate an image (hereinafter also referred to as a first edge-enhanced image) in which the edge of the first object O1 (the outline of the finger 10b) included in the first reflection image is enhanced. The width direction refers to the width direction of the captured image, and may correspond to the X-axis direction shown in the drawings. The image processing device 6 can detect the position of the first object O1 (finger 10b) based on the first edge-enhanced image. The edge enhancement processing can be performed using, for example, a Sobel filter, a Prewitt filter, or the like. To extract the first object O1 from the first edge-enhanced image, a region segmentation method such as a region merging method may be used.

[0054] The image processing device 6 may detect the fingerprint of the finger 10b, which is the first object O1, based on the first edge-enhanced image and use the detected fingerprint for fingerprint authentication. In this case, it is not necessary to detect the entire fingerprint of the finger 10b; instead, a portion of the fingerprint of the finger 10b corresponding to the edge image of the first edge-enhanced image may be detected. In this case, to improve the accuracy of authentication using the portion of the fingerprint of the finger 10b, similarity determination, classification, prediction, etc. may be performed using artificial intelligence (AI) program software. The similarity determination is, for example, a function of determining to what extent a portion of the fingerprint of the finger 10b of the user 10 matches the registered fingerprint data of the finger 10b of the user 10. The classification is, for example, a function of determining whether a portion of the fingerprint of the finger 10b of the user 10 belongs to a specific classification (e.g., a whorl-type fingerprint). The prediction is a function of predicting a portion of the fingerprint of the finger 10b of the user 10 into a more expanded version.

[0055] The image processing device 6 detects the fingerprint of the finger 10b, which is the first object O1, based on the first edge-enhanced image and performs fingerprint authentication. The control device 7 may be configured to control the aerial image R based on one or both of the individual differences (e.g., fingerprint) of the finger 10b, which is the first object O1, and the individual differences (e.g., iris) of the eye 10a, which is the second object. For example, the aerial image R may be displayed for a specific user 10 authorized by fingerprint authentication. In this case, the aerial image R may be a unique image including personal information (e.g., the user's 10 age, address, nationality, race, job description, health information, and asset information). The control device 7 may also start (activate) the display of the aerial image R for the specific user 10 authorized by fingerprint authentication, change the aerial image R from a still image to a video, or increase the brightness of the aerial image R. The control device 7 may also emit a voice message confirming the name of the specific user 10 authorized by fingerprint authentication, a voice message guiding the operation, or the like.

[0056] The individual difference of the finger 10b, which is the first object O1, may be at least a part of the fingerprint, at least a part of the shape of the finger, at least a part of the size of the finger, at least a part of the line pattern on the palm, etc., or may be a combination of two or more of these. The individual difference of the eye 10a, which is the second object, may be at least a part of the iris, at least a part of the shape of the eye, at least a part of the size of the eye, the interocular distance, at least a part of the blood vessel pattern in the white of the eye, etc., or may be a combination of two or more of these.

[0057] When detecting the fingerprint of the finger 10b and performing fingerprint authentication, the imaging unit 5 may receive ambient light reflected by the finger 10b. Furthermore, the imaging light L from the optical device 3 may be white light (image light). In this case, the brightness of the reflected light of the imaging light L reflected by the finger 10b can be increased. This white light may be a portion of the imaging light L. For example, the finger 10b may be brought into contact with a portion of the imaging light L (the white light portion).

[0058] The image processing device 6 performs edge enhancement processing in the height direction on the grayscale-converted captured image (grayscale-converted luminance distribution data) to generate an image (hereinafter also referred to as a second edge-enhanced image) in which the edges of the first object O1 (the outline of the fingers 10b) and the second object O2 (the outline of the eye 10a) included in the second reflection image are enhanced. The height direction refers to the height direction in the captured image, and may correspond to the Y-axis direction shown in the drawings. The image processing device 6 can detect the position of the second object O2 (the eye 10a) based on the second edge-enhanced image. The edge enhancement processing can be performed using, for example, a Sobel filter, a Prewitt filter, or the like. Furthermore, a region segmentation method such as a region integration method may be used to extract the second object O2 from the second edge-enhanced image.

[0059] The image processing device 6 may detect the position, line of sight, gaze point, etc. of the eye 10a of the user 10 based on the second edge-enhanced image. When the control device 7 determines that the user 10 is not gazing at the aerial image R based on the position, line of sight, gaze point, etc. of the eye 10a of the user 10 detected by the image processing device 6, the control device 7 may reduce the brightness of the aerial image R (the brightness of the image displayed on the display surface 4a) or temporarily stop displaying the aerial image R (the display of the image on the display surface 4a). In this case, the power consumption of the aerial image display device 1 can be reduced. Furthermore, if the aerial image R contains highly confidential information such as personal information, the risk of the aerial image R being viewed by a third party other than the user 10 can be reduced. The aerial image display device 1 may detect facial features of the user 10 based on the second edge-enhanced image. The aerial image display device 1 may authenticate the user 10 based on the detection results of the facial features of the user 10.

[0060] The image processing device 6 may detect the iris (the area surrounding the pupil) of the eye 10a, which is the second object O2, based on the second edge-enhanced image and use it for iris authentication. In this case, it is not necessary to detect the entire iris; instead, a portion of the iris corresponding to the edge image of the second edge-enhanced image may be detected. In this case, similarity judgment, classification, prediction, etc. may be performed using AI program software to improve the accuracy of authentication using the portion of the iris. Similarity judgment is, for example, a function of determining to what extent a portion of the iris matches the iris data of the registered eye 10a of the user 10. Classification is, for example, a function of determining whether a portion of the iris of the eye 10a of the user 10 belongs to a specific classification (e.g., a specific pattern). Prediction is a function of predicting a portion of the iris of the eye 10a of the user 10 in a more expanded form.

[0061] The image processing device 6 detects the iris of the eye 10a, which is the second object O2, based on the second edge-enhanced image and performs iris authentication. The control device 7 may, for example, display an aerial image R to a specific user 10 authorized through iris authentication. In this case, the aerial image R may be a unique image including personal information (e.g., the user's 10 age, address, nationality, race, job description, health information, and asset information). The control device 7 may also start (activate) the display of the aerial image R for the specific user 10 authorized through iris authentication, change the aerial image R from a still image to a video, or increase the brightness of the aerial image R. The control device 7 may also issue a voice message confirming the name of the specific user 10 authorized through iris authentication, a voice message guiding the operation, or the like.

[0062] The image processing device 6 may detect the number of fingers 10b in contact with the aerial image R based on the first edge-enhanced image. The image processing device 6 may determine that a connected region (a set of connected pixels having a luminance value exceeding a predetermined threshold T1) included in the first edge-enhanced image corresponds to one finger 10b in contact with the aerial image R. The image processing device 6 may generate a plurality of temporally consecutive first edge-enhanced images, and track the position of the finger 10b based on the generated plurality of first edge-enhanced images.

[0063] When the number of fingers 10b in contact with the aerial image R is one, the control device 7 may determine that the user 10 is tapping, swiping, or flicking the aerial image R. The control device 7 may control the display device 2 to rotate or move the aerial image R based on the tracking result of the single finger 10b.

[0064] If the control device 7 determines that the number of fingers 10b in contact with the aerial image R is one, it may further determine which finger (e.g., the index finger) that finger is. Generally, when a person operates a screen with a finger, it is highly likely that the index finger is used. Therefore, if the control device 7 determines that the index finger is used, it can be predicted that the user 10 is attempting to operate the screen. If the control device 7 determines that the index finger is not used, it can be predicted that the user 10 is not attempting to operate the screen, but is instead attempting, for example, fingerprint authentication. In this case, to improve the accuracy of determining which finger 10b is used, a similarity determination may be performed using AI program software. The similarity determination is, for example, a function that determines to what extent each of the fingers 10b of the user 10 matches the data of each of the registered fingers 10b of the user 10.

[0065] When the number of fingers 10b in contact with the aerial image R is two, the control device 7 may determine that the user 10 is pinching out or in on the aerial image R. The control device 7 may control the display device 2 to enlarge or reduce the aerial image R based on the tracking result of the two fingers 10b.

[0066] Next, detection of a gesture operation will be described using several examples of the first edge-enhanced image. FIG. 3 is a diagram illustrating an example of the position of the fingers 10b relative to the aerial image R. FIG. 3 corresponds to a case where the user 10 uses one finger 10b to tap, swipe, or flick the aerial image R (hereinafter, also referred to as a content operation). In FIG. 3, "101" indicates a case where the finger 10b is farther away from the optical device 3 than the aerial image R, "102" indicates a case where the finger 10b is in contact with the aerial image R, and "103" indicates a case where the finger 10b is closer to the optical device 3 than the aerial image R.

[0067] Fig. 4 shows, from the left, a first edge-enhanced image when finger 10b is in the position indicated by "101" in Fig. 3, a first edge-enhanced image when finger 10b is in the position indicated by "102" in Fig. 3, and a first edge-enhanced image when finger 10b is in the position indicated by "103" in Fig. 3. As shown in Fig. 4, when finger 10b is in the position indicated by "102" in Fig. 3, the first edge-enhanced image includes one connected region (a set of a series of pixels that are connected to each other) 10r representing one finger 10b, and when finger 10b is in the position indicated by "101" or "103" in Fig. 3, the first edge-enhanced image does not include connected region 10r. Therefore, based on the first edge-enhanced image, the image processing device 6 can determine the number of fingers 10b in contact with the aerial image R (the number of connected regions 10r included in the first edge-enhanced image) and the positions of the fingers 10b in contact with the aerial image R (the positions of the connected regions 10r in the first edge-enhanced image).

[0068] Fig. 5 is a diagram illustrating an example of the position of the fingers 10b relative to the aerial image R. Fig. 5 corresponds to a case where the user 10 pinches out or in on the aerial image R using two fingers 10b. In Fig. 5, "201" indicates a case where the two fingers 10b are farther away from the optical device 3 than the aerial image R, "202" indicates a case where the two fingers 10b are in contact with the aerial image R, and "203" indicates a case where the two fingers 10b are closer to the optical device 3 than the aerial image R.

[0069] Fig. 6 shows, from the left, a first edge-enhanced image when the two fingers 10b are in the position indicated by "201" in Fig. 5, a first edge-enhanced image when the two fingers 10b are in the position indicated by "202" in Fig. 5, and a first edge-enhanced image when the two fingers 10b are in the position indicated by "203" in Fig. 5. As shown in Fig. 6, when the two fingers 10b are in the position indicated by "202" in Fig. 5, the first edge-enhanced image includes two connected regions 10r representing the two fingers 10b, and when the two fingers 10b are in the position indicated by "201" or "203" in Fig. 5, the first edge-enhanced image does not include the connected region 10r. Therefore, based on the first edge-enhanced image, the image processing device 6 can determine the number of fingers 10b in contact with the aerial image R (the number of connected regions 10r included in the first edge-enhanced image) and the positions of the fingers 10b in contact with the aerial image R (the positions of the connected regions 10r in the first edge-enhanced image).

[0070] When determining the number of connected regions 10r included in the first edge-enhanced image, the image processing device 6 may exclude connected regions 10r having a pixel count less than a predetermined threshold T2 from the determination of the number of connected regions 10r. This makes it possible to eliminate connected regions caused by noise such as external light, and to accurately determine the number of fingers 10b in contact with the aerial image R. As a result, it is possible to accurately detect the gesture operation of the user 10 on the aerial image R. The threshold T2 may be set as appropriate depending on the required specifications of the aerial image display device 1, etc.

[0071] The control device 7 controls the image processing device 6 to generate a plurality of temporally consecutive first edge-enhanced images, determine the number and positions of fingers 10b in contact with the aerial image R based on the generated plurality of first edge-enhanced images, and sequentially output the determination results. The control device 7 sequentially acquires the determination results from the image processing device 6, and determines the content of the gesture operation of the user 10 on the aerial image R (whether it is a content operation or a pinch out or pinch in) based on the acquired series of determination results. The control device 7 rotates, moves, enlarges, or reduces the aerial image R based on the determination result of the content of the gesture operation.

[0072] 7 is a flowchart illustrating a first example of the operation of detecting a gesture operation. This flowchart is executed by the control device 7, starts when the aerial image display device 1 forms an aerial image R, and is repeatedly executed while the aerial image display device 1 forms the aerial image R. Steps S1 to S7 are operation or determination steps, and in the following description, step S1 will be referred to as S1, for example.

[0073] In [S1], the image processing device 6 acquires luminance distribution data of a reflected image from the imaging unit 5. The reflected image is an image generated based on the first reflected light and the second reflected light.

[0074] In [S2], the image processing device 6 converts the luminance distribution data acquired from the imaging unit 5 into grayscale.

[0075] In [S3], the image processing device 6 performs edge enhancement processing in the width direction on the grayscale converted luminance distribution data to generate a first edge-enhanced image.

[0076] In [S4], the image processing device 6 determines, based on the first edge-enhanced image, whether or not the number of fingers 10b in contact with the aerial image R is one or more. If the number of fingers 10b in contact with the aerial image R is one or more [Yes], the process proceeds to [S5]. If the number of fingers 10b in contact with the aerial image R is less than one [No], the process ends this flowchart.

[0077] In [S5], the image processing device 6 determines, based on the first edge-enhanced image, whether or not the number of fingers 10b in contact with the aerial image R is two or more. If the number of fingers 10b in contact with the aerial image R is two or more [Yes], the process proceeds to [S6]. If the number of fingers 10b in contact with the aerial image R is less than two [No], the process proceeds to [S7]. In [S5], the image processing device 6 determines the position of one finger 10b and the positions of two or more fingers 10b based on the first edge-enhanced image.

[0078] In [S6], the control device 7 determines that the gesture operation of the user 10 on the aerial image R is a pinch out or a pinch in. The control device 7 also acquires the positions of two or more fingers 10b from the image processing device 6 and stores the positions of the two or more fingers 10b in memory. If the positions of two or more fingers 10b have already been stored in memory, the control device 7 compares the positions of the two or more fingers 10b acquired from the image processing device 6 with the positions of the two or more fingers 10b stored in memory, and determines whether the gesture operation of the user 10 on the aerial image R is a pinch out or a pinch in. The control device 7 enlarges or reduces the aerial image R based on the determination result of the gesture operation.

[0079] In [S7], the control device 7 determines that the gesture operation of the user 10 on the aerial image R is a content operation. The control device 7 also acquires the position of one finger 10b from the image processing device 6 and stores the position of one finger 10b in memory. If the position of one finger 10b has already been stored in memory, the control device 7 compares the position of one finger 10b acquired from the image processing device 6 with the position of one finger 10b stored in memory, and determines whether the gesture operation of the user 10 on the aerial image R is a rotation or movement of the aerial image R. The control device 7 rotates or moves the aerial image R based on the determination result of the gesture operation.

[0080] Second Example Next, a second example of the operation of detecting a gesture operation in the aerial image display device 1 will be described. The image processing device 6 acquires luminance distribution data of the captured image (color image) from the imaging unit 5. The image processing device 6 determines that data included in the luminance distribution data, whose luminance value is equal to or greater than a predetermined threshold T3, is data indicating fingers 10b in contact with the aerial image R. In making this determination, the image processing device 6 may convert the luminance distribution data acquired from the imaging unit 5 to grayscale and compare the luminance value of the grayscale-converted luminance distribution data with the threshold T3. The image processing device 6 may perform edge enhancement processing in the width direction on the luminance distribution data acquired from the imaging unit 5 to generate a first edge-enhanced image, and determine that data included in the luminance distribution data of the first edge-enhanced image, whose luminance value is equal to or greater than the threshold T3, is data indicating fingers 10b in contact with the aerial image R.

[0081] The image processing device 6 may determine the number of fingers 10b in contact with the aerial image R based on the luminance distribution data acquired from the imaging unit 5. The image processing device 6 may determine that a connected region (a set of connected pixels having a luminance value exceeding the threshold T3) having a luminance value equal to or greater than a threshold T3 corresponds to one finger 10b in contact with the aerial image R. The image processing device 6 may track the position of the finger 10b based on a plurality of pieces of temporally consecutive luminance distribution data.

[0082] When the brightness value of the captured image is expressed as a numerical value between 0 and 255, the threshold value T3 may be, for example, 32, 64, or 128. The image processing device 6 may exclude connected regions whose pixel count is less than a predetermined threshold value T4 from the determination of the number of connected regions. In this case, connected regions caused by noise such as external light can be excluded, and the number of fingers 10b in contact with the aerial image R can be accurately determined. As a result, the gesture operation of the user 10 on the aerial image R can be accurately detected.

[0083] When the number of fingers 10b in contact with the aerial image R is one, the control device 7 may determine that the gesture operation of the user 10 is a content operation. The control device 7 may control the display device 2 to rotate or move the aerial image R based on the tracking result of the single finger 10b.

[0084] At this time, as described above, if the control device 7 determines that the number of fingers 10b in contact with the aerial image R is one, it may further determine which finger that one finger is (for example, the index finger).

[0085] The control device 7 may determine that the gesture operation of the user 10 is a pinch out or a pinch in when two or more fingers 10b are in contact with the aerial image R. The control device 7 may control the display device 2 to enlarge or reduce the aerial image R based on the tracking result of the two or more fingers 10b.

[0086] If the control device 7 determines that two or more fingers 10b are in contact with the aerial image R, it may further determine which fingers (e.g., index fingers and thumbs) those fingers are. Generally, when a person pinches out or in on a screen, it is highly likely that they are pinched out or in using their index fingers and thumbs. Therefore, if the control device 7 determines that the fingers are the index fingers and thumbs, it can be predicted that the user 10 is attempting to perform a pinch out or pinch in screen operation. If the control device 7 determines that the fingers are not the index fingers and thumbs, it can be predicted that the user 10 is not attempting a pinch out or pinch in screen operation, but is instead attempting, for example, content operation or fingerprint authentication. In this case, to improve the accuracy of determining which fingers 10b are the two or more fingers 10b, a similarity determination may be performed using AI program software. The similarity determination is, for example, a function that determines to what extent each of the fingers 10b of the user 10 matches the data of each of the registered fingers 10b of the user 10.

[0087] Next, detection of gesture operations will be described using several examples of captured images. Fig. 8 shows, from the left, a captured image when the finger 10b is at the position indicated by "101" in Fig. 3 , a captured image when the finger 10b is at the position indicated by "102" in Fig. 3 , and a captured image when the finger 10b is at the position indicated by "103" in Fig. 3 . Fig. 8 shows an image represented by luminance distribution data obtained by grayscale conversion of luminance distribution data acquired from the imaging unit 5. As shown in Fig. 8 , when the finger 10b is at the position indicated by "102" in Fig. 3 , the captured image includes one connected region 10r representing one finger 10b, whereas when the finger 10b is at the position indicated by "101" or "103" in Fig. 3 , the captured image does not include the connected region 10r. Therefore, based on the first edge-enhanced image, the image processing device 6 can determine the number of fingers 10b in contact with the aerial image R (the number of connected areas 10r included in the captured image) and the positions of the fingers 10b in contact with the aerial image R (the positions of the connected areas 10r in the captured image).

[0088] 9 shows, from left to right, a captured image when two fingers 10b are in the position indicated by "201" in FIG. 5, a captured image when two fingers 10b are in the position indicated by "202" in FIG. 5, and a captured image when two fingers 10b are in the position indicated by "203" in FIG. 5. As shown in FIG. 9, when two fingers 10b are in the position indicated by "202" in FIG. 5, the captured image includes two connected regions 10r representing the two fingers 10b, whereas when two fingers 10b are in the positions indicated by "201" or "203" in FIG. 5, the captured image does not include the connected regions 10r. Therefore, based on the captured images, the image processing device 6 can determine the number of fingers 10b in contact with the aerial image R (the number of connected regions 10r included in the captured image) and the positions of the fingers 10b in contact with the aerial image R (the positions of the connected regions 10r in the captured image). Note that the captured image on the left and the captured image on the right in Figure 9 contain connected regions with higher brightness values ​​than the surrounding pixels, but because the brightness values ​​of the connected regions are lower than the threshold value, they are not determined to be connected regions representing fingers 10b in contact with the aerial image R.

[0089] 10 is a flowchart illustrating a second example of the gesture operation detection operation. This flowchart is executed by the control device 7, starts when the aerial image display device 1 forms an aerial image R, and is repeatedly executed while the aerial image display device 1 is forming the aerial image R.

[0090] In [S11], the image processing device 6 acquires luminance distribution data of the captured image from the imaging unit 5. The reflected image is an image generated based on the first reflected light and the second reflected light. The image processing device 6 may perform grayscale conversion on the luminance distribution data acquired from the imaging unit 5. The image processing device 6 may also perform edge enhancement processing in the width direction on the luminance distribution data acquired from the imaging unit 5.

[0091] In [S12], the image processing device 6 determines, based on the brightness distribution data, whether the number of fingers 10b in contact with the aerial image R (a set of connected pixels having a brightness value exceeding the threshold T2) is one or more. If the number of fingers 10b in contact with the aerial image R is one or more [Yes], the process proceeds to [S13]. If the number of fingers 10b in contact with the aerial image R is less than one [No], the process ends this flowchart.

[0092] In [S13], the image processing device 6 determines, based on the luminance distribution data, whether or not the number of fingers 10b in contact with the aerial image R is two or more. If the number of fingers 10b in contact with the aerial image R is two or more [Yes], the process proceeds to [S14], and if the number of fingers 10b in contact with the aerial image R is less than two [No], the process proceeds to [S15]. In [S13], the image processing device 6 determines the position of one finger 10b and the positions of two or more fingers 10b based on the luminance distribution data.

[0093] In [S14], the control device 7 determines that the gesture operation of the user 10 on the aerial image R is a pinch out or a pinch in. The control device 7 also acquires the positions of two or more fingers 10b from the image processing device 6 and stores the positions of the two or more fingers 10b in memory. If the positions of two or more fingers 10b have already been stored in memory, the control device 7 compares the positions of the two or more fingers 10b acquired from the image processing device 6 with the positions of the two or more fingers 10b stored in memory, and determines whether the gesture operation of the user 10 on the aerial image R is a pinch out or a pinch in. The control device 7 enlarges or reduces the aerial image R based on the determination result of the gesture operation.

[0094] In [S15], the control device 7 determines that the gesture operation of the user 10 on the aerial image R is a content operation. The control device 7 also acquires the position of one finger 10b from the image processing device 6 and stores the position of one finger 10b in memory. If the position of one finger 10b has already been stored in memory, the control device 7 compares the position of one finger 10b acquired from the image processing device 6 with the position of one finger 10b stored in memory, and determines whether the gesture operation of the user 10 on the aerial image R is a rotation or movement of the aerial image R. The control device 7 rotates or moves the aerial image R based on the determination result of the gesture operation.

[0095] The aerial image display device 1 may detect both the gesture operation described in the flowchart of Fig. 7 and the gesture operation described in the flowchart of Fig. 10. In this case, it is possible to more accurately detect the gesture operation of the user 10 with respect to the aerial image R.

[0096] The image processing device 6 may be configured to detect a gesture of the fingers 10b, which are the first object O1, based on the first edge-enhanced image, etc., and to detect the gaze direction and blinking of the eyes 10a, which are the second object O2, based on the second edge-enhanced image, etc. (so-called eye tracking). The control device 7 may be configured to enable highly accurate character input, button operation, etc., using the gestures and eye tracking of the fingers 10b. For example, if one of the eyes 10a is closed during a character input operation using the fingers 10b, the image processing device 6 may recognize that the desired character has not been input, and the control device 7 may undo the character input operation. At this time, the character input operation may be undoed by one character or by one fraction of a character. For example, if one of the eyes 10a is closed during a button operation using the fingers 10b, the image processing device 6 may recognize that the desired button operation has not been performed, and the control device 7 may undo the button operation. At this time, all of the button operations may be undone, or one of the button operations may be undone. This button operation may also include keyboard operations.

[0097] Furthermore, for example, during a character input operation using a gesture of the fingers 10b, the viewpoints of both eyes 10a on the aerial image R may be displayed as points on the aerial image R to assist in character input. For example, during a character input operation using a gesture of the fingers 10b, a tag for character correction, such as [NO], [Correction], [Re-input], or [Undo], may be associated with the tip of a drawn line of a character being input in the middle of input, and when the viewpoint is placed on the tag, the image processing device 6 recognizes that the desired character has not been input, and the control device 7 restores the character input. Furthermore, for example, during a button operation using a gesture of the fingers 10b, the viewpoints of both eyes 10a on the aerial image R may be displayed as points on the aerial image R to assist in button operation. For example, during a button operation using a gesture of the fingers 10b, the operated button may be displayed on the aerial image R, and a tag for button correction such as [NO], [Correction], [Re-input], or [Undo] may be associated with the button, and when the viewpoint is placed on the tag, the image processing device 6 may recognize that the desired button operation has not been performed, and the control device 7 may undo the button operation. This button operation may include keyboard operation.

[0098] Next, an aerial image display device according to another embodiment of the present disclosure will be described. Fig. 11 is a diagram showing an aerial image display device according to another embodiment of the present disclosure, and Fig. 12 is a diagram showing another example of the aerial image display device of Fig. 11. An aerial image display device 1A of this embodiment differs from the aerial image display device 1 of the above embodiment in that it includes an imaging device and a reflective polarizing plate, but otherwise has the same configuration, so detailed description of the same configuration will be omitted.

[0099] 11 and 12 , the aerial image display device 1A of this embodiment includes a display device 2, an optical device 3, an imaging device 8, and a reflective polarizing plate 9. The aerial image display device 1A further includes an image processing device 6 and a control device 7.

[0100] The display device 2 includes a display unit 4. The display unit 4 may be a transmissive display panel or a self-luminous display panel. The display device 2 does not need to include an imaging unit 5.

[0101] The optical device 3 may be composed of a first concave mirror 31, a convex mirror 32, and a second concave mirror 33, as shown in Figure 11, or may be composed of a first concave mirror 31 and a second concave mirror 33, as shown in Figure 12.

[0102] The imaging device 8 is located at the confocal position of the optical device 3. The imaging device 8 may have a configuration similar to that of the imaging unit 5 of the aerial image display device 1, except that the imaging device 8 is not located behind the display unit 4. The imaging device 8 may have a polarizing plate 81 located on the imaging surface. The polarizing plate 81 may reduce the direct incidence of image light L0 emitted from the display surface 4 a on the imaging device 8.

[0103] 11 and 12 , the reflective polarizing plate 9 is located in front of the display surface 4 a of the display device 2 and in front of the imaging surface of the imaging device 8. The reflective polarizing plate 9 may be composed of, for example, a half mirror, a beam splitter, or the like. The reflective polarizing plate 9 transmits a portion of the image light L0 emitted from the display surface 4 a. The reflective polarizing plate 9 reflects a portion of the first reflected light M1 reflected by the first concave mirror 31 toward the imaging surface of the imaging device 8, and reflects a portion of the second reflected light M2 reflected by the first concave mirror 31 toward the imaging surface of the imaging device 8.

[0104] The operation of the aerial image display device 1A is similar to that of the aerial image display device 1. Like the aerial image display device 1, the aerial image display device 1A can accurately detect gesture operations of the user 10 with respect to the aerial image R. Furthermore, like the aerial image display device 1, the aerial image display device 1A can detect the position, line of sight, point of gaze, etc. of the eye 10a of the user 10, and can reduce the power consumption of the aerial image display device 1A based on the detected position, line of sight, point of gaze, etc. of the eye 10a of the user 10.

[0105] The aerial image display device 1A is not configured so that the imaging device 8 receives light (first reflected light and second reflected light) that has passed through the display unit 4, and therefore it is possible to increase the pixel density of the display unit 4. As a result, it is possible to improve the resolution of the aerial image R.

[0106] In the aerial image display device 1A, the optical path length from the display device 2 to the reflective polarizing plate 9 can be made to coincide with the optical path length from the reflective polarizing plate 9 to the image capturing device 8, and therefore the image capturing device 8 can be disposed at the confocal position of the optical device 3. As a result, the detection position accuracy of the finger 10b can be improved, and the gesture operation of the user 10 relative to the aerial image R can be detected with high accuracy.

[0107] Next, an aerial image display device according to another embodiment of the present disclosure will be described. Fig. 13 is a diagram showing an aerial image display device according to another embodiment of the present disclosure, and Fig. 14 is a plan view of a display device included in the aerial image display device of Fig. 13. Fig. 14 is a plan view of the display device as viewed along the emission direction of image light, with the liquid crystal panel omitted. In Fig. 14, the infrared light source is hatched for ease of illustration. The aerial image display device 1B of this embodiment differs from the aerial image display device 1A of the above embodiment in the configuration of the display device, but is otherwise similar in configuration, and therefore detailed description of the similar configuration will be omitted.

[0108] 13 , the aerial image display device 1B of this embodiment includes a display device 2, an optical device 3, an imaging device 8, and a reflective polarizing plate 9. The aerial image display device 1B further includes an image processing device 6 and a control device 7.

[0109] As shown in Fig. 13, the display device 2 includes a display unit 4. The display unit 4 includes a liquid crystal panel 11 and a backlight 12. As shown in Fig. 14, the backlight 12 includes a substrate 13, a plurality of visible light sources 14, and a plurality of infrared light sources 15. The plurality of visible light sources 14 and the plurality of infrared light sources 15 are arranged in a matrix on the substrate 13.

[0110] Visible light source 14 may be configured to emit white light. Visible light source 14 may be composed of a red LED that emits red light, a green LED that emits green light, and a blue LED that emits blue light. Visible light source 14 may be composed of a white LED that emits white light. The white LED may be composed of an ultraviolet LED that emits ultraviolet light and a phosphor that converts the wavelength of the ultraviolet light emitted from the ultraviolet LED into white light. The white LED may be composed of a blue LED that emits blue light and a phosphor that converts the wavelength of the blue light emitted from the blue LED into white light.

[0111] The infrared light source 15 may be an infrared LED that emits infrared light. The infrared LED may be near-infrared light with a wavelength of approximately 0.78 μm to 2.5 μm or a near-infrared LED that emits infrared light. The infrared light emitted from the infrared light source 15 may be imaging light that is projected (irradiated) onto a first object (finger 10b) and a second object (eye 10a) to capture images of the first object and second object. The first reflected light M1 reflected by the first object and the second reflected light M2 reflected by the second object may include infrared light.

[0112] Liquid crystal panel 11 displays an image formed as aerial image R on display surface 4a using light emitted from multiple visible light sources 14. Liquid crystal panel 11 transmits infrared light emitted from multiple infrared light sources 15.

[0113] As shown in Figure 13, the optical device 3 may be composed of a first concave mirror 31, a convex mirror 32, and a second concave mirror 33, or may be composed of the first concave mirror 31 and the second concave mirror 33.

[0114] The imaging device 8 is composed of an infrared camera and receives the first reflected light M1 and the second reflected light M2. The imaging device 8 is located at the confocal position of the optical device 3. Except for being composed of an infrared camera, the imaging device 8 may have a configuration similar to that of the imaging unit 5 of the aerial image display device 1A. The imaging device 8 may have a polarizing plate 81 located on the imaging surface. The polarizing plate 81 may reduce the amount of infrared light emitted from the infrared light source 15 of the backlight 12 that is directly incident on the imaging device 8.

[0115] The operation of the aerial image display device 1B is similar to that of the aerial image display devices 1 and 1A. Like the aerial image display devices 1 and 1A, the aerial image display device 1B can accurately detect gesture operations of the user 10 with respect to the aerial image R. Furthermore, like the aerial image display devices 1 and 1A, the aerial image display device 1B can detect the position, line of sight, point of gaze, etc. of the eye 10a of the user 10, and can reduce power consumption of the aerial image display device 1B based on the detected position, line of sight, point of gaze, etc. of the eye 10a of the user 10.

[0116] In the aerial image display device 1B, the imaging device 8 is not configured to receive light (first reflected light and second reflected light) that has passed through the display unit 4, and therefore the pixel density of the display unit 4 can be increased, thereby improving the resolution of the aerial image R.

[0117] In the aerial image display device 1B, the optical path length from the display device 2 to the reflective polarizing plate 9 can be made to coincide with the optical path length from the reflective polarizing plate 9 to the image capturing device 8, and therefore the image capturing device 8 can be disposed at the confocal position of the optical device 3. As a result, the detection position accuracy of the finger 10b can be improved, and the gesture operation of the user 10 relative to the aerial image R can be detected with high accuracy.

[0118] The aerial image display device 1B captures images of the first object and the second object using an infrared camera, which reduces the risk that the color, pattern, etc. of the aerial image R or the rotation, movement, enlargement, reduction, etc. of the aerial image R will affect the image captured by the imaging device 8. As a result, the gesture operation of the user 10 with respect to the aerial image R can be detected with high accuracy.

[0119] Next, another example of the aerial image display device 1 (see Figures 1 and 2) will be described. Figures 15 and 16 are plan views showing another example of the display device 2 provided in the aerial image display device 1 of Figure 1. Figures 15 and 16 are plan views of the display device 2 viewed along the emission direction of image light, and the liquid crystal panel 11 is omitted from the illustration. In Figures 15 and 16, the image sensor is hatched to facilitate illustration.

[0120] In the aerial image display device 1, the display device 2 does not have to be composed of a display unit 4 (transparent display) and an imaging unit 5 located behind the display unit 4. As shown in Fig. 15, the display device 2 may have a configuration in which a plurality of visible light emitting elements 17 and a plurality of imaging elements 18 are arranged in a matrix on a substrate 16. The visible light emitting elements 17 may be LEDs, organic EL elements, inorganic EL elements, etc. Each of the plurality of visible light emitting elements 17 may be composed of a red light emitting element 17r, a green light emitting element 17g, and a blue light emitting element 17b. The imaging element 18 may be a CCD imaging element or a CMOS imaging element.

[0121] The visible light emitting element 17 and the image sensor 18 do not have to be configured so that the red light emitting element 17r, the green light emitting element 17g, the blue light emitting element 17b, and the image sensor 18 are aligned in a predetermined direction. The visible light emitting element and the image sensor may be configured so that the red light emitting element 17r, the green light emitting element 17g, the blue light emitting element 17b, and the image sensor 18 are arranged in a matrix, as shown in Fig. 16 .

[0122] The optical device of the present disclosure makes it possible to focus and form floating images at different positions in the optical axis direction. Furthermore, the floating image display device of the present disclosure makes it possible to form floating images at different positions in the optical axis direction and to capture and detect multiple objects at different positions in the optical axis direction. Furthermore, the floating image display device of the present disclosure makes it possible to accurately detect gesture operations by a user on a floating image.

[0123] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the above-described embodiments, and various modifications, improvements, etc. are possible within the scope that does not deviate from the gist of the present disclosure.

[0124] The present disclosure can be implemented in the following configurations (1) to (20).

[0125] (1) An optical device that generates light including a first light and a second light having different focal depths.

[0126] (2) The optical device according to the above configuration (1), wherein the first focal length of the first light and the second focal length of the second light are different.

[0127] (3) The optical device according to the above configuration (1) or (2), comprising a reflective optical element having an anisotropic curvature of a reflective surface.

[0128] (4) A floating image display device comprising the optical device according to any one of the above configurations (1) to (3), wherein the optical device comprises: a display device; and an optical element that generates imaging light including the first light and the second light having different focal depths based on light from the display device, and forms a floating image using one or both of the first light and the second light.

[0129] (5) The floating image display device according to the above configuration (4), wherein the first focal length of the first light and the second focal length of the second light are different.

[0130] (6) The floating image display device according to the above configuration (4) or (5), wherein the optical element includes a free-form reflecting mirror.

[0131] (7) The display device includes the display unit and an imaging unit, and the imaging unit receives one or both of a first reflected light obtained by reflecting the first light by a first object located in a projection range of the imaging light, and a second reflected light obtained by reflecting the second light by a second object located in the projection range. This is a floating image display device described in any one of the above configurations (4) to (6).

[0132] (8) The floating image display device according to the above configuration (7), wherein the focal depth of the first light is smaller than the focal depth of the second light, the first object is in contact with the floating image, and the second object is farther away from the optical device than the floating image.

[0133] (9) The floating image display device according to the above configuration (8), wherein the first object is a finger of a user, and the second object is an eye of the user.

[0134] (10) The floating image display device according to any one of the above configurations (7) to (9), further comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and converts the luminance distribution data into grayscale.

[0135] (11) The floating image display device according to any one of the above configurations (7) to (9), further comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and determines data included in the luminance distribution data whose luminance value is equal to or greater than a predetermined threshold value as data representing the first object.

[0136] (12) A floating image display device according to any one of the above configurations (4) to (6), comprising: a reflective polarizing plate located on the optical path of the imaging light between the display device and the optical element; and an imaging device that receives the light reflected by the reflective polarizing plate, wherein the imaging unit is located at a confocal position of the optical device, and the optical path from the imaging unit to the floating image is different from the optical path from the display device to the floating image.

[0137] (13) The floating image display device described in the above configuration (12), wherein the imaging unit receives first reflected light, which is the first light reflected by a first object located in the projection range of the imaging light, and second reflected light, which is the second light reflected by a second object located in the projection range.

[0138] (14) The floating image display device according to the above configuration (13), wherein the focal depth of the first light is smaller than the focal depth of the second light, the first object is in contact with the floating image, and the second object is farther away from the optical device than the floating image.

[0139] (15) The floating image display device according to the above configuration (13) or (14), wherein the first object is a finger of a user, and the second object is an eye of the user.

[0140] (16) The floating image display device according to any one of the above configurations (13) to (15), further comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and converts the luminance distribution data into grayscale.

[0141] (17) A floating image display device according to any one of the above configurations (13) to (15), comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and determines data included in the luminance distribution data whose luminance value is equal to or greater than a predetermined threshold value as data indicating the first object.

[0142] (18) A floating image display device comprising: an optical device that generates imaging light including first light and second light having different focal depths, and forms a floating image using one or both of the first light and the second light; an imaging device that receives one or both of a first reflected light obtained by reflecting the first light by a first object located in a projection range of the imaging light, and a second reflected light obtained by reflecting the second light by a second object located in the projection range; and a control device that controls the floating image based on one or both of individual differences of the first object and individual differences of the second object.

[0143] (19) The floating image display device according to the above configuration (18), wherein the focal depth of the first light is smaller than the focal depth of the second light, the floating image is formed by the first light, the first object is in contact with the floating image, and the second object is farther away from the optical device than the floating image.

[0144] (20) The floating image display device according to the above configuration (18) or (19), wherein the first object is a finger of a user, and the second object is an eye of the user. Industrial application fields

[0145] The floating image display device of the present disclosure allows touchless operation of floating images, and as a result can be used in various product fields such as, but not limited to, a communication device for conversation, communication, etc., accompanied by a floating image, a medical interview device in which a doctor interviews a patient through a floating image, a navigation device and operation control device for vehicles such as automobiles, an order placement and reception device and cash register device for stores, etc., an operation panel for buildings, elevators, etc., a learning device for teaching or taking classes accompanied by a floating image, an office machine for business communication, instructions, etc. accompanied by a floating image, an amusement machine for playing games accompanied by a floating image, a projection device for projecting images onto the ground, walls, etc. at amusement parks, game centers, etc., a simulator device for conducting simulation experiments, etc., using floating images at universities, medical institutions, etc., a large display that displays prices, etc. at markets, stock exchanges, etc., and a video viewing device for viewing images of floating images.

[0146] 1, 1A, 1B Floating image display device (aerial image display device) 2 Display device 3 Optical device 4 Display unit 4a Display surface 5 Imaging unit 6 Image processing device 7 Control device 8 Imaging device 9 Reflective polarizing plate 10 User 10a Eye 10b Finger 10r Connecting region 11 Liquid crystal panel 12 Backlight 13 Substrate 14 Visible light source 15 Infrared light source 16 Substrate 17 Visible light emitting element 17b Blue light emitting element 17g Green light emitting element 17r Red light emitting element 18 Imaging element 31 First concave mirror 32 Convex mirror 33 Second concave mirror 81 Polarizing plate F Floating image L Light (image light) L0 Light (image light) L1 First light (first image light) L2 Second light (second image light) O Object O1 First object O2 Second object OD Optical element R Aerial image

Claims

1. An optical device that generates light including a first light and a second light having different focal depths.

2. The optical device of claim 1, wherein the first focal length of said first light and the second focal length of said second light are different.

3. The optical device according to claim 1 or 2, comprising a reflective optical element having an anisotropic degree of curvature of a reflective surface.

4. A floating image display device comprising the optical device according to any one of claims 1 to 3, wherein the optical device comprises: a display device; and an optical element that generates imaging light including the first light and the second light having different focal depths based on light from the display device, and forms a floating image using one or both of the first light and the second light.

5. The floating image display device according to claim 4, wherein the first focal length of the first light and the second focal length of the second light are different.

6. The floating image display device according to claim 4 or 5, wherein the optical element includes a free-form reflecting mirror.

7. A floating image display device according to any one of claims 4 to 6, wherein the display device includes the display unit and an imaging unit, and the imaging unit receives one or both of a first reflected light obtained by reflecting the first light by a first object located in the projection range of the imaging light, and a second reflected light obtained by reflecting the second light by a second object located in the projection range.

8. The floating image display device according to claim 7, wherein the focal depth of the first light is smaller than the focal depth of the second light, the first object is in contact with the floating image, and the second object is farther away from the optical device than the floating image.

9. The floating image display device according to claim 8, wherein the first object is a finger of the user, and the second object is an eye of the user.

10. A floating image display device as described in any one of claims 7 to 9, further comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and converts the luminance distribution data into grayscale.

11. A floating image display device as described in any one of claims 7 to 9, comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and determines that data included in the luminance distribution data having a luminance value equal to or greater than a predetermined threshold value is data indicative of the first object.

12. A floating image display device according to any one of claims 4 to 6, comprising: a reflective polarizing plate located on the optical path of the imaging light between the display device and the optical element; and an imaging device that receives the light reflected by the reflective polarizing plate, wherein the imaging unit is located at a confocal position of the optical device, and the optical path from the imaging unit to the floating image is different from the optical path from the display device to the floating image.

13. The floating image display device according to claim 12, wherein the imaging unit receives a first reflected light obtained by reflecting the first light by a first object located within the projection range of the imaging light, and a second reflected light obtained by reflecting the second light by a second object located within the projection range.

14. A floating image display device according to claim 13, wherein the focal depth of the first light is smaller than the focal depth of the second light, the first object is in contact with the floating image, and the second object is farther away from the optical device than the floating image.

15. The floating image display device according to claim 13 or 14, wherein the first object is a user's finger, and the second object is the user's eye.

16. A floating image display device as described in any one of claims 13 to 15, comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and converts the luminance distribution data into grayscale.

17. A floating image display device as described in any one of claims 13 to 15, comprising an image processing device, wherein the imaging unit generates a first reflected image based on the first reflected light, and the image processing device acquires luminance distribution data of the first reflected image from the imaging unit, and determines that data included in the luminance distribution data having a luminance value equal to or greater than a predetermined threshold value is data indicative of the first object.

18. A floating image display device comprising: an optical device that generates imaging light including a first light and a second light having different focal depths, and forms a floating image using one or both of the first light and the second light; an imaging device that receives one or both of a first reflected light obtained by reflecting the first light by a first object located in the projection range of the imaging light, and a second reflected light obtained by reflecting the second light by a second object located in the projection range; and a control device that controls the floating image based on one or both of an individual difference of the first object and an individual difference of the second object.

19. A floating image display device as described in claim 18, wherein the focal depth of the first light is smaller than the focal depth of the second light, the floating image is formed by the first light, the first object is in contact with the floating image, and the second object is farther away from the optical device than the floating image.

20. The floating image display device according to claim 18 or 19, wherein the first object is a user's finger, and the second object is the user's eye.

Images