Air floating image display apparatus and correction method

Abstract

To provide a more suitable air floating image display apparatus that contributes to "3 Good Health and Well-being," "9 Industry, Innovation and Infrastructure," and "11 Sustainable Cities and Communities" of the Sustainable Development Goals (SDGs).SOLUTION: An air floating image display apparatus comprises a correction unit that corrects a deviation between an operation target point in an in-plane direction of a screen and a detection point relating to the aerial operation of a user with respect to the screen of the air floating image. The correction unit corrects the deviation on the basis of the shape of a fingertip of a finger serving as an operation finger during the user's aerial operation.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present invention relates to a floating-in-the-air image display device. [Background technology]

[0002] The floating information display technology is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-128722 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the disclosure of Patent Document 1 does not sufficiently consider a configuration for obtaining practical brightness and quality for the floating image, or a configuration for allowing the user to view the floating image more enjoyably.

[0005] An object of the present invention is to provide a more suitable floating-in-the-air image display device. [Means for solving the problem]

[0006] In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problems, and one example thereof may be configured as follows: A floating-in-the-air image display device, comprising: an image processing unit that processes image; a display unit that displays image processed by the image processing unit; an optical system that generates a floating-in-the-air image based on the image displayed by the display unit; a detection mechanism that detects a user's mid-air operation on a screen of the floating-in-the-air image; and a correction unit that corrects a deviation between an operation target point and a detection point in a plane direction of the screen, regarding the user's mid-air operation on the screen of the floating-in-the-air image, wherein the correction unit corrects the deviation based on the shape of the fingertip of the user's operating finger when performing the mid-air operation. [Effects of the Invention]

[0007] According to the present invention, a more suitable floating-in-the-air image display device can be realized. Other problems, configurations, and effects will become clear in the following description of the embodiments. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of a usage form of a space floating image display device according to an embodiment of the present invention; [Figure 2A] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 2B] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 2C] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 2D] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a floating-in-the-air image display device according to an embodiment of the present invention; [Figure 2E] 1 is a projection view of a retroreflector constituting a floating-in-the-air image display device according to an embodiment of the present invention; [Figure 2F]1 is a top view of a retroreflector constituting a floating-in-the-air image display device according to an embodiment of the present invention; FIG. [Figure 2G] FIG. 1 is a perspective view showing a corner reflector that constitutes a retroreflector that constitutes a floating-in-the-air image display device according to an embodiment of the present invention. [Figure 2H] FIG. 1 is a top view showing a corner reflector that constitutes a retroreflector that constitutes a floating-in-the-air image display device according to an embodiment of the present invention. [Figure 2I] FIG. 1 is a side view showing a corner reflector that constitutes a retroreflector that constitutes a floating-in-the-air image display device according to an embodiment of the present invention. [Figure 3] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4A] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4B] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4C] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4D] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4E] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4F] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4G] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4H] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4I] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4J] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4K]1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4L] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4M] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4N] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4O] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 5] 1 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. [Figure 6] 1 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. [Figure 7] 1 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. [Figure 8] 1 is a layout diagram showing a main part of a space floating image display device according to an embodiment of the present invention; [Figure 9] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 10] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 11] 1 is an explanatory diagram for explaining the light source diffusion characteristics of an image display device according to an embodiment of the present invention. [Figure 12] 1 is an explanatory diagram for explaining the diffusion characteristics of a video display device according to an embodiment of the present invention; [Figure 13A] 1 is a diagram illustrating an example of a problem to be solved by image processing according to an embodiment of the present invention; [Figure 13B] FIG. 10 is an explanatory diagram of an example of image processing according to an embodiment of the present invention. [Figure 13C] FIG. 10 is an explanatory diagram of an example of a video display process according to an embodiment of the present invention. [Figure 13D] FIG. 10 is an explanatory diagram of an example of a video display process according to an embodiment of the present invention. [Figure 14] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 15] 1A and 1B are diagrams illustrating an example of the configuration of a space floating image display device and a correction method according to an embodiment of the present invention. [Figure 16] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 17] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 18] FIG. 1 is a diagram illustrating an outline of the configuration of each embodiment. [Figure 19] FIG. 1 is a diagram illustrating an example of a configuration including a correction unit of a space floating image display device according to an embodiment of the present invention. [Figure 20A] FIG. 2 is a diagram illustrating an example of the configuration of a correction unit in Method 1 according to an embodiment of the present invention. [Figure 20B] FIG. 2 is a diagram illustrating an example of the configuration of a correction unit in Method 1 according to an embodiment of the present invention. [Figure 21] FIG. 10 is a diagram illustrating an example of the configuration of a correction unit in Method 2 according to an embodiment of the present invention. [Figure 22A] 10A-10C illustrate example screens for finger scanning guidance according to one embodiment of the present invention. [Figure 22B] 10A-10C illustrate example screens for finger scanning guidance according to one embodiment of the present invention. [Figure 23] FIG. 1 is an explanatory diagram of ellipse approximation fitting of Method 1 according to an embodiment of the present invention. [Figure 24] FIG. 1 is an explanatory diagram of a user's body skeleton estimation model according to an embodiment of the present invention. [Figure 25A] 10A and 10B are explanatory diagrams relating to the finger direction (particularly the y-axis) and posture in Method A according to one embodiment of the present invention. [Figure 25B] 10 is an explanatory diagram relating to the finger direction (particularly the y-axis) and posture in Method B according to one embodiment of the present invention. FIG. [Figure 26] 1A to 1C are explanatory diagrams illustrating examples of floating images in space and camera placement according to an embodiment of the present invention. [Figure 27A] 10 is an explanatory diagram relating to the finger direction (particularly the x-axis), posture, etc. in Method A according to one embodiment of the present invention. FIG. [Figure 27B] 10 is an explanatory diagram relating to the finger direction (particularly the x-axis) and posture in Method B according to one embodiment of the present invention. FIG. [Figure 28A] 1A and 1B are diagrams showing an example of a screen of a space floating image and a first example of an operation target point according to an embodiment of the present invention. [Figure 28B] 10A and 10B are diagrams showing an example of a screen of a space floating image and a second example of an operation target point according to an embodiment of the present invention. [Figure 28C] 10A and 10B are diagrams showing an example of a screen of a space floating image and a third example of an operation target point according to an embodiment of the present invention. [Figure 29] 10A and 10B are diagrams illustrating examples of finger directions relative to the plane of a floating image in space in Method A and Method X according to an embodiment of the present invention. [Figure 30A] 10A and 10B are diagrams illustrating examples of finger directions (especially the x-axis) relative to the plane of a floating image in space in Methods A and Y according to an embodiment of the present invention. [Figure 30B] 10A and 10B are diagrams illustrating examples of finger directions (especially the y-axis) relative to the plane of a floating image in space in Method A and Method Y according to an embodiment of the present invention. [Figure 31A] 10A and 10B are diagrams illustrating examples of finger directions (especially the x-axis) relative to the plane of a floating image in space in Method B and Method Y according to an embodiment of the present invention. [Figure 31B] 10A and 10B are diagrams illustrating examples of finger directions (especially the y-axis) relative to the plane of a floating image in space in Method B and Method Y according to an embodiment of the present invention. [Figure 32] 10A and 10B are diagrams illustrating concepts such as the deviation between an operation target point and a detection point on the xy plane of a space floating image according to one embodiment of the present invention. [Figure 33] FIG. 10 illustrates a mathematical expression for angles (θ, φ) corresponding to finger orientation according to one embodiment of the present invention. [Figure 34] FIG. 10 is an explanatory diagram illustrating details of ellipse approximation fitting in Method 1 according to one embodiment of the present invention. [Figure 35A]FIG. 10 is a diagram illustrating mathematical formulas for ellipse approximation according to one embodiment of the present invention. [Figure 35B] FIG. 10 is a diagram illustrating mathematical expressions for detection points according to one embodiment of the present invention. [Figure 35C] FIG. 10 is a diagram showing a mathematical formula for the correction amount (number of correction pixels) according to one embodiment of the present invention. [Figure 36] FIG. 1 is a plan view illustrating a floating image in space according to a method X according to an embodiment of the present invention. [Figure 37] FIG. 10 is an explanatory diagram of a modified example of scheme X according to an embodiment of the present invention. [Figure 38A] FIG. 10 is an explanatory diagram showing an example of mid-air operation when the dominant hand is right, according to an embodiment of the present invention. [Figure 38B] FIG. 10 is an explanatory diagram showing an example of mid-air operation when the dominant hand is the left hand, according to one embodiment of the present invention. [Figure 39] 10A and 10B are explanatory diagrams showing an example in which the user's viewpoint position deviates from the observation reference direction according to an embodiment of the present invention; [Figure 40] FIG. 10 is a diagram illustrating an example of the configuration of an aerial operation detection sensor as an example of a user operation detection mechanism according to an embodiment of the present invention. [Figure 41] FIG. 2 is a diagram illustrating an example of the configuration of a three-dimensional sensor as an example of a user operation detection mechanism according to an embodiment of the present invention. [Figure 42] FIG. 10 is a diagram showing an example of a screen configuration in a modified example related to calibration in an embodiment of the present invention. [Figure 43] FIG. 10 is a diagram illustrating an example of the configuration of a correction unit in a modified example in which method A and method B are used in combination, according to an embodiment of the present invention. [Figure 44] FIG. 10 is a diagram illustrating an example of the configuration of input and output of a correspondence table in a correction unit in method 2 according to an embodiment of the present invention. [Figure 45] FIG. 10 is a diagram showing a specific example of the configuration of a correspondence table according to an embodiment of the present invention. [Figure 46] FIG. 10 is a diagram showing an example of the configuration of a correspondence table for each user attribute as a modified example in accordance with an embodiment of the present invention. [Figure 47]1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 48] 1A to 1C are diagrams showing an example of a screen of a floating image in space (particularly a menu screen) and an example of an operation performed in the air by a user, according to one embodiment of the present invention. [Figure 49A] 1A and 1B are diagrams showing an example of a screen (particularly a sub-screen) of a floating image in space and an example of an operation performed in the air by a user, according to one embodiment of the present invention. [Figure 49B] 1A and 1B are diagrams showing an example of a screen (particularly a sub-screen) of a floating-in-space image and an example of a physical operation by a user, according to one embodiment of the present invention. [Figure 50] FIG. 10 is a diagram showing an example of a processing flow of a space floating image display device according to an embodiment of the present invention. [Figure 51] FIG. 10 is a diagram showing an example of a screen of a floating image in space (particularly a calibration transition notification) according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, components having the same functions are given the same reference numerals, and repeated explanations thereof may be omitted.

[0010] The following examples relate to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member. In the following explanation of the examples, the image floating in space is expressed using the term "floating image in space." Instead of this term, it is also acceptable to express it as "aerial image," "spatial image," "floating image in space," "floating optical image of displayed image," "floating optical image of displayed image," etc. The term "floating image in space," which is mainly used in the explanation of the examples, is used as a representative example of these terms.

[0011] According to the following embodiments, an image display device suitable for, for example, bank ATMs, train station ticket machines, digital signage, and the like can be realized. For example, currently, bank ATMs, train station ticket machines, and the like typically use touch panels. However, by using a transparent glass surface or a light-transmitting plate, high-resolution image information can be displayed in a floating state on the glass surface or light-transmitting plate. In this case, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflector. This improves light utilization efficiency and suppresses the ghost images that occur in addition to the main floating image, which is a problem with conventional retroreflection systems, thereby achieving a clear floating image. Furthermore, by using a device including the light source of this embodiment, a novel and highly usable floating image display device (floating image display system) can be provided that can significantly reduce power consumption. Furthermore, a floating image display device for a vehicle can be provided that can display a so-called unidirectional floating image that can be viewed inside and / or outside the vehicle. Example 1

[0012] <Example of how to use the space floating image display device> FIG. 1 is a diagram showing an example of how a space-floating image display device according to an embodiment of the present invention is used, illustrating the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using FIG. 2 and other figures. Light with a narrow-angle directional characteristic and specific polarization is emitted from image display device 1 as an image beam, and after reflection by the optical system within the space-floating image display device, it first enters retroreflector 2, retroreflects, and passes through transparent member 100 (glass, etc.), forming a real aerial image (space-floating image 3) on the outside of the glass surface. In the following embodiments, the retroreflector 2 (retroreflector) is used as an example of a retroreflector. However, the retroreflector 2 of the present invention is not limited to a planar plate, but is used as an example of a concept including a sheet-like retroreflector attached to a planar or non-planar member, or an entire assembly in which a sheet-like retroreflector is attached to a planar or non-planar member. Furthermore, since the light rays reflected by the retroreflector 2 have the optical property of forming an image, the retroreflector 2 may also be expressed as an imaging optical member or an imaging optical plate.

[0013] In addition, in a store or the like, a space is partitioned by a show window (also called "window glass") 105, which is a translucent member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit such a transparent member and display a floating image in one direction to the outside and / or inside of the store (space).

[0014] 1, the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing a means for reflecting specific polarized waves on the window glass 105, it is possible to reflect the waves and form an aerial image at a desired position inside the store.

[0015] <Configuration example of optical system for space floating image display device> 2A is a diagram showing an example of the configuration of an optical system of a space-floating image display device according to one embodiment of the present invention. The configuration of the space-floating image display device will be described in more detail using FIG. 2A. As shown in FIG. 2A(1), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light with narrow-angle diffusion characteristics.

[0016] Image light of a specific polarization from the display device 1 is reflected by a polarization separator 101 (in the figure, the polarization separator 101 is formed into a sheet and adhered to the transparent member 100) that has a film that selectively reflects image light of a specific polarization and is provided on a transparent member 100, and then enters the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector 2. The image light passes through the λ / 4 plate 21 twice, once when it enters the retroreflector 2 and once when it exits, thereby undergoing polarization conversion from the specific polarization to the other polarization. Here, the polarization separator 101 that selectively reflects image light of a specific polarization has the property of transmitting the polarized light of the other polarization that has been polarization-converted, so the image light of the specific polarization after polarization conversion passes through the polarization separator 101. The image light that has passed through the polarization separator 101 forms a space-floating image 3, which is a real image, outside the transparent member 100. 2A shows an example in which the chief ray of the image light incident on the retroreflector 2 is incident at an angle of 90° to the retroreflector 2. However, the incident angle of the chief ray of the image light on the retroreflector 2 is not limited to 90°, and an angle of, for example, 90°±15° can also be used.

[0017] Here, a first example of polarization design for the optical system of FIG. 2A will be described. For example, S-polarized image light may be emitted from display device 1 to polarization separator 101, which may have the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the S-polarized image light reaching polarization separator 101 from display device 1 is reflected by polarization separator 101 and travels toward retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, thereby converting the S-polarized image light to P-polarized light. The P-polarized image light then travels back toward polarization separator 101. Here, polarization separator 101 has the property of reflecting S-polarized light and transmitting P-polarized light, so the P-polarized image light passes through polarization separator 101 and then through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0018] Next, a second example of polarization design for the optical system of FIG. 2A will be described. For example, P-polarized image light may be emitted from display device 1 to polarization separator 101, which may have the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the P-polarized image light reaching polarization separator 101 from display device 1 is reflected by polarization separator 101 and travels toward retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, converting the P-polarized light to S-polarized light. The S-polarized image light then travels back toward polarization separator 101. Here, polarization separator 101 has the property of reflecting P-polarized light and transmitting S-polarized light, so the S-polarized image light passes through polarization separator 101 and then through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0019] The light that forms the floating image 3 is a collection of light rays that converge from the retroreflector 2 to the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is a highly directional image, unlike the diffused image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2A, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be perceived as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0020] Depending on the performance of the retroreflector 2, the polarization axis of the reflected image light may become irregular. The reflection angle may also become irregular. Such irregular light may not maintain the polarization state and propagation angle assumed in the design. For example, light with an unintended polarization state and propagation angle may re-enter the image display surface of the liquid crystal display panel 11 directly from the position of the retroreflector 2 without passing through a polarization separation member. Such light with an unintended polarization state and propagation angle may re-enter the image display surface of the liquid crystal display panel 11 after being reflected by components within the space-floating image display device. Such light re-entering the image display surface of the liquid crystal display panel 11 may be re-reflected by the image display surface of the liquid crystal display panel 11 constituting the display device 1, potentially generating ghost images and degrading the image quality of the space-floating image. Therefore, in this embodiment, an absorbing polarizer 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorptive polarizer 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorptive polarizer 12, thereby suppressing the re-reflection. This makes it possible to prevent degradation of image quality due to ghost images of spatially floating images. Specifically, if the display device 1 is configured to emit S-polarized image light to the polarization separation member 101, the absorptive polarizer 12 may be a polarizer that absorbs P-polarized light. Furthermore, if the display device 1 is configured to emit P-polarized image light to the polarization separation member 101, the absorptive polarizer 12 may be a polarizer that absorbs S-polarized light.

[0021] The polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

[0022] 2A(2) shows an example of the surface shape of a typical retroreflector 2. Light rays incident on the interior of the regularly arranged hexagonal prisms are reflected by the walls and bottoms of the hexagonal prisms and emitted as retroreflected light in a direction corresponding to the incident light, and a real floating image is displayed based on the image displayed on the display device 1.

[0023] The resolution of this floating image in space depends not only on the resolution of the liquid crystal display panel 11, but also on the outer diameter D and pitch P of the retroreflective portion of the retroreflector 2 shown in Figure 2A(2). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, one pixel of the floating image in space will be equivalent to 300 μm. As a result, the effective resolution of the floating image in space will be reduced to about one-third.

[0024] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire caused by the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Also, it is advisable to arrange the shape so that none of the sides of the retroreflective portion overlaps with any of the sides of one pixel of the liquid crystal display panel.

[0025] The surface shape of the retroreflector according to this embodiment is not limited to the above example. Various surface shapes that achieve retroreflection may be used. Specifically, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which triangular pyramidal prisms, hexagonal pyramidal prisms, other polygonal prisms, or a combination thereof are periodically arranged. Alternatively, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which these prisms are periodically arranged to form cube corners. These may also be referred to as corner reflector arrays or polyhedral reflector arrays. Alternatively, the surface of the retroreflector according to this embodiment may be provided with capsule lens-type retroreflection elements in which glass beads are periodically arranged. The detailed configuration of these retroreflection elements can be achieved using existing technology, so a detailed description will be omitted. Specifically, the techniques disclosed in Japanese Patent Laid-Open Nos. 2001-33609, 2001-264525, 2005-181555, 2008-70898, and 2009-229942 may be used.

[0026] <Another configuration example 1 of the optical system of the space floating image display device> Another example of the configuration of the optical system of the space floating image display device will be explained using Fig. 2B. In Fig. 2B, components with the same reference numerals as Fig. 2A have the same functions and configurations as Fig. 2A. For the sake of simplicity, repeated explanations of such components will be omitted.

[0027] In the optical system of FIG. 2B, as in FIG. 2A, image light of a specific polarization is output from the display device 1. The image light of a specific polarization output from the display device 1 is input to a polarization separator 101B. The polarization separator 101B is a member that selectively transmits image light of a specific polarization. Unlike the polarization separator 101 of FIG. 2A, the polarization separator 101B is not integrated with the transparent member 100 but has an independent plate-like shape. Therefore, the polarization separator 101B may also be referred to as a polarization separator plate. The polarization separator 101B may be configured as a reflective polarizer configured by attaching a polarization separator sheet to a transparent member. Alternatively, the transparent member may be formed of a metal multilayer film that selectively transmits specific polarization and reflects polarization of other specific polarizations. In FIG. 2B, the polarization separator 101B is configured to transmit image light of a specific polarization output from the display device 1.

[0028] The image light that has passed through the polarization separation member 101B is incident on the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector. The image light is polarized and converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when it enters the retroreflector and once when it leaves. Here, the polarization separation member 101B has the property of reflecting the polarized light of the other polarization that has been polarized and converted by the λ / 4 plate 21, so the image light after polarization conversion is reflected by the polarization separation member 101B. The image light reflected by the polarization separation member 101B passes through the transparent member 100 and forms a space-floating image 3, which is a real image, outside the transparent member 100.

[0029] Here, a first example of polarization design for the optical system of FIG. 2B will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from display device 1 to polarization separator 101B, and polarization separator 101B has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized image light that reaches polarization separator 101B from display device 1 passes through polarization separator 101B and proceeds to retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, and the image light is converted from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again to polarization separator 101B. Here, polarization separator 101B has the property of reflecting S-polarized light and transmitting P-polarized light, so the S-polarized image light is reflected by polarization separator 101 and passes through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a space-floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the space-floating image 3 to be formed optimally.

[0030] Next, a second example of polarization design for the optical system of FIG. 2B will be described. For example, S-polarized image light may be emitted from display device 1 to polarization separator 101B, which may have the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized image light reaching polarization separator 101B from display device 1 passes through polarization separator 101B and proceeds to retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, converting the image light from S-polarized light to P-polarized light. The P-polarized image light then proceeds again to polarization separator 101B. Here, polarization separator 101B has the property of reflecting P-polarized light and transmitting S-polarized light, so the P-polarized image light is reflected by polarization separator 101 and passes through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a space-floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the space-floating image 3 to be formed optimally.

[0031] In FIG. 2B , the image display surface of the display device 1 and the surface of the retroreflector 2 are arranged parallel to each other. The polarization separator 101B is arranged tilted at an angle α (e.g., 30°) relative to the image display surface of the display device 1 and the surface of the retroreflector 2. When the polarization separator 101B reflects the image light, the direction of the image light reflected by the polarization separator 101B (the direction of the chief ray of the image light) differs by an angle β (e.g., 60°) from the direction of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light). With this configuration, the optical system of FIG. 2B outputs the image light toward the outside of the transparent member 100 at a predetermined angle shown in the figure, forming the space-floating image 3, which is a real image. In the configuration of FIG. 2B , when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be perceived as an image at all. This characteristic is extremely suitable for use in a system that displays images that require high security or highly confidential images that should be concealed from people directly facing the user.

[0032] As described above, the optical system of FIG. 2B has a different configuration from the optical system of FIG. 2A, but can form a suitable floating image in space, similar to the optical system of FIG. 2A.

[0033] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized waves of the image light from the polarization separation member 101B and absorb the polarized waves that are 90° out of phase with the polarized waves of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2B due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0034] <Another configuration example 2 of the optical system of the space floating image display device> Another example of the configuration of the optical system of the space floating image display device will be explained using Fig. 2C. In Fig. 2C, components with the same reference numerals as Fig. 2B have the same functions and configurations as Fig. 2B. For the sake of simplicity, such components will not be described repeatedly.

[0035] The only difference between the optical system in Figure 2B and the optical system in Figure 2C is the angle at which the polarization separation member 101B is disposed relative to the image display surface of the display device 1 and the surface of the retroreflector 2. All other configurations are the same as those of the optical system in Figure 2B, so repeated explanations will be omitted. The polarization design of the optical system in Figure 2C is also the same as that of the optical system in Figure 2B, so repeated explanations will be omitted.

[0036] In the optical system of FIG. 2C , the polarization separator 101B is tilted at an angle α with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. In FIG. 2C , the angle α is 45°. With this configuration, when the polarization separator 101B reflects, the angle β between the direction of propagation of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light) and the direction of propagation of the image light reflected by the polarization separator 101B (the direction of the chief ray of the image light) is 90°. With this configuration, the image display surface of the display device 1 and the surface of the retroreflector 2 are perpendicular to the direction of propagation of the image light reflected by the polarization separator 101B, simplifying the angular relationships of the surfaces that make up the optical system. By arranging the surface of the transparent member 100 so that it is perpendicular to the direction of propagation of the image light reflected by the polarization separator 101B, the angular relationships of the surfaces that make up the optical system can be further simplified. In the configuration of Figure 2C, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0037] As described above, the optical system of Fig. 2C has a different configuration from the optical systems of Fig. 2A and Fig. 2B, but can form a suitable floating image in space similar to the optical systems of Fig. 2A and Fig. 2B. In addition, the angles of the surfaces constituting the optical system can be made simpler.

[0038] An absorptive polarizer may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizer may transmit the polarized waves of the image light from the polarization separation member 101B and absorb the polarized waves that are 90° out of phase with the polarized waves of the image light from the polarization separation member 101B. This allows the image light for forming the space-floating image 3 to be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This allows the stray light in the optical system of FIG. 2C due to the external light incident on the space-floating image 3 side of the transparent member 100 to be reduced.

[0039] <Another configuration example 3 of the optical system of the space floating image display device> Another example of the configuration of the optical system of the space floating image display device will be described with reference to FIG. 2D. The optical system of FIG. 2D is an optical system that uses a retroreflector 5 that is different from the retroreflector 2 used in FIGS. 2A to 2C. Hereinafter, another example of the configuration 3 of the optical system will be described in more detail with reference to FIGS. 2D to 2I. In FIG. 2D, components that are assigned the same reference numerals as those in FIGS. 2A to 2C have the same functions and configurations as those in FIGS. 2A to 2C. For the sake of simplicity, such components will not be described repeatedly.

[0040] 2D is a diagram showing an example of the main components and retroreflection components of a space-floating image display device according to an embodiment of the present invention. A display device 10 that emits image light is provided obliquely on a transparent member 100 such as glass. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light.

[0041] A chief ray 9020 representing the light beam emitted from the display device 10 travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α may be, for example, 45°. However, the incident angle α is not limited to 45°, and may also be, for example, 45°±15°.

[0042] The retroreflector 5 is an optical element having the optical property of retroreflecting light rays in at least some directions. Furthermore, since the reflected light rays have the optical property of forming an image, the retroreflector 5 may also be referred to as an imaging optical element or an imaging optical plate.

[0043] 2E, 2F, etc., the principal ray 9020 travels in the z direction while being retroreflected in the x and y directions by the retroreflector 5. As a result, the reflected ray 9021 travels in a direction away from the retroreflector 5 along an optical path that is mirror-symmetrical with respect to the principal ray 9020 with the retroreflector 5 as the reference, passes through the transparent member 100, and forms the spatial floating image 3 as a real image on the imaging plane.

[0044] The light beam that forms the floating image 3 is a collection of light rays that converge from the retroreflector 5 to the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is an image with high directionality, unlike a diffuse image formed on a screen by a general projector or the like. Therefore, in the configuration of Figure 2, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be perceived as an image at all. This characteristic is suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0045] An example of the configuration of the retroreflector 5 will be described using Figures 2E and 2F. The retroreflector 5 has a configuration in which multiple corner reflectors 9040 are arranged in an array on the surface of a transparent member 50. This may also be called a corner reflector array or a polyhedral reflector array. The specific configuration of the corner reflector 9040 will be described in detail using Figures 2G, 2H, and 2I. Light rays 9111, 9112, 9113, and 9114 emitted from a light source 9110 are reflected twice by two mirror surfaces 9041 and 9042 of the corner reflector 9040, becoming reflected light rays 9121, 9122, 9123, and 9124. This double reflection is retroreflection in the x and y directions, where the light returns to the same direction as the incident direction (traveling in a direction rotated 180 degrees), and specular reflection in the z direction, where the angle of incidence and the angle of reflection match due to total reflection.

[0046] That is, light rays 9111 to 9114 generate reflected light rays 9121 to 9124 on straight lines symmetrical in the z direction with respect to corner reflector 9040, and form aerial real image 9120. Note that light rays 9111 to 9114 emitted from light source 9110 are four light rays that represent the diffused light from light source 9110, and although the light rays incident on retroreflector 5 are not limited to these four light rays depending on the diffusion characteristics of light source 9110, all incident light rays cause similar reflections and form aerial real image 9120. Note that for ease of viewing the drawing, the position of light source 9110 and the position of aerial real image 9120 are shown shifted in the x direction, but in reality, the position of light source 9110 and the position of aerial real image 9120 in the x direction are the same and are overlapping when viewed from the z direction.

[0047] 2G, 2H, and 2I, the configuration and effects of the corner reflector 9040 that constitutes the retroreflector 5 will be described. The corner reflector 9040 is a rectangular parallelepiped with only two specific faces being mirror surfaces 9041 and 9042, and the other four faces being made of transparent materials. The retroreflector 5 has a configuration in which these corner reflectors 9040 are arrayed so that corresponding mirror surfaces face in the same direction.

[0048] When viewed from the top (+z direction), a light ray 9111 emitted from a light source 9110 enters the mirror surface 9041 (or the mirror surface 9042) at a specific angle of incidence, is totally reflected at a reflection point 9130, and then is totally reflected again at a reflection point 9132 on the mirror surface 9042 (or the mirror surface 9041).

[0049] If the angle of incidence of light ray 9111 with respect to mirror surface 9041 (or mirror surface 9042) is θ, then the angle of incidence of first reflected light ray 9131 reflected by mirror surface 9041 (or mirror surface 9042) with respect to mirror surface 9042 (or mirror surface 9041) can be expressed as 90°-θ. Therefore, with respect to light ray 9111, second reflected light ray 9121 is rotated by 2θ after the first reflection and by 2×(90°-θ) after the second reflection, resulting in a total reversal optical path of 180°. On the other hand, when viewed from the side (the direction halfway between -x and -y), total reflection in the z direction occurs only once. Therefore, if the angle of incidence with respect to mirror surface 9041 or mirror surface 9042 is φ, then reflected light ray 9121 is rotated by 2×φ after one reflection with respect to light ray 9111.

[0050] As described above, light rays incident on the corner reflector 9040 undergo retroreflection, which creates an inverted optical path in the x and y directions, and specular reflection due to total reflection in the z direction. Considering the retroreflector 5, similar reflections occur in each optical path, so that an image is formed at a point symmetrical with respect to the z axis direction by an inverted optical path that is convergent in the x and y directions.

[0051] 2A to 2C, the retroreflector 2 has retroreflection properties in three axes. As a result, when a diffusive incident light beam is incident on the retroreflector 2, a convergent reflected light beam travels toward the side of the retroreflector 2 where the light source of the incident light beam is located. The convergent reflected light beam forms an image in the air, forming a space-floating image 3. The traveling direction of the chief ray of the convergent reflected light beam reflected from the retroreflector 2 is opposite to the traveling direction of the chief ray of the diffusive incident light beam incident on the retroreflector 2.

[0052] In contrast, in the optical system of Fig. 2D, the retroreflector 5 has retroreflection properties in two axial directions and specular reflection in the other axial direction. As a result, when a diffusive incident light beam is incident on the retroreflector 5, the convergent reflected light beam is reflected by the corner reflector array and travels in the direction opposite to the side of the retroreflector 5 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air and forms the space floating image 3.

[0053] The traveling direction of the chief ray of the convergent reflected light beam reflected by the corner reflector array of the retroreflector 5 is not the opposite direction to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5. The normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5 and the normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray after being reflected by the retroreflector 5 and becoming a convergent reflected light beam continue to travel in a straight line, unchanged before and after reflection by the corner reflector array.

[0054] That is, the diffusive incident light beam is converted into a convergent reflected light beam by reflection on the retroreflector 5, but in the normal direction to the plate-shaped surface of the retroreflector 5, the light beam travels as if passing through the retroreflector 5. Here, the diffusive incident light beam incident on the retroreflector 5 and the convergent reflected light beam emerging from the retroreflector 5 are in a geometrically symmetrical relationship with respect to the plate-shaped surface of the retroreflector 5.

[0055] The resolution of the space-floating image formed by the light beams from the video output unit 10 depends not only on the resolution of the liquid crystal display panel 11 but also on the diameter D and pitch P (not shown) of the retroreflector 5 shown in Figures 2E and 2F. For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflector is 240 μm and the pitch P is 300 μm, one pixel of the space-floating image will be equivalent to 300 μm. As a result, the effective resolution of the space-floating image will be reduced to about one-third.

[0056] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 10, it is desirable to make the diameter D and pitch P of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire due to the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Also, it is advisable to arrange the shape of the retroreflective portion so that none of its sides overlaps any of the sides of one pixel of the liquid crystal display panel.

[0057] The shape of the retroreflector (imaging optical plate) according to this embodiment is not limited to the above example. It may have various shapes that achieve retroreflection. Specifically, it may be a variety of cubic corner bodies, a corner reflector array, a slit mirror array, a dihedral corner reflector array, a polyhedral reflector array, or a shape in which a combination of these reflective surfaces is periodically arranged. Alternatively, a capsule lens-type retroreflector element with periodically arranged glass beads may be provided on the surface of the retroreflector according to this embodiment. The detailed configuration of these retroreflectors can be achieved using existing technology, so a detailed description will be omitted. Specifically, the technology disclosed in JP 2017-33005 A, JP 2019-133110 A, JP 2017-67933 A, WO 2009 / 131128 A, etc. may be used.

[0058] 2D, the image light emitted from the display device 10 may be in either polarization state, either S-polarized or P-polarized.

[0059] As explained above, the optical system of Figure 2D is an optical system that uses a retroreflector different from the optical systems of Figures 2A to 2C, but it can form a more suitable floating image in space, similar to the optical systems of Figures 2A to 2C.

[0060] According to the optical systems of FIGS. 2A, 2B, 2C, and 2D described above, it is possible to provide brighter, higher quality floating images in space.

[0061] <<Block diagram of the internal configuration of the space floating image display device>>

[0062] Next, a description will be given of a block diagram of the internal configuration of the space-floating image display device 1000. Fig. 3 is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

[0063] The space-floating image display device 1000 includes a retroreflection unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power supply input interface 1111, an operation input unit 1107, a nonvolatile memory 1108, a memory 1109, a control unit 1110, a video signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, a microphone 1139, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc. In addition, the space-floating image display device 1000 may also include a removable media interface 1134, an attitude sensor 1113, a transmissive self-luminous image display device 1650, a second display device 1680, or a secondary battery 1112.

[0064] Each component of the space floating image display device 1000 is disposed in a housing 1190. The imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0065] The retroreflecting portion 1101 in Fig. 3 corresponds to the retroreflector 2 in Fig. 2A, Fig. 2B, and Fig. 2C. The retroreflecting portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected from the retroreflecting portion 1101, the light output to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0066] 3 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. The light source 1105 in FIG. 3 corresponds to the light source device 13 in FIGS. 2A, 2B, and 2C. The image display unit 1102, the light guide 1104, and the light source 1105 in FIG. 3 correspond to the display device 1 in FIGS. 2A, 2B, and 2C.

[0067] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 (described later). The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0068] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power source 1106 converts AC current input from the outside via the external power input interface 1111 into DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each unit within the space-floating image display device 1000. The secondary battery 1112 stores the power supplied from the power source 1106. The secondary battery 1112 also supplies power to the light source 1105 and other components that require power via the external power input interface 1111 when power is not supplied from the outside. In other words, when the space-floating image display device 1000 is equipped with the secondary battery 1112, the user can use the space-floating image display device 1000 even when power is not supplied from the outside.

[0069] The light guide 1104 guides light generated by the light source 1105 and irradiates it onto the video display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called a backlight for the video display unit 1102. The light guide 1104 may be configured mainly using glass. The light guide 1104 may be configured mainly using plastic. The light guide 1104 may be configured using a mirror. Various methods are possible for combining the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0070] The aerial operation detection sensor 1351 is a sensor that detects an operation on the floating in space image 3 by the finger of the user 230. The aerial operation detection sensor 1351 senses, for example, an area that overlaps with the entire display area of ​​the floating in space image 3. Note that the aerial operation detection sensor 1351 may only sense an area that overlaps with at least a portion of the display area of ​​the floating in space image 3.

[0071] Specific examples of the aerial operation detection sensor 1351 include a distance sensor that uses invisible light such as infrared light, an invisible laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may also be configured to detect coordinates on a two-dimensional plane by combining multiple sensors. The aerial operation detection sensor 1351 may also be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) or an image sensor.

[0072] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect touch operations, etc., made by the user with their finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0073] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and based on the sensing signal, determines whether or not the finger of the user 230 has made contact with an object in the floating in space image 3, and calculates the position (contact position) where the finger of the user 230 has made contact with the object. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array), for example. Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0074] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally as a separate entity from the space-floating image display device 1000. When provided as a separate entity from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to be able to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0075] Also, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the space-floating image display device 1000 without the aerial operation detection function is used as the main body, and only the aerial operation detection function can be added as an option. Also, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the space-floating image display device 1000 may be used. In cases such as when it is desired to more freely position the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0076] The imaging unit 1180 is a camera with an image sensor, and captures images of the space near the floating-in-space image 3 and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180, or by using an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in detecting the touch operation of the floating-in-space image 3 by the user 230. The imaging unit 1180 may be provided separately from the floating-in-space image display device 1000. When the imaging unit 1180 is provided separately from the floating-in-space image display device 1000, it is sufficient to configure it so that an imaging signal can be transmitted to the floating-in-space image display device 1000 via a wired or wireless communication connection path or the like.

[0077] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that targets a plane (intrusion detection plane) including the display surface of the spatial floating image 3 and detects whether or not an object has intruded into this intrusion detection plane, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object (e.g., a user's finger) that has not intruded into the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0078] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the captured images of the multiple imaging units 1180 and object depth information from the depth sensor. These pieces of information and various pieces of information such as the distance between the object and the intrusion detection plane are used for various display controls for the floating in space image 3.

[0079] Furthermore, without using the mid-air operation detection sensor 1351, the mid-air operation detection unit 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging unit 1180.

[0080] Furthermore, the imaging unit 1180 may capture an image of the face of the user 230 operating the space-floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not there is another person standing around or behind the user 230 operating the space-floating image 3 and peeking at the operation of the user 230 on the space-floating image 3, the imaging unit 1180 may capture an image of a range including the user 230 operating the space-floating image 3 and the surrounding area of ​​the user 230.

[0081] The operation input unit 1107 is, for example, an operation button, a signal receiving unit such as a remote controller, or an infrared light receiving unit, and inputs a signal for an operation different from the air operation (touch operation) by the user 230. Apart from the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may also be used by, for example, an administrator to operate the space floating image display device 1000.

[0082] The video signal input unit 1131 is connected to an external video output device and inputs video data. Various digital video input interfaces are possible for the video signal input unit 1131. For example, it may be configured with a video input interface conforming to the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, a video input interface conforming to the DVI (Digital Visual Interface) standard, or a video input interface conforming to the DisplayPort standard.

[0083] Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio signal input unit 1133 may be configured as an HDMI-standard audio input interface, an optical digital terminal interface, a coaxial digital terminal interface, or the like. In the case of an HDMI-standard interface, the video signal input unit 1131 and the audio signal input unit 1133 may be configured as an interface in which a terminal and a cable are integrated. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may be configured as a speaker.

[0084] The audio output unit 1140 may also output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 1140 may be configured to output a digital signal to an external device, like the Audio Return Channel function defined in the HDMI standard. The microphone 1139 is a microphone that collects sounds around the space-floating image display device 1000, converts them into signals, and generates audio signals. The microphone may record a person's voice, such as a user's voice, and the control unit 1110, which will be described later, may perform voice recognition processing on the generated audio signal to obtain text information from the audio signal.

[0085] The nonvolatile memory 1108 stores various data used by the space floating image display device 1000. The data stored in the nonvolatile memory 1108 includes, for example, data for various operations to be displayed on the space floating image 3, display icons, data and layout information for objects to be operated by user operations, etc. The memory 1109 stores image data to be displayed as the space floating image 3, data for controlling the device, etc.

[0086] The control unit 1110 controls the operation of each connected unit. In addition, the control unit 1110 may cooperate with a program stored in the memory 1109 to perform calculations based on information acquired from each unit in the space floating image display device 1000.

[0087] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. If the communication unit 1132 has a wired communication interface, the wired communication interface may be configured, for example, as an Ethernet LAN interface. If the communication unit 1132 has a wireless communication interface, the interface may be configured, for example, as a Wi-Fi communication interface, a Bluetooth communication interface, or a mobile communication interface such as 4G or 5G. Various types of data, such as video data, image data, and audio data, are transmitted and received through communication via the communication unit 1132.

[0088] Furthermore, the removable media interface 1134 is an interface for connecting a removable recording medium (removable media). The removable recording medium (removable media) may be composed of a semiconductor device memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disk. The removable media interface 1134 can read various information such as video data, image data, and audio data recorded on the removable recording medium. The video data, image data, etc. recorded on the removable recording medium are output as the floating image 3 via the video display unit 1102 and the retroreflection unit 1101.

[0089] The storage unit 1170 is a storage device that records various types of information such as video data, image data, audio data, etc. The storage unit 1170 may be configured with a magnetic recording medium recording device such as a hard disk drive (HDD), or a semiconductor element memory such as a solid state drive (SSD). For example, various types of information such as video data, image data, audio data, etc. may be recorded in advance in the storage unit 1170 at the time of product shipment. Furthermore, the storage unit 1170 may record various types of information such as video data, image data, audio data, etc. acquired from an external device, an external server, etc. via the communication unit 1132.

[0090] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflection unit 1101. The video data, image data, etc. of the display icons and objects for the user to operate, etc., displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0091] Layout information of display icons, objects, etc. displayed as the space floating image 3, and various metadata information related to the objects, etc. are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0092] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. The video control unit 1160 may be called a video processing circuit, and may be configured with hardware such as an ASIC, FPGA, or video processor. The video control unit 1160 may also be called a video processing unit or an image processing unit. The video control unit 1160 controls video switching, such as which video signal to input to the video display unit 1102, between the video signal to be stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131, for example.

[0093] In addition, the video control unit 1160 may generate a superimposed video signal by superimposing the video signal to be stored in the memory 1109 and the video signal input from the video signal input unit 1131, and input the superimposed video signal to the video display unit 1102, thereby performing control to form the composite video as the floating-in-space video 3.

[0094] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing to enlarge, reduce, deform, etc. the image, brightness adjustment processing to change the brightness, contrast adjustment processing to change the contrast curve of the image, and Retinex processing to decompose the image into light components and change the weighting of each component.

[0095] Furthermore, the video control unit 1160 may perform special effect video processing or the like to assist the aerial operation (touch operation) of the user 230 on the video signal input to the video display unit 1102. The special effect video processing is performed based on, for example, the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 and the image of the user 230 captured by the imaging unit 1180.

[0096] The attitude sensor 1113 is a sensor configured by a gravity sensor or an acceleration sensor, or a combination of these, and can detect the attitude in which the space-floating image display device 1000 is installed. Based on the attitude detection result of the attitude sensor 1113, the control unit 1110 may control the operation of each connected unit. For example, when an undesirable attitude in the user's usage state is detected, the control unit 1110 may perform control such that the image being displayed on the image display unit 1102 is stopped and an error message is displayed to the user. Alternatively, when the attitude sensor 1113 detects a change in the installation attitude of the space-floating image display device 1000, the control unit 1110 may perform control such that the display direction of the image being displayed on the image display unit 1102 is rotated.

[0097] As explained above, various functions are installed in the space-floating image display device 1000. However, the space-floating image display device 1000 does not need to have all of these functions, and any configuration is acceptable as long as it has the function of forming the space-floating image 3.

[0098] <Configuration example of a space floating image display device> Next, a configuration example of the space-floating image display device will be explained. The layout of the components of the space-floating image display device according to this embodiment can be various depending on the usage form. Below, the layouts of each of Figs. 4A to 4M will be explained. In addition, in each example of Figs. 4A to 4M, the thick line surrounding the space-floating image display device 1000 indicates an example of the housing structure of the space-floating image display device 1000.

[0099] FIG. 4A is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4A is equipped with an optical system corresponding to the optical system of FIG. 2A. The space-floating image display device 1000 shown in FIG. 4A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4A, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230. Note that the x direction is the left-right direction as seen from the user, the y direction is the front-back direction (depth direction) as seen from the user, and the z direction is the up-down direction (vertical direction). Hereinafter, the definitions of the x direction, y direction, and z direction are the same in each drawing of FIG. 4, so repeated explanations will be omitted.

[0100] FIG. 4B is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4B is equipped with an optical system corresponding to the optical system of FIG. 2A. The space-floating image display device 1000 shown in FIG. 4B is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4B, the space-floating image display device is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230. 4B, the mid-air operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so this configuration can improve the accuracy of touch detection.

[0101] FIG. 4C is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4C is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4C is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4C, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the user's 230 finger.

[0102] FIG. 4D is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4D is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4D is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4D, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger. 4D, the mid-air operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so this configuration can improve the accuracy of touch detection.

[0103] FIG. 4E is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4E is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4E is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4E, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels directly upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0104] FIG. 4F is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4F is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4F is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4F, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels toward the user. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger.

[0105] FIG. 4G is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4G is equipped with an optical system corresponding to the optical system shown in FIG. 2C. In the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the central optical path of the image light emitted from the display device 1 was on the yz plane. That is, in the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the image light traveled in the front-to-back and up-to-down directions as seen from the user. In contrast, in the optical system of the space-floating image display device shown in FIG. 4G, the central optical path of the image light emitted from the display device 1 is on the xy plane. That is, in the optical system of the space-floating image display device shown in FIG. 4G, the image light travels in the left-to-right and front-to-back directions as seen from the user. The space-floating image display device 1000 shown in FIG. 4G is installed so that the surface on which the space-floating image 3 is formed faces the front of the device (toward the user 230). That is, in Fig. 4G, the space-floating image display device 1000 has the transparent member 100 installed on the front side of the device (toward the user 230). The space-floating image 3 is formed on the user side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels towards the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0106] FIG. 4H is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4H differs from the space-floating image display device of FIG. 4G in that it has a window with a transparent plate 100B made of glass or plastic on the back of the device (opposite the position where the user 230 views the space-floating image 3, i.e., opposite the traveling direction of the image light of the space-floating image 3 toward the user 230). The rest of the configuration is the same as that of the space-floating image display device of FIG. 4G, so repeated explanations will be omitted. The space-floating image display device 1000 of FIG. 4H has a window with a transparent plate 100B on the opposite side of the traveling direction of the image light of the space-floating image 3 from the space-floating image 3. Therefore, when the user 230 views the space-floating image 3, they can recognize the scenery behind the space-floating image display device 1000 as the background of the space-floating image 3. Therefore, the user 230 can perceive the space floating image 3 as floating in the air in front of the scenery behind the space floating image display device 1000. This can further emphasize the feeling of the space floating image 3 floating in the air.

[0107] Depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separator 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separator 101B and head toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, in order to prevent this stray light, the transparent plate 100B may not be provided in the window on the back of the space-floating image display device 1000.

[0108] Fig. 4I is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4I is different from the space-floating image display device of Fig. 4H in that a light-blocking door 1410 is provided in the window of the transparent plate 100B located on the back of the device (opposite the position where the user 230 views the space-floating image 3). The other configurations are the same as those of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted.

[0109] The opening and closing door 1410 of the space-floating image display device 1000 in FIG. 4I has, for example, a light blocking plate, and is equipped with a mechanism for moving (sliding), rotating, or attaching / detaching the light blocking plate, thereby switching between an open state and a light blocking state for the window (rear window) of the transparent plate 100B located at the back of the space-floating image display device 1000. The movement (sliding) and rotation of the light blocking plate by the opening and closing door 1410 may be electrically driven by a motor (not shown). The motor may be controlled by the control unit 1110 in FIG. 3. Note that the example in FIG. 4I discloses an example in which the opening and closing door 1410 has two light blocking plates. In contrast, the opening and closing door 1410 may have only one light blocking plate.

[0110] For example, when the view seen through the window of the transparent plate 100B of the space-floating image display device 1000 is outdoors, the brightness of sunlight varies depending on the weather. When the outdoor sunlight is strong, the background of the space-floating image 3 may become too bright, reducing the user 230's visibility of the space-floating image 3. In such a case, by moving (sliding), rotating, or attaching the light blocking plate of the opening / closing door 1410 to block the light from the rear window, the background of the space-floating image 3 becomes dark, thereby relatively increasing the visibility of the space-floating image 3. Such a blocking operation by the light blocking plate of the opening / closing door 1410 may be performed directly by the force of the user 230's hand. In response to an operation input via the operation input unit 1107 of FIG. 3, the control unit 1110 may control a motor (not shown) to perform the blocking operation by the light blocking plate of the opening / closing door 1410.

[0111] An illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 may control a motor (not shown) to perform the opening and closing operation of the light blocking plate of the opening and closing door 1410 according to the detection result of the illuminance sensor. By controlling the opening and closing operation of the light blocking plate of the opening and closing door 1410 in this way, it becomes possible to more suitably maintain the visibility of the space-floating image 3, even if the user 230 does not manually open and close the light blocking plate of the opening and closing door 1410.

[0112] Furthermore, the light blocking plate by the opening and closing door 1410 may be manually detachable. Depending on the intended use and installation environment of the space floating image display device 1000, the user can select whether to leave the rear window open or in a light blocking state. If it is planned to use the rear window in a light blocking state for a long period of time, the detachable light blocking plate can be fixed in the light blocking state. Also, if it is planned to use the rear window in an open state for a long period of time, it can be used with the detachable light blocking plate removed. The light blocking plate may be attached and detached using screws, a hook structure, or a fitting structure.

[0113] Even in the example of the space-floating image display device 1000 shown in FIG. 4I, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separator 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separator 101B and directed toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be perceived by the user as stray light. Therefore, to prevent this stray light, the window on the back of the space-floating image display device 1000 may be configured without the transparent plate 100B. The above-described opening / closing door 1410 may be provided in a window that does not have the transparent plate 100B. To prevent this stray light, it is desirable that the inner surface of the housing of the light-shielding plate of the above-described opening / closing door 1410 have a coating or material with low light reflectance.

[0114] Fig. 4J is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4J differs from the space-floating image display device of Fig. 4H in that instead of placing a transparent plate 100B made of glass or plastic on the rear side window, an electronically controlled transmittance variable device 1620 is placed. The other configurations are the same as those of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted. An example of the electronically controlled transmittance variable device 1620 is a liquid crystal shutter.

[0115] In other words, the liquid crystal shutter can control the amount of light transmitted by controlling the voltage of the liquid crystal element sandwiched between two polarizing plates. Therefore, if the liquid crystal shutter is controlled to increase the transmittance, the scenery through the rear window can be seen through the background of the floating image 3. Also, if the liquid crystal shutter is controlled to increase the transmittance, the scenery through the rear window can be hidden as the background of the floating image 3. Furthermore, since the liquid crystal shutter can control the intermediate length, it can also be set to a state of transmittance of 50% or the like. For example, the control unit 1110 can control the transmittance of the electronically controlled transmittance variable device 1620 in response to an operation input via the operation input unit 1107 in Fig. 3. With this configuration, in cases where a viewer wants to see the scenery through the rear window as the background of the Space Floating Image 3, but the scenery through the rear window as the background is too bright and reduces the visibility of the Space Floating Image 3, it is possible to adjust the transmittance of the electronically controlled transmittance variable device 1620 to adjust the visibility of the Space Floating Image 3.

[0116] In addition, an illuminance sensor may be provided on the back side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 in Fig. 3 controls the transmittance of the electronically controlled transmittance variable device 1620 according to the detection result of the illuminance sensor. In this way, even if the user 230 does not perform an operation input via the operation input unit 1107 in Fig. 3, the transmittance of the electronically controlled transmittance variable device 1620 can be adjusted according to the brightness of the space beyond the rear window, making it possible to more suitably maintain the visibility of the space-floating image 3.

[0117] In the above example, a liquid crystal shutter has been described as an example of the electronically controlled variable transmittance device 1620. However, electronic paper may be used as another example of the electronically controlled variable transmittance device 1620. The same effects as those described above can be obtained when electronic paper is used. Furthermore, electronic paper consumes very little power to maintain a halftone state. Therefore, a space floating image display device with lower power consumption can be realized compared to when a liquid crystal shutter is used.

[0118] Fig. 4K is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4K differs from the space-floating image display device of Fig. 4G in that it has a transmissive self-luminous image display device 1650 instead of the transparent member 100. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0119] In the space-floating image display device 1000 of FIG. 4K, an image luminous flux passes through the display surface of the transmissive self-luminous image display device 1650, and then a space-floating image 3 is formed outside the space-floating image display device 1000. That is, when an image is displayed on the transmissive self-luminous image display device 1650, which is a two-dimensional flat display, the space-floating image 3 can be displayed as a pop-up image further in front of the image displayed on the transmissive self-luminous image display device 1650. In this case, the user 230 can simultaneously view two images at different depth positions. The transmissive self-luminous image display device 1650 may be configured using existing technology such as a transmissive organic EL panel disclosed in, for example, Japanese Patent Application Laid-Open No. 2014-216761. Although not shown in FIG. 3, the transmissive self-luminous image display device 1650 may be configured as a component of the space-floating image display device 1000 of FIG. 3 and connected to other processing units such as the control unit 1110.

[0120] Here, if the transmissive self-luminous video display device 1650 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating video 3, it is possible to provide the user 230 with a more effective video experience with a surprise effect.

[0121] Furthermore, if the inside of the space-floating image display device 1000 is kept in a light-blocking state, the background of the transmissive self-luminous image display device 1650 becomes sufficiently dark. Therefore, when no image is displayed on the display device 1 or the light source of the display device 1 is turned off and an image is displayed only on the transmissive self-luminous image display device 1650, the transmissive self-luminous image display device 1650 appears to the user 230 as a normal two-dimensional flat display rather than a transmissive display (since the space-floating image 3 in the embodiment of the present invention is displayed as a real optical image in a space without a screen, if the light source of the display device 1 is turned off, the intended display position of the space-floating image 3 becomes empty space). Therefore, when the transmissive self-luminous image display device 1650 is used to display an image as if it were a normal two-dimensional flat display, characters, objects, etc. can be suddenly displayed in the air as the space-floating image 3, thereby providing the user 230 with a more effective surprise video experience.

[0122] Note that the darker the interior of the space-floating image display device 1000, the more the transmissive self-luminous image display device 1650 appears like a two-dimensional flat display. Therefore, an absorptive polarizer (not shown) that transmits the polarized waves of the image light reflected by the polarization separation member 101B and absorbs polarized waves that are 90° out of phase with the polarized waves may be provided on the surface of the transmissive self-luminous image display device 1650 facing the interior of the space-floating image display device 1000 (the surface where the image light reflected by the polarization separation member 101B enters the transmissive self-luminous image display device 1650, i.e., the surface of the transmissive self-luminous image display device 1650 opposite the space-floating image 3). This does not have a significant effect on the image light that forms the space-floating image 3, but it can significantly reduce the light that enters the interior of the space-floating image display device 1000 from the outside through the transmissive self-luminous image display device 1650, making the interior of the space-floating image display device 1000 darker, which is preferable.

[0123] 4L is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4L is a modified example of the space-floating image display device of FIG. 4K. The orientation of the components in the space-floating image display device 1000 is different from that of the space-floating image display device of FIG. 4K, and is closer to the arrangement of the space-floating image display device of FIG. 4F. The functions and operations of each component are the same as those of the space-floating image display device of FIG. 4K, so repeated explanations will be omitted.

[0124] In the space-floating image display device of FIG. 4L, after the luminous flux of image light passes through the transmissive self-luminous image display device 1650, a space-floating image 3 is formed on the user 230 side of the transmissive self-luminous image display device 1650.

[0125] In both the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, the space-floating image 3 is displayed superimposed on the image of the transmissive self-luminous image display device 1650 as seen by the user 230. Here, the position of the space-floating image 3 and the position of the image of the transmissive self-luminous image display device 1650 are configured to have a difference in the depth direction. Therefore, when the user moves their head (the position of the viewpoint), they can recognize the depth of the two images due to parallax. Therefore, by displaying two images at different depth positions, it is possible to provide the user with a more suitable three-dimensional image experience with the naked eye without the need for stereoscopic glasses or the like.

[0126] Fig. 4M is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4M is provided with a second display device 1680 on the rear side as seen from the user relative to the polarization separation member 101B of the space-floating image display device of Fig. 4G. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0127] In the configuration example shown in FIG. 4M, the second display device 1680 is provided behind the display position of the space-floating image 3, and its image display surface faces the space-floating image 3. With this configuration, from the user 230's perspective, the image of the second display device 1680 and the space-floating image 3, which are displayed at two different depth positions, can be viewed superimposed on each other. In other words, the second display device 1680 is positioned so as to display an image in the direction of the user 230 viewing the space-floating image 3. Although the second display device 1680 is not shown in FIG. 3, it may be configured to be connected to other processing units such as the control unit 1110 as one component of the space-floating image display device 1000 of FIG. 3.

[0128] Note that the image light of the second display device 1680 of the space-floating image display device 1000 of FIG. 4M is viewed by the user 230 after passing through the polarization separator 101B. Therefore, in order for the image light of the second display device 1680 to more suitably pass through the polarization separator 101B, it is desirable that the image light output from the second display device 1680 be polarized in a vibration direction that the polarization separator 101B more suitably transmits. That is, it is desirable that the image light be polarized in the same vibration direction as the polarization of the image light output from the display device 1. For example, if the image light output from the display device 1 is S-polarized, it is desirable that the image light output from the second display device 1680 is also S-polarized. Furthermore, if the image light output from the display device 1 is P-polarized, it is desirable that the image light output from the second display device 1680 is also P-polarized.

[0129] The example of the space-floating image display device of FIG. 4M also has the same effect as the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L in that a second image is displayed behind the space-floating image 3. However, unlike the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, in the example of the space-floating image display device of FIG. 4M, the luminous flux of image light for forming the space-floating image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-luminous image display device, but may be a liquid crystal display, which is a two-dimensional flat display. The second display device 1680 may also be an organic EL display. Therefore, the example of the space-floating image display device of FIG. 4M can realize the space-floating image display device 1000 at a lower cost than the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L.

[0130] Here, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and travel toward the second display device 1680. This light (a portion of the image light) may be reflected again by the surface of the second display device 1680 and may be visually recognized by the user as stray light.

[0131] Therefore, to prevent this stray light, an absorptive polarizer may be provided on the surface of the second display device 1680. In this case, the absorptive polarizer may be an absorptive polarizer that transmits the polarized waves of the image light output from the second display device 1680 and absorbs polarized waves that are 90° out of phase with the polarized waves of the image light output from the second display device 1680. If the second display device 1680 is a liquid crystal display, an absorptive polarizer is also provided on the image output side of the liquid crystal display. However, if there is a cover glass (cover glass on the image display surface side) on the output surface of the absorptive polarizer on the image output side of the liquid crystal display, it is not possible to prevent stray light caused by reflection of the cover glass by light from outside the liquid crystal display. Therefore, it is necessary to separately provide the above-mentioned absorptive polarizer on the surface of the cover glass.

[0132] When an image is displayed on the second display device 1680, which is a two-dimensional flat display, the floating-in-space image 3 can be displayed as an image further in front of the image on the second display device 1680. In this case, the user 230 can simultaneously view two images at different depth positions. By displaying a character on the floating-in-space image 3 and a background on the second display device 1680, it is possible to provide the effect that the user 230 is viewing the space in which the character exists in three dimensions.

[0133] Furthermore, if the second display device 1680 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise visual experience.

[0134] Next, Fig. 4N is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4N is a space-floating image display device that employs the optical system of Fig. 2D. As with the example of the space-floating image display device that employs the optical system of Figs. 2A to 2C, an image is formed in the air as a space-floating image 3 by image light that has passed through a transparent member 100. Furthermore, the operation of the space-floating image 3 by the user's finger 9004 can be detected using sensing light from an aerial operation detection sensor 1351 that is arranged on the back side of the transparent member 100 as seen from the user.

[0135] 2A to 2C, and in the example of the space-floating image display device employing the optical system of Fig. 2D, the space-floating image 3 is formed in front of the transparent member 100, and the operation of the space-floating image 3 by the user's finger can be detected using the sensing light of the mid-air operation detection sensor 1351 arranged on the back side of the transparent member 100 as seen from the user. Therefore, the space-floating image display device employing the optical system of Fig. 2D has a different optical system from the space-floating image display device in which the optical system of Fig. 2A to 2C is arranged on the back side of the transparent member 100 as seen from the user.

[0136] However, from the user's perspective, the usability of the space-floating image display device employing the optical system of FIG. 2D is almost the same as that of the space-floating image display device employing the optical system of FIGS. 2A to 2C.

[0137] Next, Fig. 4O is a diagram showing an example of the configuration of a space-floating image display device. Fig. 4O is a diagram showing the configuration of the internal optical system of the space-floating image display device 1000 of Fig. 4N. The space-floating image display device 1000 shown in Fig. 4O is equipped with an optical system corresponding to the optical system of Fig. 2D. The space-floating image display device 1000 shown in Fig. 4O is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward.

[0138] 4O, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0139] Here, the configuration of Fig. 4O will be compared with the configuration of Fig. 4A to confirm the differences. In Fig. 4A, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the polarization separation member 101. In contrast, in Fig. 4O, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the retroreflector 5. Furthermore, the configuration of Fig. 4A includes the retroreflector 2 and the λ / 4 plate 21, but these do not exist in Fig. 4O. Furthermore, while the presence of an absorbing polarizer 12 is more preferable in Fig. 4A, the absorbing polarizer 12 is not particularly necessary in Fig. 4O.

[0140] That is, to replace the optical system of FIG. 2A in the configuration of FIG. 4A with the optical system of FIG. 2D and to replace it with the configuration of FIG. 4O, the following can be done. That is, the polarization separation member 101 in the configuration of FIG. 4A is replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 plate 21 are removed from the configuration of FIG. 4A. The absorptive polarizer 12 is optional. By performing a replacement based on this idea, the optical system of FIGS. 2A to 2C mounted in the configuration of the space-floating image display device of FIGS. 4A to 4G can be replaced with the optical system of FIG. 2D, and the space-floating image display device can be replaced with the optical system of FIG. 2D. In this case, the polarization separation member 101 in FIGS. 4A and 4B is replaced with the retroreflector 5, and the polarization separation member 101B in FIGS. 4C to 4G is replaced with the retroreflector 5.

[0141] In this way, it is possible to realize a space-floating image display device in which the optical system in the configuration of the space-floating image display device of Figures 4A to 4G is replaced with the optical system of Figure 2D. Even in these space-floating image display devices in which the optical system of Figure 2D is replaced, it is possible to realize a space-floating image display device that is almost as easy to use as the space-floating image display device of Figures 4A to 4G.

[0142] <Display device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes its light source. Fig. 5 shows the light source device 13 together with the liquid crystal display panel as an exploded perspective view.

[0143] As shown by arrow 30 in Fig. 5, this liquid crystal display panel (image display element 11) receives an illumination light beam from light source device 13, which is a backlight device, that has narrow-angle diffusion characteristics, i.e., has strong directivity (straightness) and characteristics similar to laser light with a polarization plane aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam in accordance with an input video signal. The modulated image light is reflected by retroreflector 2 and passes through transparent member 100 to form a real image, a floating image in space (see Fig. 1).

[0144] 5 also shows a configuration including a liquid crystal display panel 11 constituting the display device 1, a light redirection panel 54 that controls the directional characteristics of the light beam emitted from the light source device 13, and a narrow-angle diffuser (not shown) as needed. Specifically, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarization is emitted with its intensity modulated by a video signal (see arrow 30 in FIG. 5). This allows a desired image to be projected as highly directional (linearly propagating) light of a specific polarization via the light redirection panel 54 toward the retroreflector 2. After being reflected by the retroreflector 2, the light is transmitted toward the eyes of an observer outside the store (space), forming a floating image 3. A protective cover 50 (see FIGS. 6 and 7) may be provided on the surface of the light redirection panel 54.

[0145] <Display device example 1> FIG. 6 shows an example of a specific configuration of the display device 1. In FIG. 6, a liquid crystal display panel 11 and a light direction conversion panel 54 are disposed on the light source device 13 shown in FIG. 5. The light source device 13 is configured on a case shown in FIG. 5, which is formed of, for example, plastic and contains LED elements 201 and a light guide 203. As shown in FIG. 5 and other figures, the end surface of the light guide 203 is provided with a lens shape whose cross-sectional area gradually increases toward the light receiving section in order to convert the divergent light from each LED element 201 into a substantially parallel beam. The lens shape has an effect of gradually reducing the divergence angle by multiple total reflections during propagation within the light guide 203. The liquid crystal display panel 11 constituting the display device 1 is attached to the top surface of the display device 1. In addition, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202 on which their control circuits are mounted are attached to one side surface (the left end surface in this example) of the case of the light source device 13. A heat sink, which is a member for cooling the heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED substrate 202.

[0146] The liquid crystal display panel frame (not shown) is attached to the top surface of the case of the light source device 13. The liquid crystal display panel 11 is attached to the frame, and an FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel 11 is also attached to the frame. That is, the liquid crystal display panel 11, which is the image display element, generates a display image by modulating the intensity of transmitted light in conjunction with the LED elements 201, which are solid-state light sources, based on a control signal from a control circuit (image control unit 1160 in FIG. 3) constituting the electronic device. The generated image light has a narrow diffusion angle and contains only specific polarization components, resulting in a novel image display device similar to a surface-emitting laser image source driven by a video signal. Currently, it is technically and safety-wise impossible to obtain a laser beam of the same size as the image obtained by the display device 1 described above using a laser device. Therefore, in this embodiment, light similar to the surface-emitting laser image light described above is obtained from a beam of light from a general light source, for example, an LED element.

[0147] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 7 as well as FIG.

[0148] 6 and 7 are cross-sectional views, and only one of the multiple LED elements 201 constituting the light source is shown, and this light is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion of the light guide end surface and the LED element are attached while maintaining a predetermined positional relationship.

[0149] Each light guide 203 is formed of a translucent resin such as acrylic. The LED light receiving surface at the end of light guide 203 has a cone-shaped outer periphery obtained by rotating a parabolic cross section, and the top of the light guide 203 has a concave portion with a convex portion (i.e., a convex lens surface) formed in the center, and the center of the flat portion has a convex lens surface (alternatively, a concave lens surface) that protrudes outward (not shown). The outer shape of the light receiving portion of the light guide to which LED element 201 is attached is a parabolic shape that forms a cone-shaped outer periphery, and is set within an angle range that allows total reflection within the light that is emitted from the LED element toward the periphery, or a reflective surface is formed.

[0150] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed to the LED collimator (light-receiving end surface 203a) so that the LED elements 201 on the surface are positioned in the center of the recessed portion described above.

[0151] According to this configuration, the shape of the light-receiving end surface 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as approximately parallel light, thereby improving the efficiency of use of the generated light.

[0152] As described above, the light source device 13 is configured by attaching a light source unit in which a plurality of LED elements 201 serving as light sources are arranged to the light-receiving end surface 203a, which is a light-receiving section provided on the end surface of the light guide 203, and the divergent light beams from the LED elements 201 are converted into approximately parallel light by the lens shape of the light-receiving end surface 203a of the light guide end surface, which is then guided inside the light guide 203 (in a direction parallel to the drawing) as shown by the arrow, and emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11, which is disposed approximately parallel to the light guide 203 (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of the light beam direction conversion means 204 depending on the shape inside or on the surface of the light guide.

[0153] The light beam direction conversion means 204 described above emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 (in a direction perpendicular to the front of the drawing) which is disposed substantially parallel to the light guide 203, by changing the shape of the surface of the light guide or by providing a portion with a different refractive index inside the light guide. In this case, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is positioned at the same position as the diagonal dimension of the screen, if the relative brightness ratio between the center of the screen and the periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

[0154] 6 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED element 201. In Fig. 6, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface of light source device 13 is attached liquid crystal display panel 11 that has polarizing plates on the light source light entrance surface and the image light exit surface.

[0155] In addition, a film- or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarization (e.g., P-wave) 212 of the natural light beam 210 emitted from the LED element 201. The reflected light is reflected again by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and directed toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49. The reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting the reflected light beam from P-polarized to S-polarized, thereby improving the utilization efficiency of the light source light as image light. The image light beam, the light intensity of which is modulated by a video signal in the liquid crystal display panel 11 (arrow 213 in Figure 6), enters the retroreflector 2. After reflection by the retroreflector 2, a real, floating image can be obtained.

[0156] 7 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED elements 201, similar to Fig. 6. Light source device 13 is similarly composed of light guide 203 formed of, for example, plastic and having light beam direction conversion means 204 on its surface or inside, LED elements 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc. On the top surface of light source device 13, a liquid crystal display panel 11 is attached as an image display element, which has polarizing plates on the light source light entrance surface and the image light exit surface.

[0157] A film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., S wave) 211 of the natural light beam 210 emitted from the LED element 201. In other words, the selective reflection characteristics of the reflective polarizing plate 49 in the example of FIG. 7 differ from those in FIG. 7. The reflected light is reflected by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and returns to the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting it from S-polarized light to P-polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam intensity-modulated by the image signal in the liquid crystal display panel 11 (arrow 214 in FIG. 7) enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0158] In the light source devices shown in Figures 6 and 7, in addition to the function of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, the reflective polarizer reflects the polarized light component on one side, so the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizer multiplied by the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In fact, experiments have confirmed that the contrast performance of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those of self-luminous organic EL displays are obtained.

[0159] <Display device example 2> 8 shows another example of the specific configuration of the display device 1. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, etc. in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. Also, an LED board on which LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and their control circuits are mounted are attached to one side of the case of light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer surface of the LED board.

[0160] The liquid crystal display panel frame attached to the top surface of the case is configured to have attached thereto a liquid crystal display panel 11 attached to the frame, and further to have attached thereto an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements 14a and 14b, which are solid-state light sources, based on a control signal from a control circuit (not shown here) that constitutes the electronic device.

[0161] <Display device example 3> Next, another example of the specific configuration of the display device 1 (Example 3 of the display device) will be described with reference to Fig. 9. The light source device of this display device 1 converts a divergent beam of light (a mixture of P-polarized and S-polarized light) from an LED into a substantially parallel beam by a collimator 18, and reflects the parallel beam toward the liquid crystal display panel 11 by the reflecting surface of a reflective light guide 304. The reflected light is incident on a reflective polarizer 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizer 49 transmits light of a specific polarization (e.g., P-polarized light) and causes the transmitted polarized light to be incident on the liquid crystal display panel 11. Here, light of polarization other than the specific polarization (e.g., S-polarized light) is reflected by the reflective polarizer 49 and directed toward the reflective light guide 304 again.

[0162] The reflective polarizing plate 49 is installed at an angle with respect to the liquid crystal display panel 11 so that the reflective polarizing plate 49 is not perpendicular to the chief ray of light from the reflective surface of the reflective light guide 304. The chief ray of the light reflected by the reflective polarizing plate 49 is incident on the transmission surface of the reflective light guide 304. The light that has entered the transmission surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 which is a retardation plate, and is reflected by the reflector 271. The light reflected by the reflector 271 passes through the λ / 4 plate 270 again, and passes through the transmission surface of the reflective light guide 304. The light that has passed through the transmission surface of the reflective light guide 304 is incident on the reflective polarizing plate 49 again.

[0163] At this time, the light that re-enters the reflective polarizer 49 has passed through the λ / 4 plate 270 twice, and therefore its polarization has been converted to a polarization (for example, P-polarized light) that is transmitted through the reflective polarizer 49. Therefore, the light whose polarization has been converted passes through the reflective polarizer 49 and enters the liquid crystal display panel 11. Note that with regard to the polarization design related to the polarization conversion, the polarization may be configured in reverse from the above explanation (S-polarized light and P-polarized light may be reversed).

[0164] As a result, the light from the LED is aligned to a specific polarization (for example, P polarization), enters the liquid crystal display panel 11, and is brightness-modulated in accordance with the video signal to display an image on the panel surface. As in the above example, multiple LEDs that make up the light source are shown (however, since this is a vertical cross section, only one is shown in Figure 9), and these are attached at predetermined positions relative to the collimator 18.

[0165] Each of the collimators 18 is formed of a translucent resin such as acrylic or glass. The collimator 18 may have a cone-shaped outer peripheral surface obtained by rotating a parabolic cross section. The collimator 18 may have a concave portion with a convex portion (i.e., a convex lens surface) formed in the center of the apex (the side facing the LED substrate 102) of the collimator 18. The collimator 18 may have a convex lens surface protruding outward (or a concave lens surface recessed inward) in the center of the flat portion (the side opposite the apex). The parabolic surface forming the cone-shaped outer peripheral surface of the collimator 18 is set within an angle range that allows total reflection of the light emitted from the LED in the peripheral direction within the parabolic surface, or a reflective surface is formed therein.

[0166] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 102. The LED substrate 102 is arranged and fixed to the collimator 18 so that the LEDs on the surface are positioned at the center of the apex of the convex cone shape (or in the recess if there is a recess at the apex).

[0167] With this configuration, the collimator 18 focuses the light emitted from the LED, particularly the light emitted from the central portion, into parallel light by the convex lens surface that forms the outer shape of the collimator 18. Light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the conical shape of the collimator 18, and is similarly focused into parallel light. In other words, the collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, makes it possible to extract almost all of the light generated by the LED as parallel light, thereby improving the utilization efficiency of the generated light.

[0168] Furthermore, the light converted into approximately parallel light by the collimator 18 shown in FIG. 9 is reflected by the reflective light guide 304. Due to the action of the reflective polarizer 49, light of a specific polarization of the light passes through the reflective polarizer 49, while light of the other polarization reflected by the reflective polarizer 49 passes through the light guide 304 again. The light is reflected by the reflector 271, which is located opposite the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarized and converted twice by passing through the λ / 4 plate 270, which is a retardation plate. The light reflected by the reflector 271 passes through the light guide 304 again and enters the reflective polarizer 49 provided on the opposite surface. Since the incident light has been polarization-converted, it passes through the reflective polarizer 49, aligns its polarization direction, and enters the liquid crystal display panel 11. As a result, all of the light from the light source can be utilized, thereby doubling the geometrical optical utilization efficiency of light. Furthermore, since the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, the use of the light source device of this embodiment significantly improves the contrast ratio of the entire display device. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 allows adjustment of the angle of light reflection and diffusion at each reflective surface. The surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 can be adjusted for each design to optimize the uniformity of the light incident on the liquid crystal display panel 11.

[0169] It should be noted that the λ / 4 plate 270, which is the retardation plate in Fig. 9, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of Fig. 9, any retardation plate may be used as long as the phase changes by 90° (λ / 2) when polarized light passes through it twice. The thickness of the retardation plate may be adjusted according to the incident angle distribution of the polarized light.

[0170] <Display device example 4> Furthermore, another example (Example 4 of Display Device) of the configuration of an optical system such as a light source device of a display device will be described with reference to Fig. 10. This is an example of a configuration in which a diffusion sheet is used instead of reflective light guide 304 in the light source device of Example 3 of the display device. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) that convert the diffusion characteristics in the vertical and horizontal directions of the drawing (front and rear directions in the drawing, not shown) are used on the light emission side of collimator 18, and light from collimator 18 is made to enter between the two optical sheets (diffusion sheets).

[0171] The optical sheet may be a single sheet instead of a two-sheet configuration. In the case of a single sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes on the front and back surfaces of the single optical sheet. Alternatively, multiple diffusion sheets may be used to share the functions. In the example of FIG. 10, the reflection and diffusion characteristics due to the front and back shapes of optical sheets 207A and 207B may be optimally designed using the number of LEDs, the divergence angle from LED substrate (optical element) 102, and the optical specifications of collimator 18 as design parameters so that the surface density of the light beam emitted from liquid crystal display panel 11 is uniform. In other words, the diffusion characteristics are adjusted by the surface shapes of multiple diffusion sheets instead of light guides.

[0172] In the example of FIG. 10, polarization conversion is performed in the same manner as in the display device example 3 described above. That is, in the example of FIG. 10, reflective polarizing plate 49 may be configured to have the property of reflecting S-polarized light (transmitting P-polarized light). In this case, the reflective polarizing plate 49 transmits P-polarized light out of the light emitted from the LED light source, and the transmitted light enters liquid crystal display panel 11. The reflective polarizing plate 49 reflects S-polarized light out of the light emitted from the LED light source, and the reflected light passes through retardation plate 270 shown in FIG. 10. The light that passes through retardation plate 270 is reflected by reflector 271. The light reflected by reflector 271 passes through retardation plate 270 again and is converted to P-polarized light. The polarization-converted light passes through reflective polarizing plate 49 and enters liquid crystal display panel 11.

[0173] It should be noted that the λ / 4 plate 270, which is the retarder in FIG. 10, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of FIG. 10, any retarder that changes the phase by 90° (λ / 2) when polarized light passes through it twice will suffice. The thickness of the retarder may be adjusted according to the distribution of incident angles of the polarized light. It should be noted that in FIG. 10 as well, the polarization design for polarization conversion may be configured in reverse (reversing the S-polarized light and P-polarized light) from the above explanation.

[0174] In a typical TV device, the light emitted from the LCD panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (shown on the X-axis in FIG. 12(a)) and the vertical direction of the screen (shown on the Y-axis in FIG. 12(b)). In contrast, the diffusion characteristics of the light beam emitted from the LCD panel of this embodiment are 1 / 5 of the 62-degree viewing angle of a typical TV device, as shown in Example 1 of FIG. 12, when the viewing angle at which the luminance is 50% of that at a front view (angle of 0 degrees) is set to 13 degrees. Similarly, the vertical viewing angle is asymmetric between the top and bottom, and the reflection angle and the area of ​​the reflective surface of the reflective light guide are optimized to keep the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness more than 50 times higher.

[0175] Furthermore, assuming the viewing angle characteristics shown in Example 2 in Figure 12, if the viewing angle at which brightness is 50% of that when viewed from the front (angle of 0 degrees) is set to 5 degrees, this will be 1 / 12 of the 62 degrees of devices used for general TV applications. Similarly, the vertical viewing angle is optimized by optimizing the reflection angle and the area of ​​the reflective surface of the reflective light guide so that the viewing angle is approximately 1 / 12 of that of devices used for general TV applications, with equal viewing angle both above and below. As a result, the amount of image light directed in the monitoring direction is significantly improved compared to conventional LCD TVs, and brightness is more than 100 times greater.

[0176] As described above, by setting the viewing angle to a narrow angle, the amount of luminous flux directed in the monitoring direction can be concentrated, significantly improving the efficiency of light utilization. As a result, even when using a liquid crystal display panel for general TV applications, by controlling the light diffusion characteristics of the light source device, it is possible to achieve a significant improvement in brightness with similar power consumption, making it possible to create a video display device that is compatible with information display systems facing bright outdoor environments.

[0177] When using a large LCD display panel, the overall brightness of the screen can be improved by directing the light from the periphery of the screen inward so that it is directed toward the observer when the observer is facing the center of the screen. Figure 11 shows the convergence angle between the long and short sides of the panel when the observer's distance from the panel, L, and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in portrait orientation, the convergence angle can be set to match the short side. For example, when using a 22-inch panel in portrait orientation and the monitoring distance is 0.8 m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the observer.

[0178] Similarly, when monitoring with a 15-inch panel in portrait orientation, if the monitoring distance is 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the monitor. As mentioned above, depending on the size of the LCD panel and whether it is used portrait or landscape, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the monitor who is in the optimum position to monitor the centre of the screen.

[0179] As shown in Figure 9, the basic configuration involves a light source device directing a light beam with a narrow angle of directionality to a liquid crystal display panel 11, which is then luminance-modulated according to a video signal. The video information displayed on the screen of the liquid crystal display panel 11 is then reflected by a retroreflector, and the resulting floating image is displayed indoors or outdoors via a transparent member 100.

[0180] By using the display device and light source device according to the embodiment of the present invention described above, it is possible to realize a space floating image display device with higher light utilization efficiency.

[0181] <Example of image display processing in a space floating image display device> Next, an example of a problem solved by the image processing of this embodiment will be described with reference to Fig. 13A. In the space-floating image display device 1000, when the far side of the space-floating image 3 from the user's perspective is inside the housing of the space-floating image display device 1000 and it is sufficiently dark, the user will visually recognize that the background of the space-floating image 3 is black.

[0182] Here, an example of displaying a character "panda" 1525 in the space floating image 3 will be described using Fig. 13A. First, the image control unit 1160 in Fig. 3 distinguishes and recognizes the pixel area where the image of the character "panda" 1525 is drawn from the transparent information area 1520 which is the background image, for an image including the pixel area where the image of the character "panda" 1525 is drawn and the transparent information area 1520 which is the background image, as shown in Fig. 13A(1).

[0183] A method for distinguishing and recognizing the character image from the background image is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the character image layer in front of the background image layer can be processed as separate layers, and the character image and background image can be distinguished and recognized based on the superimposition relationship when these layers are combined.

[0184] Here, the image control unit 1160 recognizes black pixels that depict objects such as character images and transparent information pixels as different information. However, it is assumed that both the black pixels that depict objects and the transparent information pixels have a luminance of 0. In this case, when the space floating image 3 is displayed, there is no difference in luminance between the pixels that depict black in the image of the character "panda" 1525 and the pixels of the transparent information region 1520, which is the background image. Therefore, in the space floating image 3, as shown in FIG. 13A(2), neither the pixels that depict black in the image of the character "panda" 1525 nor the pixels of the transparent information region 1520 have luminance, and they are visually perceived by the user as the same optically black space. In other words, the black parts of the image of the character "panda" 1525, which is an object, blend into the background, and only the non-black parts of the character "panda" 1525 are perceived as floating in the display region of the space floating image 3.

[0185] An example of image processing according to this embodiment will be described with reference to FIG. 13B. FIG. 13B is a diagram illustrating an example of image processing that more suitably resolves the issue of the black image region of the object blending into the background, as described in FIG. 13A. In FIGS. 13B(1) and 13B(2), the upper side shows the display state of the floating image 3 in space, and the lower side shows the input / output characteristics of the image processing of the image of the object. Note that the image of the object (character "panda" 1525) and the corresponding data may be read from the storage unit 1170 or memory 1109 in FIG. 3, or may be input from the video signal input unit 1131, or may be acquired via the communication unit 1132.

[0186] In the state shown in Figure 13B(1), the input / output characteristics of the image processing of the object image are in a linear state with no particular adjustment. In this case, the display state is the same as that shown in Figure 13A(2), and the black image area of ​​the object blends into the background. In contrast, in Figure 13B(2), the video control unit 1160 of this embodiment adjusts the input / output characteristics of the image processing of the image of the object (character "panda" 1525) to the input / output characteristics shown in the lower part.

[0187] That is, the video control unit 1160 performs image processing with input / output characteristics on the image of the object (character "panda" 1525), which has a characteristic of converting pixels in low-brightness areas of the input image into output pixels with increased brightness values. After the image of the object (character "panda" 1525) has been subjected to image processing with the input / output characteristics, a video including the image of the object (character "panda" 1525) is input to the display device 1 and displayed. Then, as shown in the upper part of FIG. 13B(2), the display state of the floating in space image 3 is such that the brightness of pixel areas depicting black in the image of character "panda" 1525 increases. This allows the user to distinguish the areas depicting black among the areas depicting the image of character "panda" 1525 without them blending into the black background, making it possible to display the object more appropriately.

[0188] 13B(2), the area displaying the image of the character "panda" 1525, which is an object, can be distinguished from the black background inside the housing of the space-floating image display device 1000 through the window, improving the visibility of the object. Therefore, for example, even if the object includes pixels with a brightness value of 0 before the image processing (i.e., when the image of the object or the corresponding data is read from the storage unit 1170 or memory 1109 in FIG. 3, or when the image of the object is input from the video signal input unit 1131, or when the data of the object is obtained via the communication unit 1132, etc.), the image processing of the input / output characteristics by the video control unit 1160 converts the object into an object with a brightness value of 0 increased for the pixels in the low-brightness area, and then the object is displayed on the display device 1 and converted into a space-floating image 3 by the optical system of the space-floating image display device 1000.

[0189] That is, the pixels that make up the object after image processing of the input / output characteristics are converted to a state in which they do not include pixels with a brightness value of 0, and then they are displayed on the display device 1 and converted into a floating image 3 in space by the optical system of the floating image display device 1000.

[0190] In the image processing of Figure 13B(2), a method for applying image processing with the input / output characteristics of Figure 13B(2) only to the image area of ​​the object (character "panda" 1525) is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the layer of the character image in front of the background image layer can be processed as separate layers, and the image processing with the input / output characteristics of Figure 13B(2) is applied to the character image layer, while not applying this image processing to the background image layer.

[0191] Then, by combining these layers, image processing with a characteristic of raising the low-brightness areas of the input image is performed only on the character image, as shown in Fig. 13B(2). Alternatively, after the character image layer and background image layer are combined, image processing with the input / output characteristics of Fig. 13B(2) may be performed only on the character image area.

[0192] Furthermore, the input / output image characteristics used in the image processing for boosting low-luminance regions of the input / output characteristics for the input image are not limited to the example shown in FIG. 13B(2). Any image processing for boosting low luminance may be used, including so-called brightness adjustment. Alternatively, image processing for improving visibility by controlling the gain that changes the weighting of Retinex processing, as disclosed in International Publication WO 2014 / 162533, may be performed.

[0193] According to the image processing of FIG. 13B(2) described above, it is possible to make the user aware of areas where black is drawn among areas where images of characters, objects, etc. are drawn without blending into the black background, thereby realizing a more suitable display.

[0194] 13A and 13B, the problems and more suitable image processing were explained using the space-floating image display device in which the background appears black (for example, the space-floating image display device 1000 in FIGS. 4A to 4G, or the space-floating image display device 1000 in the state where the rear window is shielded in FIGS. 4I and 4J). However, the image processing is also effective for devices other than these space-floating image display devices.

[0195] Specifically, in the space-floating image display device 1000 of Fig. 4H, or in Fig. 4I and Fig. 4J where the rear window is not shaded, the background of the space-floating image 3 is not black, but the scenery behind the space-floating image display device 1000 through the window. In this case, the same problems as those described in Fig. 13A and Fig. 13B exist.

[0196] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the scenery behind the space-floating image display device 1000 through the window. In this case too, by using the image processing of Fig. 13B(2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognized as being distinct from the scenery behind the space-floating image display device 1000 through the window, improving the visibility of the object.

[0197] That is, by using the image processing of FIG. 13B(2), the area displaying the image of the object character "panda" 1525 can be recognized as distinct from the scenery behind the space floating image display device 1000 through the window, and it becomes possible to more easily recognize that the object character "panda" 1525 is in front of the scenery, improving the visibility of the object.

[0198] 4K, 4L, and 4M, as described above, when another image (such as an image from the transmissive self-luminous image display device 1650 or an image from the second display device 1680) is displayed at a position different in depth from the space-floating image 3, the background of the space-floating image 3 is not black, but the other image. In this case, the problems explained in FIGS. 13A and 13B also exist.

[0199] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the other image that is displayed at a different depth from the floating image in space 3. In this case as well, by using the image processing of Fig. 13B(2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be able to be recognized as distinct from the other image, improving the visibility of the object.

[0200] In other words, by using the image processing of Figure 13B (2), the area displaying the image of the object character "panda" 1525 can be recognized as distinct from the other image, and it can be more easily recognized that the object character "panda" 1525 is in front of the other image, improving the visibility of the object.

[0201] An example of the image display process of this embodiment will be described with reference to Fig. 13C. Fig. 13C shows an example of the image display of this embodiment in which the space floating image 3 and a second image 2050, which is another image, are simultaneously displayed. The second image 2050 may correspond to the image displayed by the transmissive self-luminous image display device 1650 of Fig. 4K or Fig. 4L. The second image 2050 may also correspond to the image displayed by the second display device 1680 of Fig. 4M.

[0202] That is, the example of the image display in Fig. 13C shows a specific example of the image display examples of the space-floating image display device 1000 in Figs. 4K, 4L, and 4M. In the example of this figure, a bear character is displayed in the space-floating image 3. The area other than the bear character in the space-floating image 3 is displayed in black, and becomes transparent as a space-floating image. In addition, the second image 2050 is a background image in which a plain, a mountain, and a sun are drawn.

[0203] 13C, the floating in space image 3 and the second image 2050 are displayed at different depth positions. When the user 230 views the two images, the floating in space image 3 and the second image 2050, in the line of sight of the arrow 2040, the user 230 can view the two images in a superimposed state. Specifically, the bear character of the floating in space image 3 appears to be superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0204] Here, since the space floating image 3 is formed as a real image in the air, when the user 230 moves his / her viewpoint slightly, he / she can recognize the depth of the space floating image 3 and the second image 2050 due to parallax. Therefore, the user 230 can get a stronger sense of floating in space from the space floating image 3 while viewing the two images in an overlapping state.

[0205] An example of the video display process of this embodiment will be described with reference to Fig. 13D. Fig. 13D(1) is a diagram of the floating in space image 3, from the example of the video display of this embodiment in Fig. 13C, as seen from the line of sight of the user 230. Here, a bear character is displayed in the floating in space image 3. The area other than the bear character in the floating in space image 3 is displayed in black, and becomes transparent as a floating in space image.

[0206] 13D(2) is a diagram showing the second image 2050 in the example of the video display of this embodiment in FIG. 13C as viewed from the line of sight of the user 230. In the example of this figure, the second image 2050 is a background image in which a plain, a mountain, and a sun are drawn.

[0207] 13D(3) is a diagram showing the state in which the second image 2050 and the floating in space image 3 appear superimposed in the line of sight of the user 230, among the example of image display of this embodiment in Fig. 13C. Specifically, the bear character of the floating in space image 3 appears superimposed in front of the background of plains, mountains, and the sun drawn in the second image 2050.

[0208] Here, when simultaneously displaying the space-floating image 3 and the second image 2050, it is desirable to pay attention to the balance of brightness between the two images in order to ensure better visibility of the space-floating image 3. If the second image 2050 is too bright compared to the brightness of the space-floating image 3, the displayed image of the space-floating image 3 will be transparent, and the second image 2050, which is the background, will be strongly visible through it.

[0209] Therefore, the output of the light source of the spatially floating image 3 and the display image brightness of the display device 1, and the output of the light source of the display device displaying the second image 2050 and the display image brightness of the display device should be set so that at least the brightness per unit area of ​​the spatially floating image 3 at the display position of the spatially floating image 3 is greater than the brightness per unit area of ​​the image light that reaches the display position of the spatially floating image 3 from the second image 2050.

[0210] Note that this condition only needs to be satisfied when the space-floating image 3 and the second image 2050 are displayed simultaneously, and therefore when switching from the first display mode in which the space-floating image 3 is not displayed and only the second image 2050 is displayed to the second display mode in which the space-floating image 3 and the second image 2050 are displayed simultaneously, control may be performed to reduce the brightness of the second image 2050 by lowering the output of the light source of the display device that displays the second image 2050 and / or the display image brightness of the display device. These controls may be realized by the control unit 1110 in Fig. 3 controlling the display device 1 and the display device that displays the second image 2050 (the transmissive self-luminous image display device 1650 in Fig. 4K or Fig. 4L or the second display device 1680 in Fig. 4M).

[0211] Note that when switching from the above-described first display mode to the above-described second display mode, if control is performed to reduce the brightness of second image 2050, the brightness may be reduced uniformly across the entire screen of second image 2050. Alternatively, instead of reducing the brightness uniformly across the entire screen of second image 2050, the brightness reduction effect may be greatest in the portion where an object is displayed in space-floating image 3, and the brightness reduction effect may be gradually weakened around that portion. In other words, if the brightness of second image 2050 is reduced only in the portion where space-floating image 3 is visually recognized as being superimposed on second image 2050, the visibility of space-floating image 3 is sufficiently ensured.

[0212] Here, since the space floating image 3 and the second image 2050 are displayed at positions with different depths, when the user 230 slightly changes his / her viewpoint, the parallax causes a change in the superimposed position of the space floating image 3 relative to the second image 2050. Therefore, when switching from the above-mentioned first display mode to the above-mentioned second display mode, if the brightness is to be reduced unevenly across the entire screen of the second image 2050, it is not desirable to reduce the brightness sharply based on the outline of the object displayed in the space floating image 3, but rather it is desirable to perform a gradation process of the brightness reduction effect, which changes the brightness reduction effect stepwise depending on the position as described above.

[0213] In addition, in the space floating image display device 1000 where the position of the object displayed in the space floating image 3 is approximately at the center of the space floating image 3, the position where the brightness reduction effect of the gradation processing of the brightness reduction effect is greatest can be the center position of the space floating image 3.

[0214] According to the image display process of this embodiment described above, the user 230 can visually recognize the space floating image 3 and the second image 2050 more favorably.

[0215] Note that when displaying the space-floating image 3, control may be performed so as not to display the second image 2050. Since not displaying the second image 2050 increases the visibility of the space-floating image 3, this is suitable for applications such as the space-floating image display device 1000 where the user must be able to reliably view the space-floating image 3 when the space-floating image 3 is displayed.

[0216] <Example 2> As Example 2 of the present invention, an example of another configuration example of the space-floating image display device will be described. Note that the space-floating image display device according to this example is obtained by changing the optical system stored in the space-floating image display device described in Example 1 to the optical system shown in FIG. 14(1) or FIG. 14(2). In this example, differences from Example 1 will be described, and repeated explanations of the same configuration as Example 1 will be omitted. Note that in the following description of this example, the predetermined polarized light and the other polarized light are polarized waves whose phases differ by 90° from each other.

[0217] Fig. 14(1) shows an example of an optical system and an optical path according to this embodiment. The optical system shown in Fig. 14(1) is configured such that the display device 1 is closer to the polarization separation member 101B in the optical system of Fig. 2C, making the entire optical system more compact. In Fig. 14(1), the components denoted by the same reference numerals as in Fig. 2C will not be described in detail again.

[0218] 14(1), similar to FIG. 2C, image light of a predetermined polarized light (P polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, similar to FIG. 2C, polarization separation member 101B selectively transmits the predetermined polarized light (P polarized light in the figure) emitted from display device 1 and reflects the other polarized light (S polarized light in the figure).

[0219] Therefore, image light of a predetermined polarization (P-polarized light in the figure) traveling vertically from the image display surface of display device 1 passes through polarization separator 101B and reaches retroreflector 2 to which λ / 4 plate 21 is attached. The image light that is retroreflected by retroreflector 2 and travels again toward polarization separator 101B has passed through λ / 4 plate 21 twice, and is converted from the predetermined polarization (P-polarized light in the figure) at the time of emission from display device 1 to the other polarization (S-polarized light in the figure). The image light that travels again toward polarization separator 101B is the other polarization (S-polarized light in the figure), and is therefore reflected by polarization separator 101B toward the position where the user should be. The traveling direction of the image reflected by polarization separator 101B is determined based on the angle at which polarization separator 101B is disposed.

[0220] In the example of Figure 14(1), the image light traveling toward the polarization separation member 101B is reflected at a right angle by the polarization separation member 101B and travels as shown in the figure. The image light reflected by the polarization separation member 101B forms a space-floating image 3A. The space-floating image 3A can be viewed by the user from the direction of arrow A.

[0221] Here, due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 until it reaches the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 until it reaches the formation position of the space floating image 3A. This relationship determines the formation position of the space floating image 3A in the traveling direction of the image light reflected by the polarization separation member 101B.

[0222] In the example of FIG. 14(1), the display device 1, the polarization separating member 101B, and the retroreflector 2 are arranged closer than in the example of FIG. 2C. This allows the entire optical system to be configured more compactly. However, the amount by which the space-floating image 3A protrudes from the optical system of FIG. 14(1) is not very large. For example, as an index of the amount by which the space-floating image 3A protrudes from the optical system, the figure shows the distance from the position where the central light beam of the image light is reflected by the polarization separating member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of FIG. 14(1)).

[0223] 14(1), the characteristics of P polarization and S polarization may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S polarization, and the characteristics of P polarization and S polarization may be interchanged in the reflection characteristics of the polarization separation member 101B. In this case, the P polarization and S polarization shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0224] Next, Fig. 14(2) shows another example of an optical system and an optical path according to this embodiment. The optical system of Fig. 14(2) is a modified version of the optical system of Fig. 14(1) in order to increase the amount of the floating image projecting from the optical system while realizing the same compactness as the optical system of Fig. 14(1). In Fig. 14(2), the components with the same reference numerals as those in Fig. 14(1) will not be described in detail again.

[0225] 14(2), similar to FIG. 14(1), image light of a predetermined polarized light (P polarized light in the figure) emitted from the display device 1 travels in a direction perpendicular to the image display surface of the display device 1. Here, the polarization characteristics of the polarization separation member 101B are arranged 90 degrees differently from those in FIG. 14(1). Image light of a predetermined polarized light (P polarized light in the figure) traveling in a direction perpendicular to the image display surface of the display device 1 passes through the polarization separation member 101B.

[0226] 14(1), the image light passing through the polarization separation member 101B is not faced with the retroreflector 2 having the λ / 4 plate 21 attached thereto, but with the specular reflector 4 having the λ / 4 plate 21B attached thereto. Here, the reflection at the specular reflector 4 is specular reflection (also called regular reflection), not retroreflection.

[0227] Therefore, the image light that has passed through polarization separation member 101B is specularly reflected by specular reflector 4 to which λ / 4 plate 21B is attached. The image light that has been specularly reflected by specular reflector 4 and travels again toward polarization separation member 101B has been converted from the predetermined polarization (P polarization in the figure) at the time of emission from display device 1 to the other polarization (S polarization in the figure) by having passed through λ / 4 plate 21 twice. The image light that has traveled again toward polarization separation member 101B is the other polarization (S polarization in the figure), and is therefore reflected by polarization separation member 101B.

[0228] Here, because the orientation of the polarization separator 101B in Figure 14(2) is different from that in Figure 14(1), the image light reflected by the polarization separator 101B travels in the opposite direction from where the user should be. A retroreflector 2 with a λ / 4 plate 21C attached is disposed at the destination of the image light reflected by the polarization separator 101B. The image light is retroreflected by the retroreflector 2. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separator 101B has been converted from the other polarized light (S-polarized light in the figure) to the specified polarized light (P-polarized light in the figure) by passing through the λ / 4 plate 21C twice.

[0229] The image light that travels back toward the polarization separation member 101B is of a predetermined polarization (P polarization in the figure), so it passes through the polarization separation member 101B and continues toward the location where the user should be. The image light that has passed through the polarization separation member 101B forms a space-floating image 3B. The space-floating image 3B can be easily viewed by the user from the direction of arrow A.

[0230] 14(2), similarly to FIG. 14(1), due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the formation position of the space-floating image 3B. This relationship determines the formation position of the space-floating image 3B in the traveling direction of the image light transmitted through the polarization separation member 101B.

[0231] 14(2), the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is longer than the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 in FIG. 14(1). This is because in the optical system of FIG. 14(2), an optical path that goes back and forth between the polarization separation member 101B and the specular reflector 4, which does not exist in the optical system of FIG. 14(1), is added to the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2.

[0232] As a result, in the optical system of Figure 14(2), the distance from the position where the central ray of the image light passes through the polarization separation member 101B to the position where the image light forms the space-floating image 3B (L2 in the example of Figure 14(2)) is significantly longer than the distance from the position where the central ray of the image light is reflected by the polarization separation member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)) in the optical system of Figure 14(1).

[0233] 14(2), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0234] As described above, according to the optical systems of Fig. 14(1) and Fig. 14(2) in the second embodiment of the present invention, a more compact optical system can be realized. In particular, according to the optical system of Fig. 14(2), it is possible to increase the amount of the floating image projecting from the optical system, despite the more compact optical system.

[0235] When the optical system of Fig. 14(1) or Fig. 14(2) is incorporated into a space-floating image display device, it can be realized by replacing the optical system in the space-floating image display device described in Example 1 with the optical system of Fig. 14(1) or Fig. 14(2). Specifically, the optical system of Fig. 14(1) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4H, Fig. 4I, Fig. 4J, Fig. 4K, Fig. 4L, or Fig. 4M. In this case, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0236] Specifically, the optical system of Fig. 14(2) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4K, or Fig. 4L. In this case, it becomes possible to increase the amount of space-floating images projecting from the optical system. Also, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0237] Example 3 As a third embodiment of the present invention, a configuration example of the space floating image display device 1000 will be described. In the third embodiment, the basic configuration can be similarly applied to the space floating image display device 1000 described in each figure of the first embodiment or the second embodiment. In this embodiment, differences from the first embodiment or the second embodiment will be described, and repeated explanations of the same configuration as these embodiments will be omitted.

[0238] [Summary of Example 3] In a configuration such as the space floating image display device 1000, the aerial operation detection sensor 1351 performs detection along the imaging plane of the space floating image 3. The detection point of the aerial operation detection sensor 1351 may deviate from the coordinates intended by the user 230. To solve this problem, there is a technology such as that described in Reference 1. [Reference 1] JP 2023-6167 A

[0239] The technology disclosed in Reference 1 aims to calculate the accurate position of the finger pad by approximating the fingertip as a sphere when the detection surface of the mid-air operation detection sensor and the imaging plane of the floating image are not parallel. The technology disclosed in Reference 1 is effective only when the user's viewpoint is on the normal to the imaging plane of the floating image. In reality, the user's viewpoint is not always in an appropriate position for observation, and for more generalized coordinate correction, the viewpoint position must be taken into consideration. This example demonstrates a technology for achieving coordinate correction according to the user's viewpoint position that is different from the appropriate position, which cannot be handled by the technology disclosed in Reference 1. Therefore, compared to Reference 1, this example is more suitable because it takes the viewpoint position into consideration.

[0240] FIG. 15 shows an overview of the space-floating image display device 1000 of the third embodiment. FIG. 15 also shows an example of the configuration of the space-floating image display device 1000 applicable to the third embodiment. The space-floating image display device 1000 of FIG. 15 is an example of a configuration in which the configuration of FIG. 4O is changed to a vertical configuration based on the optical system of FIG. 2D. The upper part of FIG. 15 shows a YZ plane view of a system including the space-floating image display device 1000 (in other words, a space-floating image display system) as seen from the side (X direction) in space. The lower part of FIG. 15 shows an xz plane view of the space-floating image 3 and the finger 231 and the vicinity thereof as seen from above (y direction) in the coordinate system of the space-floating image 3, enlarging the area.

[0241] The coordinate system of the space is (X, Y, Z) as shown in the figure. The coordinate system of the floating in space image 3 is (x, y, z) as shown in the figure. The X-axis / X direction is the horizontal direction (first horizontal direction), the lateral direction, and the left-right direction as seen from the user 230. The Y-axis / Y direction is the horizontal direction (second horizontal direction), the depth direction, and the front-to-back direction as seen from the user 230. The Z-axis / Z direction is the vertical direction, the longitudinal direction, and the up-and-down direction as seen from the user 230. The x-axis / x direction is the horizontal direction and lateral direction within the screen of the floating in space image 3, and the left-to-right direction as seen from the user 230. The y-axis / y direction is the vertical direction and longitudinal direction within the screen of the floating in space image 3, and the up-and-down direction as seen from the user 230. The z-axis / z direction is the depth direction relative to the screen of the floating in space image 3, and the front-to-back direction as seen from the user 230.

[0242] In the configuration of FIG. 15, an aerial operation detection sensor 1351 and cameras 1180A and 1180B of the imaging unit 1180 are arranged on the transparent member 100 on the front side of the housing 1190. The optical axis of the aerial operation detection sensor 1351 faces the floating-in-space image 3. The optical axis of the camera 1180A faces the user 230 in the Y direction. The optical axis of the camera 1180B faces the floating-in-space image 3. The housing 1190 also includes a speaker 1140A corresponding to the audio output unit 1140 in FIG. 3, as well as physical operation buttons (not shown). Other components such as the control unit 1110 in FIG. 3 are connected to the components shown in the figure and are arranged inside or outside the housing 1190.

[0243] The mid-air operation detection sensor 1351 etc. detects the point where the finger 231 of the user 230 first comes into contact with the xy plane corresponding to the screen (display range 3R) of the floating in space image 3 as the position coordinates of the detection point.

[0244] In the configuration of Fig. 15, the retroreflective member 5 on the front surface of the housing 1190 is arranged to stand vertically (in the Z direction). In the configuration of Fig. 15, a real image, a floating-in-space image 3, is formed by image light emitted obliquely upward in the Y and Z directions from the retroreflective member 5 on the front surface. The xy plane of the floating-in-space image 3 is formed obliquely in space relative to the Y and Z directions. A user 230 views an operation target point P (for example, center point O) of the floating-in-space image 3 from a viewpoint position (observation reference point) 232. There is a line-of-sight direction 2302 when viewing. The user 230 performs a touch operation on the operation target point P as an air operation, for example, with the fingertip of a finger 231 of his right hand.

[0245] The xz plane diagram at the bottom is a schematic diagram showing the state when the fingertip of finger 231 approaches and comes into contact with operation target point P of floating-in-space image 3. As an example, it shows a case where the fingertip approaches center point O (origin O of the coordinate system) in the x direction of the screen (display range 3R) of floating-in-space image 3 as operation target point P0. At this time, there is direction 2303 as the direction in which the fingertip approaches operation target point P. Direction 2303 may be referred to as finger direction, approach direction, etc. Direction 2303 is, for example, roughly the same as line of sight direction 2302 from viewpoint position 232. Alternatively, direction 2303 is, for example, roughly the same as direction 2304 from the shoulder position described below. The direction 2303 of the fingertip relative to point P0 is, for example, roughly the same direction as the direction (z direction) of a normal 2305 to the xy plane of the floating in space image 3, or a direction having a certain angle relative to the direction (z direction) of the normal 2305. In this example, the direction 2303 of the fingertip relative to point P0 is shown as an oblique direction inclined at an angle α with respect to the normal 2305.

[0246] When a user 230 performs an operation in the air, there are factors such as the direction (line of sight direction 2302) and appearance when viewing operation target point P from viewpoint position 232, and the direction 2303 when finger 231 approaches operation target point P. Depending on these factors, when the fingertip touches operation target point P and is detected by the user operation detection mechanism, a deviation / error may occur between the detected point and the operation target point. In the case of point P0 shown in the figure, the point (sometimes referred to as the detected point) that is touched by the fingertip of the operating finger 231 and detected by, for example, the air operation detection sensor 1351 is point Q (particularly point Q0). Point Q0, which is the detected point, has a deviation / error of Δx in the x direction with respect to point P0, which is the target.

[0247] In this embodiment, the space-floating image device 1000 corrects this deviation (Δx) relative to the detection point (point Q0) and obtains the position coordinates of the corrected detection point corresponding to the target operation point (point P0). That is, the space-floating image display device 1000 has a correction function 1500 that performs such correction. The space-floating image display device 1000 grasps the viewpoint position 232 of the user 230 and the direction 2303 of the fingertip of the finger 231 as it approaches the plane of the space-floating image 3 based on the images captured by the cameras (1180A, 1180B) of the imaging unit 1180. The space-floating image display device 1000 calculates a correction amount corresponding to the deviation (e.g., deviation Δx) relative to the operation point based on the direction 2303 of the finger 231 and the shape of the fingertip. The space-floating image display device 1000 uses the correction amount corresponding to the deviation to correct the position coordinates of the detection point and obtains the corrected position coordinates of the detection point. Therefore, this embodiment is preferable to a device that does not consider the viewpoint position.

[0248] The above correction is also similar to the vertical direction (y direction) within the screen of the Space Floating Image 3. The above correction is also similar to points at other positions within the screen, such as point P1 on the right side of the center point O in the x direction.

[0249] In addition, in the third embodiment and the like, deviations in the x and y directions as in-plane directions on the xy plane of the screen of the floating image 3 are corrected, and deviations in the depth direction (z direction) of the screen are not handled.

[0250] [Correction method] An outline of the correction method by the space-floating image display device 1000 of this embodiment is as follows. In this embodiment, in Method 1 (fitting method) described below, an observation reference point 232, which is the viewpoint position 232 of the user 230, and the shape of the fingertip of a finger 231 approaching an operation target point P of the space-floating image 3 are captured based on analysis of an image from the imaging unit 1180. In this embodiment, as an approximate shape of the fingertip in Method 1, for example, parameter values ​​(such as the major axis and minor axis) of an ellipse approximation are calculated. In addition, this embodiment captures a direction 2303 when the fingertip approaches the operation target point P. This direction 2303 (finger direction / approach direction described below) corresponds to the angle made by the fingertip with respect to the normal direction or in-plane direction of the space-floating image 3 (finger angle / approach angle described below). More specifically, this direction 2303 may be a line of sight direction 2302 from a viewpoint position 232 in method A, which will be described later, or a direction 2304 from a shoulder position in method B.

[0251] This embodiment calculates a correction amount corresponding to the deviation / error (Δ) between the operation target point and the detection point based on the viewpoint position 232, parameter values ​​of the approximate ellipse of the fingertip shape, finger direction / finger angle, etc., in other words, depending on the user's observation direction, finger movement, posture, and other circumstances. This correction amount is an amount for correcting the deviation / error between the actual detection point detected by the user operation detection mechanism and the target operation target point. This embodiment can obtain the position coordinates of the corrected detection point corresponding to the operation target point by performing a calculation that reflects the correction amount in the position coordinates of the actual detection point. This embodiment can use the calculated correction amount as an offset for the actual detection point depending on the observation direction of the operation target point P on the screen of the floating-in-space image 3 from the user's viewpoint position 232, the direction of approach 2303 of the finger, etc. Therefore, this embodiment is preferable to one that does not consider the viewpoint position.

[0252] In this embodiment, the deviation / error between the intersection point (operation target point P) with the plane of the floating-in-space image 3 and the contact point (detection point Q) between the shape of the fingertip and the plane is corrected according to the finger direction 2303 (straight line 3410, which is an extension line in FIG. 34 ) and angle (θ, φ) of the finger 231, which is the user's operating finger. More specifically, in this embodiment, the correction amount is calculated using the user's reference position (e.g., viewpoint position 232) captured based on the camera image of the imaging unit 1180, parameter values ​​(major axis a, etc.) of the approximate ellipse 501 of the fingertip shape, the finger direction 2303 according to the line of sight direction 2302 / arm direction 2304, etc., and the angle (θ, φ) made by the straight line 3410 of the finger direction 2303 and the plane of the screen. In this embodiment, the correction amount between the detection point Q and the operation target point P is calculated as the number of pixels based on these values, and this correction amount is used as the offset.

[0253] [Configuration example (2) of the space floating image display device of the third embodiment] FIG. 16 shows another example configuration (second example configuration) of the space-floating image display device 1000 applicable to the third embodiment. The space-floating image display device 1000 of FIG. 16 is based on the optical system of FIG. 2A and is an example configuration similar to the horizontal configuration of FIG. 4A, for example. FIG. 16 shows a YZ plane view of the space-floating image display system as seen from the side (X direction) in space. In the configuration of FIG. 16, the space-floating image 3 is formed obliquely at a position above the transparent member 100 on the top surface of the housing 1190. In the configuration of FIG. 16, as an example of the arrangement of the cameras of the imaging unit 1180, the camera 1180A is arranged at a position above the space-floating image 3, similar to FIG. 15. The camera 1180B is arranged at the same position as the aerial operation detection sensor 1351.

[0254] [Configuration example (3) of the space floating image display device of the third embodiment] FIG. 17 shows another example configuration (third example configuration) of the space-floating image display device 1000 applicable to the third embodiment. The space-floating image display device 1000 of FIG. 17 is based on the configuration of FIG. 4C etc., and the display device 1 is arranged on the upper side in the vertical direction (Z direction) in a vertical housing 1190. FIG. 17 shows a YZ plane view of the space-floating image display system as seen from the side (X direction) in space. In the configuration of FIG. 17, the space-floating image 3 is formed to stand upright on a stage (lower housing) 1190S by image light emitted in the horizontal direction (Y direction) from the transparent member 100 on the front of the housing 1190. In the configuration of FIG. 17, the housing 1190 also has a stage (lower housing) 1190S and a roof (upper housing) 1190T that protrude from the front. In the configuration of FIG. 17, an aerial operation detection sensor 1351 and a camera 1180B are arranged facing downward within the roof (upper housing) 1190T. Furthermore, camera 1180A is placed at a position above space floating image 3, facing towards space floating image 3 and user 230 in front of it.

[0255] [Each method of Example 3] FIG. 18 is a table summarizing the configuration outline of the system of the space floating image display device 1000 of each embodiment belonging to Example 3 and Example 4 described later. As shown in the figure, embodiments are possible by combining various systems. The table of FIG. 18 shows the configuration of the combination of which of the various systems that are components are to be adopted for each detailed embodiment. The systems that can be selected as components in the combination are as follows. For convenience, they are referred to as System 1, etc. The table of FIG. 18 has columns of "Example," "System 1 / System 2," "System A / System B," and "System X / System Y." Each row shows an embodiment (including modified examples) by combining systems. Each individual embodiment is also referred to as Example 3AX, etc.

[0256] Method 1 is a fitting method using ellipse approximation. Example 3 (Example 3AX, etc.) is an example that uses the fitting of Method 1. Method 1 performs a fitting process to approximate the shape of a fingertip to an ellipse from an image of a user's finger photographed and scanned with a camera. Method 1 uses parameter values ​​obtained as a result of the fitting, that is, parameter values ​​that constitute the approximate ellipse of the fingertip. Method 1 also uses the direction and angle of the fingertip (such as the finger direction described below) when touching the operation target point as an air operation from the observation reference position of the user's body (for example, the viewpoint position in Method A, or the shoulder position in Method B). Method 1 calculates a correction amount for the deviation and error between the operation target point on the plane of the floating image 3 and the actual detection point based on the parameter value and the direction and angle of the fingertip. Method 1 can achieve high accuracy by performing a fitting calculation process every time.

[0257] In Method 1 and Method 2, the space-floating image display device 1000 obtains images from the camera of the imaging unit 1180, and camera images of the user's body, such as the face, torso, shoulders, arms, etc. The space-floating image display device 1000 may grasp the user's reference position (e.g., viewpoint position 232), attributes (e.g., gender), and physical characteristics (e.g., height, dominant hand, etc.) based on recognition processing and analysis processing of the face and other images captured in the camera image. The space-floating image display device 1000 grasps at least the user's viewpoint position 232 (FIG. 15). The viewpoint position 232 corresponds to the observation reference point 232 of the space-floating image 3.

[0258] As a variant, in Method 1 and Method 2, it may be determined from the camera image whether Method A or Method B is more suitable for the finger direction 2303 of the user's fingertip, and one of them may be selected (described later).

[0259] Method 2 is a correspondence table method. Example 4 (Example 4AX, etc.) described later is an example that uses the correspondence table of Method 2. Method 2 is a method that refers to a preset correspondence table and refers to a correction amount according to the user's position / attributes / characteristics or finger direction 2303, etc. In Method 2, for the calculation of ellipse approximation fitting of the fingertip in Method 1, various information is associated and input / set in the correspondence table in advance based on the results of statistical processing of data of multiple people or publicly known statistical information. In the correspondence table, statistical values, for example, average values, are set in advance as correction amounts according to the user's position / attributes / characteristics, finger direction, etc. In other words, the correspondence table of Method 2 is a lookup table or the like.

[0260] In method 2, the correction amount is obtained as an output value from the correspondence table based on input values ​​(for example, viewpoint position, finger direction, etc.) according to the user. This makes it possible to obtain the correction amount during use without having to perform calculations such as fitting an ellipse approximation of the fingertip. Method 2 can omit the calculation process for fitting each time, thereby achieving efficiency. Therefore, this embodiment is preferable to methods that do not consider viewpoint position.

[0261] As an example of how to create the correspondence table, information such as the user's finger direction obtained by the processing of method 1 and parameter values ​​of the ellipse approximation fitting of the fingertip may be associated and stored in memory, and the average value of the parameters may be calculated for each attribute or characteristic, and the average value may be set in the correspondence table. As another example of how to create the correspondence table, the correspondence table may be set using statistical information on the characteristics of the human body, without using the results of the processing of method 1.

[0262] The correction amount in method 1 etc. is an amount for correcting the deviation / error between the operation target point P as seen by the user 230 subjectively on the xy plane of the floating-in-space image 3 in FIG. 15 and the actual detection point by the user operation detection mechanism (for example, the mid-air operation detection sensor 1351). The operation target point P is, in other words, the target position coordinates that the user 230 intends to operate. The actual detection point is detected by the user operation detection mechanism as position coordinates that may have deviation / error from the operation target point on the plane. If there is such deviation / error, the detection point is a position not intended by the user 230. As will be described later, the actual detection point corresponds to the position coordinates where the fingertip first touches the plane of the floating-in-space image 3. By approximating the shape of the fingertip to an ellipse, the deviation / error between the detection point and the operation target point can be captured with higher accuracy.

[0263] The space-floating image display device 1000 obtains the corrected position coordinates of the detection point by performing a calculation that reflects the correction amount obtained by method 1 or method 2 in the position coordinates of the actual detection point. The space-floating image display device 1000 can use the corrected position coordinates of the detection point to perform a predetermined process, such as determining whether or not a touch operation has been performed on the object image in the space-floating image 3.

[0264] Method A and Method B are two methods regarding how to define the direction of approach / contact of the fingertip to the operation target point on the plane of the floating image 3 in space (finger direction 2303 in FIG. 15).

[0265] In method A, angles θ and φ corresponding to the finger direction 2303 are found using a line of sight direction 2302 from a viewpoint position 232 as a finger direction 2303, and the amount of correction is calculated from the input values ​​of the angles θ and φ. In method B, angles θ' and φ' corresponding to the finger direction 2303 are found using a direction 2304 from a shoulder position 233 as a finger direction 2303, and the amount of correction is calculated from the input values ​​of the angles θ' and φ'.

[0266] In Method A, the direction of the fingertip approaching the operation target point P on the plane of the floating image 3 from the viewpoint position 232 (FIG. 15), which is the observation reference point 232 of the user 230, is used as this finger direction 2303. In Method A, the viewpoint position 232 is used as the reference position of the user's body. In other words, the reference position is a position that serves as a reference or starting point for defining the finger direction 2303. The viewpoint position 232 may be, for example, the center position of both eyes on the face. In Method A, the finger direction 2303 is the direction of a straight line from the viewpoint position 232 to the operation target point P, and this direction roughly coincides with the gaze direction 2302. In other words, in Method A, the direction in which the finger 231 approaches the plane is considered to roughly coincide with the gaze direction 2302 from the viewpoint position 232 to the operation target point P, and the finger direction 2303 is defined using the gaze direction 2302.

[0267] In Method B, the direction (sometimes referred to as arm direction) of the fingertip when approaching the operation target point P on the plane of the floating in space image 3 from the shoulder position of the user 230 via the arm is used as the finger direction 2303. In Method B, the shoulder position, or the center of the shoulder joint described below, is used as the reference position of the user's body. In Method B, the direction 2304 of the straight line when the fingertip approaches the operation target point P on the screen from the shoulder position is used as the finger direction 2303. In Method B, it is assumed that the direction in which the finger 231 approaches the plane roughly coincides with the direction 2304 from the shoulder position to the operation target point P (FIG. 15), and this direction 2304 is used to define the finger direction 2303.

[0268] In method B, the position of the arm or shoulder on either the left or right side of the trunk such as the torso is used as the reference position of the user's body, rather than the viewpoint position 232. For example, the position of the arm or shoulder that can be recognized from a camera image can be used. Because the arms and shoulders are located on the left and right sides of the trunk such as the torso due to the physical structure, the positions of the arms and shoulders are shifted to the left or right with respect to the viewpoint position 232. The position of the shoulder may be considered as the origin of the upper arm or the position of the shoulder joint. The position of the arm may be considered as the origin of the forearm or the position of the elbow joint. In this embodiment, a case will be described in which the shoulder joint center is used as the shoulder position, but this is not limiting.

[0269] In Method A and Method B, the amount of correction is calculated using the angle corresponding to the finger direction 2303 relative to the screen. The angle that the finger direction 2303 makes with respect to the normal from the operation target point P on the xy plane of the screen of the floating image in space 3 is also referred to as the finger angle or approach angle. Different amounts of correction can be calculated depending on the finger angle.

[0270] Method X and Method Y are two different methods for the reference position of the screen of the space floating image 3 regarding correction.

[0271] Method X is a method that calculates the amount of correction for the deviation between the operation target point P and the detection point, using the position of, for example, the center point O (FIG. 15) on the xy plane of the screen (display range 3R) of the space floating image 3 as the reference position. Method X is a method that similarly applies the amount of correction at the center point O, which is the reference position, to each point on the screen other than the center point O.

[0272] Method Y is a method in which the amount of correction is calculated for each point on the xy plane of the screen (display range 3R) of the floating in space image 3, and the amount of correction can be different for each point. In method Y, the amount of correction is calculated for each position of the operation target point P on the screen of the floating in space image 3, corresponding to the finger direction 2303 and finger angle, which can vary.

[0273] In FIG. 18, roughly speaking, Example 3 is an example in which the fitting of Method 1 is applied, and Example 4 is an example in which the correspondence table of Method 2 is applied. As Example 3, Example 3AX in particular is a combination in which Method A and Method X are applied to Method 1. In particular, Example 3AY is a combination in which Method A and Method Y are applied to Method 1. In particular, Example 3BX is a combination in which Method B and Method X are applied to Method 1. In particular, Example 3BY is a combination in which Method B and Method Y are applied to Method 1. As Example 4, Example 4AX in particular is a combination in which Method A and Method X are applied to Method 2. In particular, Example 4AY is a combination in which Method A and Method Y are applied to Method 2. In particular, Example 4BX is a combination in which Method B and Method X are applied to Method 2. In particular, Example 4BY is a combination in which Method B and Method Y are applied to Method 2.

[0274] The space-floating image display device 1000 has an implementation corresponding to at least one embodiment or modification of Fig. 18. The space-floating image display device 1000 may have an implementation corresponding to a plurality of embodiments or modifications of Fig. 18. In that case, the space-floating image display device 1000 may have a plurality of modes corresponding to a plurality of embodiments, etc., and may be controlled to switch the applied mode according to system settings or user settings, etc.

[0275] [Basic configuration example of a space floating image display device] Fig. 19 shows an example of the basic configuration of a space-floating image display device 1000 of Example 3. A detailed configuration example will be described later. The space-floating image display device 1000 of Fig. 19 includes an imaging unit 1180, a correction unit 1900, an image processing unit 1901, a display unit 1902, an optical system 1903, and a user operation detection mechanism 1904. Note that the correction unit 1900 may be implemented integrally with components such as the image processing unit 1901.

[0276] The image processing unit 1901 corresponds to the control unit 1110, the image control unit 1160, or the mid-air operation detection unit 1350 in FIG. 3. The image processing unit 1901 controls the display unit 1902 (for example, the image display unit 1102 in FIG. 3, the display device 1 in FIG. 2A, etc.) based on the display target data to display an image on the screen of the display unit 1902. The display unit 1902 emits image light based on the image on the screen. The optical system 1903 guides the image light and performs magnification and optical adjustment. An example of the optical system 1903 is the retroreflector 1101 in FIG. 3. The image light from the optical system 1903 forms the space-floating image 3 at a predetermined position. In addition, the image processing unit 1901 controls the user operation detection mechanism 1904, and the user operation detection mechanism 1904 detects an mid-air operation (for example, a touch operation) by the finger 231 of the user 230 on the screen of the space-floating image 3. An example of the user operation detection mechanism 1904 is the mid-air operation detection sensor 1351 in FIG.

[0277] The correction unit 1900 is provided as a functional block behind the imaging unit 1180. Based on an image 1910 captured by the camera of the imaging unit 1180, the correction unit 1900 calculates and obtains a correction amount corresponding to the deviation / error between the detection point detected by the user operation detection mechanism 1904 on the screen of the space floating image 3 and the operation target point P. The correction unit 1900 also receives detection information (position coordinates of the detection point) from the user operation detection mechanism 1904 or the image processing unit 1901. The correction unit 1900 obtains the position coordinates of the detection point after correction by performing a calculation that reflects the correction amount in the position coordinates of the detection point. The correction unit 1900 can output information 1911 of the position coordinates of the detection point after correction (or the correction amount) to the image processing unit 1901. The image processing unit 1901 uses the information 1911 of the position coordinates of the detection point after correction to perform predetermined processing such as determining a touch operation.

[0278] 19 shows a case where the correction unit 1900 calculates up to the position coordinates of the detection points after correction, but this is not limiting, and the correction unit 1900 may calculate and output up to the correction amount, and the image processing unit 1901 or the like may use the correction amount to obtain the position coordinates of the detection points after correction. Furthermore, the correction unit 1900 or the image processing unit 1901 or the like may output information on the correction amount or the position coordinates of the detection points after correction to the outside.

[0279] In this embodiment, the correction unit 1900 is implemented in the space-floating image display device 1000. In a modified example, the correction unit 1900 may be implemented as a correction device connected to the outside of the space-floating image display device 1000. Moreover, the correction unit 1900 may be implemented as a single functional block integrated with the imaging unit 1180. Moreover, the correction unit 1900 may be implemented as a single functional block integrated with the user operation detection mechanism 1904.

[0280] [Example of correction unit configuration: Method 1] Fig. 20A shows a detailed configuration example of the correction unit 1900 in Fig. 19. Fig. 20A shows a configuration example for Method 1. The correction unit 1900 has a fitting processing unit 2001, a parameter calculation unit 2002, a correction amount calculation unit 2003, a detection coordinate correction unit 2004, and a memory 2005.

[0281] Before the user 230 performs an operation in the air on the floating-in-space image 3, the floating-in-space image display device 1000 photographs and scans the finger 231 or the like with the camera of the imaging unit 1180 to obtain an image showing the fingertip. The imaging unit 1180 sends an image 1910 showing the fingertip or the like to the correction unit 1900. The image 1910 here particularly includes a finger image 2011 showing the fingertip of the user 230 and a face image 2012 showing the face of the user 230 or the like. The finger image 2011 is obtained by, for example, the camera 1180B in FIG. 15. The face image 2012 is obtained by, for example, the camera 1180A in FIG. 15. Note that the imaging unit 1180 is not limited to one, and there may be multiple imaging units, as in the example of the camera 1180A and the camera 1180B. The camera that obtains the finger image 2011 and the camera that obtains the face image 2012 may be integrated or separate.

[0282] The fitting processing unit 2001 receives an input of a finger image 2011. The parameter calculation unit 2002 receives an input of a face image 2012. The fitting processing unit 2001 performs a fitting process using elliptical approximation based on the finger image 2011. In this embodiment, the fitting is performed using elliptical approximation. This allows for obtaining fitting parameter values ​​corresponding to the shape of the fingertip, in other words, parameter values ​​of elliptical approximation. Parameter information 2013 obtained by the fitting processing unit 2001 is sent to the parameter calculation unit 2002.

[0283] The parameter calculation unit 2002 performs a predetermined calculation using the face image 2012 and fitting parameter information 2013. First, the parameter calculation unit 2002 determines an observation reference position as the position of the user 230 relative to the space floating image 3 based on an analysis of the face image 2012. This observation reference position is the viewpoint position 232 described above, or the like. Furthermore, the parameter calculation unit 2002 determines and estimates the attributes (e.g., gender) or physical characteristics of the user 230 based on face recognition processing of the face image 2012. Note that the processing for determining the observation reference position and attributes / characteristics as described above may be performed by a processing unit separate from the parameter calculation unit 2002 (FIG. 20B).

[0284] 20B, a user position ascertaining unit 2021 and a user attribute / characteristic recognition unit 2022 are provided between the imaging unit 1180 and the parameter calculation unit 2002. Memory 2005 is omitted in FIG. 20B. The user position ascertaining unit 2021 ascertains the observation reference position (e.g., viewpoint position 232) of the user 230 based on the face image 2012, and sends the user position information to the parameter calculation unit 2002. The user attribute / characteristic recognition unit 2022 ascertains and estimates the attribute or characteristic of the user 230 based on the face image 2012, and sends the user attribute / characteristic information to the parameter calculation unit 2002.

[0285] The parameter calculation unit 2002 calculates predetermined parameter values ​​such as finger direction 2303 (FIG. 15) according to fitting parameter information 2013, the observation reference position of the user 230 (e.g., viewpoint position 232), and the attributes / characteristics of the user 230. The parameter calculation unit 2002 outputs calculation result information 2014 including the predetermined parameter values.

[0286] The correction amount calculation unit 2003 calculates a correction amount for correcting the deviation between the operation target point and the detection point according to a parameter value of the calculation result information 2014, for example, the finger direction 2303. The correction amount calculation unit 2003 outputs correction amount information 2015. This correction amount is obtained as a correction pixel number corresponding to the pixel configuration on the screen of the display unit 1902 corresponding to the screen of the space floating image 3.

[0287] The detection coordinate correction unit 2004 performs a calculation to correct the position coordinates of the detection point detected by the user operation detection mechanism 1904 using correction amount information 2015 from the correction amount calculation unit 2003, and obtains the corrected position coordinates of the detection point. This calculation may include a calculation to add the correction amount as an offset to the position coordinates of the detection point. As a result of the calculation by the detection coordinate correction unit 2004, information 2016 of the corrected position coordinates of the detection point is obtained as output information 2016.

[0288] 19 can then execute a predetermined process using the corrected position coordinate information 2016 of the detection point. For example, the image processing unit 1901 can use the corrected information 2016 instead of the pre-correction detection information 1914 to determine whether or not a touch operation has been performed on an object image in the space floating image 3, and can perform a process corresponding to a predetermined function associated with the object image according to the determination result.

[0289] In the configuration of FIG. 20A , various data and information (e.g., fitting parameter information 2013, calculation result information 2014 including predetermined parameter values, and correction amount 2015) are stored in memory 2005. In other words, memory 2005 is a parameter storage unit. Various data and information may be stored in association with each other in memory 2005. For example, the observation reference position and attribute / characteristic information of user 230, fitting parameter information 2013, and correction amount may be stored as a set in memory 2005. By making memory 2005 a nonvolatile storage device, the data and information may be retained for a long period of time. This makes it possible to obtain data for taking statistics corresponding to method 2 from memory 2005.

[0290] In the configuration of FIG. 20A, emphasis is placed on real-time detection, and a series of processes up to correction of the detected point in the detected coordinate correcting unit 2004 are immediately executed, but the configuration is not limited to this.

[0291] [Example of correction unit configuration: Method 2] Fig. 21 shows a detailed configuration example of the correction unit 1900 in Fig. 19 in the case of Method 2. In Fig. 21, the correction unit 1900 has a user position grasping unit 2021, a user attribute / characteristic recognition unit 2022, a parameter calculation unit 2002, a correspondence table 2100 in a memory 2005, a detected coordinate correction unit 2004, and a correspondence table setting unit 2101. In Fig. 21, the fitting processing unit 2001 and the correction amount calculation unit 2003 in Figs. 20A and 20B are not necessary.

[0292] In a correspondence table 2100 (in other words, a parameter correspondence table) of the memory 2005, parameter values ​​including correction amounts are input and set as various parameter values ​​(for example, average values) obtained in advance based on statistical processing or the like.

[0293] In method 2, the face image 2012 in particular of the video 1910 is used to grasp the observation reference position and attributes / characteristics of the user 230, and the parameter calculation unit 2002 grasps the finger direction 2303 and the like. Note that in method 2, the finger image 2011 is not essential, but may be input and used.

[0294] 21 is, in other words, a correction amount reference unit. The parameter calculation unit 2002 searches the correspondence table 2100 in the memory 2005 based on information 2014 such as the grasped finger direction 2303. In other words, the memory 2005 or the detected coordinate correction unit 2004 searches the correspondence table 2100 in the memory 2005 based on information 2014 such as the finger direction 2303 from the parameter calculation unit 2002. Here, the information 2014 such as the finger direction 2303 is an input value of the correspondence table 2100. As a result of searching the correspondence table 2100, correction amount information 2015 is obtained as an output value. The detected coordinate correction unit 2004 performs a calculation to correct the position coordinates of the detected points using the correction amount information 2015, and outputs information 2016 of the position coordinates of the corrected detected points.

[0295] Furthermore, the set values ​​of the correspondence table 2100 may be fixed, but are not limited to this, and the set values ​​of the correspondence table 2100 may be variably set using the correspondence table setting unit 2101. When setting this correspondence table 2100, a screen with a dedicated GUI for the space floating image 3 may be provided. The contents of the correspondence table 2100 are displayed on this screen.

[0296] [Method 1: Fingertip scanning and guidance] In Example 3 using Method 1, in order to suitably achieve ellipse approximation fitting of the fingertip, the camera of the imaging unit 1180 is arranged so as to obtain an image that clearly captures the fingertip of the finger 231, which is the operating finger of the user 230 (see FIG. 15, etc.). In other words, guide, guidance, etc. may be provided so that the camera of the imaging unit 1180 is in a state where the operating finger 231 is clearly captured. In this example, as the guide, guidance, a user interface and time are provided so that the operating finger of the user 230 is held over the camera of the imaging unit 1180. This user interface is accompanied by a GUI and audio output in the space-floating image 3. Following this guide, the user 230 holds their fingertip over a predetermined position, etc. The space-floating image display device 1000 obtains a suitable image that clearly captures the fingertip using the camera of the imaging unit 1180 during this time. In other words, the space-floating image display device 1000 scans the fingertip during this time. The space floating image display device 1000 can achieve correction with higher accuracy based on this suitable camera image.

[0297] As a variant, the use of a sensor capable of obtaining three-dimensional / stereoscopic information about the fingertip, such as LiDAR (Light Detection and Ranging), may be applied to scan the fingertip, rather than being limited to a camera that obtains two-dimensional images.

[0298] Fig. 22A shows an example of providing a specific GUI in the floating image 3 in order to scan and guide the fingertip in the case of Method 1. In Fig. 22A, an image of the GUI for guiding and directing the fingertip scan is displayed on the screen (display range 3R) of the floating image 3 in space. The audio of the guide may be output together with the display of the guide.

[0299] First, FIG. 22A shows a first example of a guide. The first example corresponds to a case where a camera for capturing a finger image 2011 and a camera for capturing a face image 2012 are respectively provided. A designated area 2201 is displayed at a predetermined position on the screen of the floating image 3. For example, a guide message 2202 such as "Coordinate adjustment will be performed. Please bring your finger, which controls the floating image, closer to the designated area" is displayed. An arrow image accompanying the message 2202 points to the designated area 2201. The message 2202 and the like prompt the user 230 to bring their finger 231 closer to the screen. Following the guidance, the user 230 brings their finger 231 closer to the designated area 2201 in the forward / backward and up / down / left / right directions relative to the screen.

[0300] The space-floating image display device 1000 uses the camera of the imaging unit 1180 to capture an image of the finger 231 that has been brought close to the designated area 2201. In detail, the space-floating image display device 1000 detects the position coordinates of the finger 231 when it approaches the designated area 2201 using, for example, the mid-air operation detection sensor 1351, which is a non-contact sensor of the user operation detection mechanism 1904. Then, this coordinate detection serves as a trigger to start photographing, scanning, and acquiring the finger image 2011 by the camera of the imaging unit 1180.

[0301] At this time, the imaging unit 1180 and a subsequent processing unit (for example, the correction unit 1900 in FIG. 20A) may determine / estimate which of the fingers 231 (right hand fingers / left hand fingers) on the left or right arm of the user 230 is being photographed and scanned. In other words, the dominant hand of the user 230 regarding the aerial operation may be determined / estimated. The obtained dominant hand information may be used for the correction direction in the correction by the correction unit 1900, etc.

[0302] FIG. 22B shows a second example of a guide. This second example corresponds to the case where a finger image 2011 and a face image 2012 are captured by a single camera of the imaging unit 1180. A guide message 2203 such as "Coordinate adjustment will be performed. Please bring your finger, which controls the aerial image, closer to the front camera" is displayed on the screen of the floating image 3. An arrow image accompanying the message 2203 points in the direction of the front camera. In this example, the front camera of the imaging unit 1180 is located above the screen (not shown). The optical axis of the front camera passes above the screen. The message 2203 and the like prompt the user 230 to move their finger 231 closer to the front camera. Following the guidance, the user 230 moves their finger 231 closer to the front camera, which is located above the screen.

[0303] The space-floating image display device 1000 uses the front camera to capture an image of the finger 231 that is brought close to the front camera. In detail, the space-floating image display device 1000 detects the distance between the finger 231 and the front camera, and when the detected distance information becomes equal to or less than a certain value, starts capturing, scanning, and acquiring the finger image 2011 by the front camera of the imaging unit 1180. At this time, the dominant hand may be determined as in the first example.

[0304] In other embodiments, instead of providing a specific GUI for the above-described guide and guidance, images of the fingers and other objects captured by the camera during normal operation of the user 230 may be used for appropriate correction.

[0305] [Method 1: Ellipse fitting] Fig. 23 is an explanatory diagram of fitting by ellipse approximation for finger 231 of user 230 by fitting processing unit 2001 of Fig. 20A in the case of method 1. Fig. 23 shows an enlarged view of fingertip 231A of finger 231, which is the operating finger, approaching operation target point P on the xy plane of the screen of floating in space image 3. Fig. 23 shows an xz plane view of the xy plane of floating in space image 3 seen from above (y direction). The operating finger is finger 231 that performs an operation in the air on the screen, and in this example, it is the index finger of the right hand.

[0306] The correction unit 1900 of the space floating image display device 1000 acquires an image of the operating finger as a camera image. The image of the operating finger corresponds to a finger image 2011 in which a fingertip 231A is captured. As the image of the operating finger, an image (particularly an image on the xz plane) including a portion 2300 enclosed by a dashed line frame in FIG. 23 is obtained. The image of portion 2300 in FIG. 23 corresponds to a partially enlarged view of the finger image 2011 in which the fingertip 231A of the operating finger is captured. In this example, the direction of the optical axis of the camera (for example, camera 1180B in FIG. 15) corresponds to the vertical direction (y direction) of the space floating image 3. A point CC is the center point in the camera image, and its position coordinates are (xc, zc).

[0307] The fitting processing unit 2001 of the correction unit 1900 performs an ellipse approximation fitting process based on the image of the operating finger. Corresponding to the xy plane of the screen (display range 3R) of the floating-in-space image 3, the horizontal direction within the screen is the x-axis, the vertical direction within the screen is the y-axis, and the depth direction is the z-axis. Point O is the origin, which is the center point of the screen, and its position coordinates are (x0, z0). Operation target point P is a target point that the user perceives as wanting to perform a touch operation when viewed from the viewpoint position 232, and its position coordinates are (x1, z1). Point Q is the actual detection point detected by the mid-air operation detection sensor 1351 when the fingertip 231A first touches the xy plane, and its position coordinates are (x2, z2).

[0308] As an example of ellipse approximation fitting, ellipse approximation by the least squares method can be applied. Ellipse approximation by the least squares method is as follows. As shown in the figure, with respect to the coordinates (xc, zc) of the center CC of the camera image and fixed x, y coordinates, the center 502 of the approximate ellipse 501 of the shape of the fingertip 231A, in other words, the ellipse center 502, is set to point D, and the position coordinates are set to (xd, zd). The major axis of the approximate ellipse 501 of the surface of the fingertip 231A is set to a, and the minor axis is set to b. The angle between the direction of the major axis a and the x-axis is set to θ. The direction of the major axis a corresponds to the finger direction 2303. Furthermore, the angle α is set to the angle that the direction of the major axis a (finger direction 2303) makes with respect to the normal 2305 of the screen of the floating in space image 3. In this example, the finger direction 2303, which is the direction of the major axis a, is a direction in which an extension line passes through the operation target point P.

[0309] The point on the approximated ellipse 501 that corresponds to the outline of the fingertip 231A is defined as point E, and its position coordinates are defined as (xi, zi). The approximated ellipse 501 is illustrated as a dashed ellipse that overlaps with the outline of the fingertip 231A. The sum of squares of the distance between point E(xi, zi) on the approximated ellipse 501 and the point on the approximated ellipse 501 can be calculated using Equation 1 in the figure. Using the least squares method, parameter values ​​such as the ellipse center 502 (xd, yd), angle θ, major axis a, and minor axis b that minimize the value of this sum of squares can be determined.

[0310] [Method 1: Skeleton estimation model] FIG. 24 is an explanatory diagram of a skeleton estimation model of the body of user 230 in the case of method 1. The correction unit 1900 in FIG. 20A and other figures performs processing based on such a skeleton estimation model. The body of user 230, who is in front of the camera of the imaging unit 1180, is captured by the camera. The camera image capturing the body is defined as body image 2400. The body image 2400 includes at least the face image 2012 described above. The parameter calculation unit 2002 and other units in FIG. 20A apply the skeleton estimation model to the body (at least the head and face 234) captured in the body image 2400.

[0311] From the face image 2012, both left and right eyes 235 (right eye and left eye) in the head and face 234 can be detected. The midpoint between the eyes 235 can be detected as the observation reference point 232 (in other words, the viewpoint position 232). This observation reference point 232 is used as the reference position in Method A.

[0312] In this embodiment, shoulders 242 and arms 241 (241a, 241b) on the left and right sides of a body trunk 240 (chest, torso, etc.) are considered. Specifically, the arms 241 include the forearms and upper arms. A shoulder joint center (shoulder position) 233 is considered to be the point where the arms 241 (241a, 241b) connect to the shoulders 242. The correction unit 1900 can estimate and detect the shoulder joint centers 233 (233a, 233b) from the body image 2400 based on a skeleton estimation model. This shoulder joint center 233 is used as a reference position in Method B.

[0313] Furthermore, in the body image 2400 and the skeleton estimation model, it is possible to estimate the distance and displacement in the horizontal direction (here, the X-axis) and vertical direction (here, the Z-axis) from the viewpoint position 232, which is the observation reference point 232, to the shoulder joint center 233 and the finger 231. For example, height h1 is the height distance from the ground 2500 to the observation reference point 232 in the vertical direction (Z-axis). Height h2 is the height distance from the ground 2500 to the shoulder position 231, which is the shoulder joint center 233, in the vertical direction (Z-axis). Width w1 is the distance from the observation reference point 232 to, for example, the right shoulder joint center 233a in the horizontal direction (X-axis). Width w2 is the distance from the observation reference point 232 to, for example, the right shoulder joint center 233a in the vertical direction (Z-axis). Furthermore, width (length) w3 is the distance, for example, from the right shoulder joint center 233a to the finger 231a of the right hand in the X-axis and Z-axis.

[0314] In this example, the arm 241 is shown as a straight line from the shoulder position 233 to the finger 231, and the forearm and upper arm are not separated, but the forearm and upper arm may be separated and the elbow joint may also be used to calculate the parameter values. Also, without using a skeleton estimation model of the body image 2400, for example, only the height h1 of the observation reference point 232 may be used as a representative value, and parameter values ​​such as a statistical average value according to the height h1 (e.g., h2, w1, w2, w3, etc.) may be referenced.

[0315] [Method A: Finger direction (y-axis)] 25A and 25B are explanatory diagrams relating to the relationship between the observation reference point 232 and the finger direction 2303 of the finger 231, which is the operating finger, based on the skeleton estimation model of Fig. 24, etc. Fig. 25A illustrates, as a first example, the finger direction 2303 in the case of Method A in a YZ plane view. Fig. 25B illustrates, as a second example, the finger direction 2303 in the case of Method B in a YZ plane view.

[0316] 25A, it is assumed that user 230 is viewing an object image located at center point O (indicated by a black circle) of the screen (display range 3R) of the Space Floating Image 3 as operation target point 2301(P) from viewpoint position 232, which is observation reference point 232 corresponding to eye 235. The line of sight from observation reference point 232 to operation target point 2301(P) is shown as line of sight direction 2302. In this example, line of sight direction 2302 is roughly the same as normal direction 2305 to the screen (display range 3R) of the Space Floating Image 3. Normal direction 2305 is a direction that coincides with the optical axis direction of image light a1 that forms the Space Floating Image 3.

[0317] In the state of FIG. 25A , the tip of the finger 231, which is an operating finger, is approximately aligned with the position of the operation target point 2301(P). At this time, the direction 2303 indicated by the large arrow is the finger direction 2303 when the finger 231 approaches the operation target point 2301(P), in other words, the approach direction. In this example, this finger direction 2303 roughly coincides with the line of sight direction 2302 and the normal direction 2305 from the observation reference point 232. In addition, the direction from the shoulder joint center 233 to the position of the operation target point 2301(P) at the fingertip is illustrated as a direction (arm direction) 2304. In this example, the finger direction 2303 differs from the direction 2304 from the shoulder.

[0318] 25A, in method A, when calculating a finger direction 2303 when a finger 231 approaches an operation target point 2301(P), a gaze direction 2302 from the observation reference point 232 to the operation target point 2301 at the fingertip is used. That is, in method A, the finger direction 2303 is considered to be approximately the same as the gaze direction 2302.

[0319] 25B is different from FIG. 25A in that the finger direction 2303 when the finger 231 approaches the operation point 2301(P) roughly coincides with the direction (arm direction) 2304 from the shoulder joint center 233, and differs from the line of sight direction 2302. In a case like FIG. 25B, as method B, the direction 2304 from the shoulder joint center 233 is used when calculating the finger direction 2303 when the finger 231 approaches the operation point 2301. For the direction 2304 from the shoulder joint center 233, widths w1, w2, w3, etc. can be used as in FIG. 24.

[0320] 25A etc., the angle formed by the line of sight 2302 (method A) as the finger direction 2303 with respect to the normal 2305 from the operation target point 2301 on the xy plane of the floating in space image 3, in other words the angle (θ, φ described below) when observing the operation target point 2301 from the viewpoint position 232, may be referred to as the finger angle, approach angle or observation angle. Since Fig. 25A etc. is a YZ plane view, the angle of the fingertip with respect to the vertical direction (y-axis) of the plane of the floating in space image 3 is shown, but the same applies to the angle of the fingertip with respect to the horizontal direction (x-axis) of the plane of the floating in space image 3.

[0321] 25A enlarges the area around the finger 231, and illustrates the angle φ that the finger direction 2303 makes with respect to the vertical direction (y direction) on the plane of the floating in space image 3 when the fingertip approaches. In this example, this angle φ is approximately 90 degrees. The angle that the fingertip direction 2303 makes with respect to the normal direction 2305 is illustrated as angle (90°-φ). In this example, this angle (90°-φ) is approximately 0 degrees. Similarly, in FIG. 25B, angle φ' is approximately 60 degrees, and angle (90°-φ') is approximately 30 degrees. In FIG. 25B, the angle corresponding to angle φ in FIG. 25A is referred to as angle φ' for the sake of distinction. Note that in FIG. 25A etc., angle φ (and angle θ as well) is the angle made with respect to the plane of the floating in space image 3, but since they are similar depending on the definition, the angle made between the normal 2305 and the direction 2303 of the finger 231 may also be referred to as angle φ (and angle θ as well).

[0322] There are individual differences in the method of mid-air operation of user 230. An example of such individual differences is shown in Figures 25A and 25B. In this embodiment, a case is considered in which observation reference point 232 is on normal line 2305. Also, in this embodiment, a case is considered in which finger 231, which is an operating finger, is on line of sight 2302 in which operation target point 2301(P) is observed from observation reference point 232, and operation target point 2301(P) is the center point O of the screen.

[0323] [Method A: Finger direction and camera placement] FIG. 26 shows an example of the camera position of the imaging unit 1180 with respect to the finger direction 2303 as shown in FIG. 25A. FIG. 26 shows an example corresponding to FIG. 25A, but the same applies to FIG. 25B. In FIG. 26, the origin O, which is the center point O of the screen of the floating in space image 3, is set as the operation target point 2301 (P). FIG. 26 shows a normal 2305, which is a straight line that passes through the center point O and is perpendicular to the floating in space image 3. The normal 2305 coincides with the optical axis direction of the image light a1. The direction corresponding to this normal 2305 (observation reference direction) is a more suitable direction when observing and viewing the floating in space image 3 from the viewpoint position 232 of the user 230. In this example, the line of sight 2302 from the observation reference point 232, which is the viewpoint position 232, coincides with the direction of the normal 2305.

[0324] The camera is arranged so that the imaging unit 1180 can optimally capture images of the user's 230 finger 231, which is an operating finger, and the face, shoulders, arms, and other body parts. In this embodiment, since there are body parts that are preferably photographed depending on the method, the camera is arranged so that the body parts to be photographed can be optimally photographed. Figure 26 etc. shows an example of a suitable camera arrangement.

[0325] In order to more suitably capture a finger image 2011 of a finger 231, which is an operating finger, it is preferable to place the camera of the imaging unit 1180 in a position as shown in the figure, for example. The camera for capturing the finger image 2011 is placed, for example, in a position shown as camera 1180B. This position is a position that is downward in the vertical direction (y-axis) within the plane of the floating-in-space image 3. The shooting direction of camera 1180B corresponds to the upward direction in the vertical direction (y-axis) within the plane. In this case, it is possible to more suitably capture the displacement or tilt of the finger 231, particularly when the finger 231 is displaced or tilted in the left-right direction (x-axis) within the plane. However, the camera 1180B may be placed in a position that can capture the finger 231. In another example, the camera may be positioned upward in the vertical direction within the plane of the floating-in-space image 3, or may be positioned left or right in the horizontal direction within the plane.

[0326] Furthermore, in order to more suitably capture the facial image 2012 that shows the face 234, it is preferable to place the camera of the imaging unit 1180 in a position as shown in the figure, for example. The camera for capturing the facial image 2012 is placed, for example, in the position shown as camera 1180A. This position is a position that is above in the Z-axis direction and behind in the Y-axis direction with respect to the plane of the floating-in-space image 3. The shooting direction of the camera 1180A corresponds to a direction facing the side where the user 230 is located, which is the front side in the Y-axis direction, for example. In this case, it is possible to particularly suitably shoot an area including the face 234 of the user 230. However, the camera 1180A is not limited to this, and it is sufficient that it is placed in a position where it can shoot the face 234 and the like.

[0327] Furthermore, if the shooting range of a camera placed at any position covers the plane of the space-floating image 3, it is also possible to use that camera to shoot a finger 231 approaching the plane of the space-floating image 3 and obtain the finger image 2011. As another example of camera placement, as shown as camera 1180C, it may be positioned above the xy plane of the space-floating image 3. As another example of camera placement, as shown as camera 1180d, it may be positioned behind the xy plane of the space-floating image 3, where it can cover the shooting of the screen.

[0328] Even if the camera is placed at a position other than that shown in the figure, the correction function can be similarly realized by performing image processing using the correction unit 1900 or the like in accordance with the camera's shooting position and angle.

[0329] The space-floating image display device 1000 in FIG. 26 is mounted on a housing (not shown) and placed on the ground 2500. The spatial coordinate system (X, Y, Z) in FIG. 26 has the center point O of the space-floating image 3 as its origin O. The height h0 is the height of the origin O of the space-floating image 3 from the ground 2500. Depending on the hardware configuration, the space-floating image 3 in FIG. 26 is tilted at a predetermined angle β, for example, 45 degrees, with respect to the Y-axis and Z-axis as shown. In the space coordinate system (X, Y, Z), the position coordinates of the observation reference point 232 of the user 230 are (Xe, Ye, Ze). The position coordinates of the shoulder position 233 are (Xs, Ys, Zs). The observation reference point 232 is assumed to be on the normal line 2305 of the space-floating image 3.

[0330] In this case, (Xe, Ye, Ze) as the position coordinates that allow the space floating image 3 to be more suitably observed as the appropriate coordinates of the observation reference point 232 is given by, for example, formula A in the figure. That is, it is obtained as (Xe, Ye, Ze) = (0, - (h1 - h0), + (h1 - h0)).

[0331] Furthermore, the shoulder position 233 corresponding to the observation reference point 232 is given by, for example, equation B in the figure. That is, it is obtained as (Xs, Ys, Zs) = (w2, - (h1 - h0) - w4, h2 - h0). The shoulder position 233 may also be determined from the relative positional relationship from the observation reference point 232. The X coordinate of the shoulder position 233 is obtained from the X coordinate of the observation reference point 232 using a width w1. The Z coordinate of the shoulder position 233 is obtained from the Z coordinate of the observation reference point 232 using a width w2. The Y coordinate of the shoulder position 233 is obtained from the Y coordinate of the observation reference point 232 using a width w4. Note that the Y coordinate of the observation reference point 232 and the Y coordinate of the shoulder position 233 may be considered to be approximately the same, and width w4 = 0 may be set.

[0332] [Method A: Finger direction (x-axis)] Fig. 27A illustrates, in an XY plane view of a space as seen from above, such as Fig. 25A or Fig. 26, in particular the approach direction (finger direction 2303) and angle of finger 231 relative to the x-axis, which is the horizontal direction within the screen (display range 3R) of the floating in space image 3. Fig. 27A is for method A and corresponds to Fig. 25A. Fig. 27B is for method B and corresponds to Fig. 25B. While Fig. 25A etc. shows the angle φ relative to the y-axis, Fig. 27A etc. shows the angle θ relative to the x-axis.

[0333] In FIG. 27A, the tip of a finger 231 (231a) of a right hand, which is an operating finger of a user 230, is approaching, for example, origin O as an operation target point 2301 (P) on the xy plane of the screen. A position on normal 2305, which is a straight line that passes through origin O, which is the center point O of the xy plane of the screen, and is perpendicular to the plane, is a more suitable position for observing the floating in space image 3. In FIG. 27A, an observation reference point 232, which is a viewpoint position 232 of the user 230, is located on this normal 2305. The user 230 visually recognizes, for example, origin O as an operation target point 2301 (P) in a line of sight 2302 from viewpoint position 232, which is observation reference point 232 on normal direction 2305. In FIG. 27A, a finger direction 2303, which is a direction of approach at this time, roughly coincides with normal direction 2305 of the screen and line of sight direction 2302.

[0334] Here, the angle that the finger direction 2303 (extension line) makes with respect to the x-axis direction of the screen is defined as θ. In other words, the angle θ is the finger angle or approach angle. The angle that the finger direction 2303 makes with respect to the normal direction of the screen is 90°-θ. However, this is not limited to this, and depending on the definition, for example, the angle (90°-θ) that the finger direction 2303 makes with respect to the normal direction of the screen may be defined as θ.

[0335] The angle at which the operation target point P is viewed from the viewpoint position 232 is defined as the observation angle. For example, the observation angle is the angle formed by the line of sight 2302 with respect to the normal line 2305 of the screen. In the example of Fig. 27A, the line of sight 2302 is almost the same as the normal line 2305, so the observation angle is almost 0°.

[0336] In method A, the angle θ corresponding to the finger direction 2303 in FIG. 27A is obtained as an angle according to the line of sight direction 2302 from the observation reference point 232 to the operation target point P.

[0337] 27B, a finger direction 2303, which is a direction 2303 of approach of a finger 231 (231a) that is an operating finger relative to an operation target point 2301 (for example, origin O) on the screen, roughly coincides with a direction (arm direction) 2304 from a shoulder position (shoulder joint center) 233. In FIG. 27B, the angle corresponding to the angle θ in FIG. 27A is referred to as angle θ' for distinction.

[0338] In this embodiment, we consider a case where the arm, hand, and fingers 231 are stretched in a roughly straight line from shoulder position 233, which is the shoulder joint center 233, to operation target point 2301 (P), which is the center point O of the floating in space image 3. In other words, in the basic model as shown in Fig. 24, a straight line is defined from shoulder position 233 to operation target point P at the fingertip of finger 231, and a direction (arm direction) 2304 corresponding to this line is considered. Note that the actual arm posture is not limited to a straight posture, and it may be bent at the joint.

[0339] In this case, in the example of FIG. 27B, the finger direction 2303 in which the fingertip approaches the operation target point P may be considered to be substantially the same as the direction 2304 from the shoulder position 233.

[0340] In method B, the angle θ' corresponding to the finger direction 2303 in FIG. 27B is obtained as an angle according to the direction 2304 from the shoulder position 233 to the operation target point P. This angle θ' may also be calculated using the angle θ in method A and parameter values ​​such as h1 and w1 in FIG. 24, for example, by equation A in the figure. That is, θ'=θ-tan -1 (w1 / (h1-h0)).

[0341] [Video floating in space] Fig. 28A shows a display example of the screen of the floating image in space 3. Fig. 28A shows a case where the user 230 observes the screen from the front in a line of sight 2302 from an observation reference point 232 on a normal 2305 to the xy plane of the screen, that is, in the z-axis direction mentioned above. The screen has, as GUI object images, number buttons 2801 such as 0 to 9 and a phone button 2802. Numbers can be input by touching the number button 2801, and instructions such as a call can be input by using the phone button 2802. In this example, the number button "8" is located at the position of the origin O, which is the center point O of the screen.

[0342] An object such as the numeric button 2801 has a predetermined pixel area (for example, a roughly rectangular area as shown), and the pixel of the operation point that is in contact with the fingertip may be any pixel within that pixel area. For example, the operation point P of the numeric button "8" is illustrated as the origin O, but this is not limiting, and any point within the pixel area of ​​the numeric button "8" may be the operation point P.

[0343] FIG. 28A also shows a schematic diagram of a situation where a user 230 attempts to touch the numeric button "8" near the origin O with a finger 231 (for example, the index finger of the right hand) which is an operating finger on this screen.

[0344] As in the example of Fig. 28A, when an object at origin O is operated with finger 231, the apparent coordinates when observation reference point 232 of user 230 exists on the z-axis almost coincide with the coordinates of the detected point by user operation detection mechanism 1904. In other words, there is almost no deviation / error between operation target point P and the detected point. The apparent coordinates here are the intersection point (point P in the example of Fig. 23) where an extension line of a straight line connecting observation reference point 232 of user 230 and the tip of a finger (point F in the example of Fig. 23) approaching operation target point P (for example, origin O) on the xy plane of the floating in space image 3 intersects with the xy plane.

[0345] In FIG. 28A, the screen size of the floating image 3 in space is such that the distance from the origin O to the right end and the distance to the left end in the x-axis direction are distance lx, and the distance from the origin O to the top end and the distance to the bottom end in the y-axis direction are distance ly.

[0346] 28A, the apparent coordinates from user 230 and the operation target point P match, and it is recognized that there is no deviation between the apparent coordinates and the actual detected point, so correction for the deviation is not necessary. In this case, the amount of correction by the correction function is 0.

[0347] Next, FIG. 28B schematically illustrates a situation in which a user 230 attempts to touch the numeric button "9" located immediately to the right of the numeric button "8" in the x direction with a finger 231 on a screen similar to that of FIG. 28A. In this example, the object to be operated is the numeric button "9." An example of an operation target point 2301(P) corresponding to the numeric button "9," which is the object to be operated, is illustrated as an operation target point 2803(P1) located at the center of the pixel area of ​​the numeric button "9." Here, one operation target point 2803(P1) corresponds to one pixel. The user 230 visually recognizes the operation target point 2803(P1) of the numeric button "9" from the observation reference point 232 and moves the finger 231 toward the operation target point 2803(P1) to perform a touch operation. In the example of FIG. 28B, the finger direction 2303 and the like are not taken into consideration. If the finger direction 2303 is in the normal direction to the operation point 2803 (P1), it is recognized that there is no deviation between the operation point P / apparent coordinates and the detected point, as in FIG. 28A, and therefore no correction for the deviation is required.

[0348] Next, FIG. 28C shows another example in which user 230 attempts to operate numeric button "9" with finger 231, which is the operating finger, on a screen similar to that shown in FIG. 28A. Depending on user 230 and the situation, the actual operation may end up in the state shown in FIG. 28C rather than FIG. 28B. In the state shown in FIG. 28C, the operation target point / apparent coordinates of numeric button "9" as viewed from observation reference point 232 are points (pixels), such as point P1 or point P2. Even if user 230 intends to touch numeric button "9," the actual detection point by user operation detection mechanism 1904 may be point Q1, for example, depending on finger direction 2303, etc. The detected point, point Q1, is within the pixel area of ​​numeric button "8." In this case, the touch operation is determined as being on numeric button "8," resulting in an erroneous operation on numeric button "8."

[0349] This embodiment has a correction function 1500 (FIG. 15) to reduce such a deviation between the detection point and the operation target point and prevent erroneous operation due to the deviation.

[0350] In this embodiment, as in the example of FIG. 28A etc., by correcting the position coordinate difference between the operation target point and the detection point for mid-air operations on button objects on the screen of the Floating in Space Image 3, it is possible to reduce the number of times the button object is pressed by mistake. This correction function is not limited to this, but can also be applied to other screen contents in the same way. For example, this correction function can also be applied to a user interface that accepts predetermined mid-air operations on the screen (for example, touch operations, tap operations, slide operations, other gestures, etc.) when no object images are displayed on the screen of the Floating in Space Image 3. This correction function can improve the accuracy of detection and determination of operation target points and predetermined mid-air operations.

[0351] [Changes in observation angle and errors (1)] FIG. 29 shows the change and error in the azimuth angle θ (similar to the angle θ in FIG. 23, etc.) corresponding to the observation angle (the angle corresponding to the direction from the observation reference point 232) in the x direction of the plane of the floating image 3 for methods A and X. In FIG. 29, the finger direction 2303 is the direction from the viewpoint position 232, which is the observation reference point 232, to the operation target point P, and is the direction corresponding to the azimuth angle θ. Note that in FIG. 29, in order to show an example of the deviation between the operation target point and the detection point, the finger direction 2303, which corresponds to the direction of the fingertip, is illustrated as being slightly inclined with respect to the normal 2305 (z-axis) for emphasis in explanation. In this example, it is assumed that the line of sight direction 2302 when viewing the origin O from the viewpoint position 232, which is the observation reference point 232, and this finger direction 2303 are approximately the same. When methods A and X are used, the finger direction 2303 is calculated as the direction from the observation reference point 232 to the origin O.

[0352] In method X, the angle θ when the finger 231 approaches the origin O as the operation target point P0 is used as a reference. The origin O has position coordinates x0, y0. In the case of FIG. 29, a touch operation on the operation target point P0 may cause a deviation in the actual detection point. An example of the actual detection point is shown as point Q0. Point Q0 is located at a position (pixel) that is shifted to the right on the x-axis from point P0, for example. In method X, a correction amount corresponding to such a deviation between the operation target point and the detection point at the origin O is calculated.

[0353] 29 also shows, as examples of other operation points, an operation point 2901 (P1) located to the right of the origin O (positive direction of the x-axis, +x) and an operation point 2902 (P2) located to the left of the origin O (negative direction of the x-axis, -x). The operation point 2901 (P1) has position coordinates x1, y1. The operation point 2902 (P2) has position coordinates x2, y2. In method X, such operation points other than the origin O are considered to be roughly the same as the finger direction 2303 with respect to the origin O, as shown in the figure, and the correction amount at the origin O is applied in the same way.

[0354] [Changes in observation angle and errors (2)] Fig. 30A shows the change and error in the azimuth angle θ (similar to angle θ in Fig. 23 etc.) corresponding to the observation angle (angle corresponding to the direction from observation reference point 232) in the x direction of the plane of the space floating image 3 for Method A and Method Y. In Fig. 30A, finger direction 2303 is the direction from viewpoint position 232, which is observation reference point 232, to operation target point P, and is the direction corresponding to azimuth angle θ. In this example, line of sight direction 2302 from viewpoint position 232, which is observation reference point 232, and finger direction 2303 are almost the same.

[0355] In FIG. 30A , the angle θ corresponding to the finger direction 2303 may differ for each operation target point P on the screen. For example, in the case of operation target point P0 corresponding to origin O, the angle at which the fingertip approaches operation target point P0 is angle θ. On the other hand, the angle at which the fingertip approaches operation target point 2901 (P1) on the right side is, for example, angle θ+Δθ, as shown in the figure. Here, Δθ is the error angle. Corresponding to gaze direction 2302 / finger direction 2303, an error angle Δθ is added to angle θ as a positive error. On the other hand, as shown in the figure, the angle θ at which the fingertip approaches operation target point 2902 (P2) on the left side is, for example, angle θ-Δθ. Corresponding to gaze direction 2302 / finger direction 2303, an error angle Δθ is added to angle θ as a negative error.

[0356] In the case of Figure 30A, different deviations may occur in the actual detected points for each operation target point / apparent coordinate. In the example of Figure 30A, there is no deviation of the detected points from the origin O. The detected point Q1 for the point P1 on the right side is shifted to the left on the x-axis relative to point P1. The point Q2 for the point P2 on the left side is shifted to the right on the x-axis relative to point P2.

[0357] FIG. 30B shows the change and error in the elevation / depression angle φ (similar to the angle φ in FIG. 25A, etc.) corresponding to the observation angle in the y direction of the plane of the floating image 3 in space, for the same concept as FIG. 30A. As examples of other operation points, position 2903 (P3) located above the origin O (positive direction of the y axis, +y) and position 2904 (P4) located below the origin O (negative direction of the y axis, -y) are shown. The angle φ when the fingertip approaches the operation point P3 is, for example, φ+Δφ as shown in the figure. Here, Δφ is the error angle. Corresponding to the gaze direction 2302 / finger direction 2303, an error angle Δφ is added to the angle φ as a positive error. The angle φ when the fingertip approaches the operation point P4 is, for example, φ-Δφ as shown in the figure. Corresponding to the gaze direction 2302 / finger direction 2303, an error angle Δφ is added to the angle φ as a negative error.

[0358] In the case of Figure 30B, different deviations may occur in the actual detected points for each operation target point / apparent coordinate. In the example of Figure 30B, there is no deviation of the detected points from the origin O. The detected point Q3 for the upper point P3 is shifted downward on the y-axis relative to point P3. The point Q4 for the lower point P4 is shifted upward on the y-axis relative to point P4.

[0359] In the case of FIG. 30A etc., in method Y, for each operation point on the screen, the angle (θ, φ) corresponding to the finger direction 2303 is used to calculate the amount of correction corresponding to the deviation from the detection point.

[0360] The size of the screen (display range 3R) of the floating image in space 3 is set to 2lx x 2ly based on FIG. 28A. The observation distance is set to d. As shown in the figure, the observation distance d is the distance in the z-axis direction from the observation reference point 232 to the plane of the floating image in space 3. Then, the error angle Δθ is within the range of formula A shown in the figure. For example, the error angle Δθ is calculated by -tan -1 (lx / d) is larger than tan -1 (lx / d). Similarly, the error angle Δφ is within the range of equation B shown in the figure.

[0361] [Changes in observation angle and errors (3)] FIG. 31A and the like show explanatory diagrams similar to FIG. 30A and the like for Method B and Method Y. The difference is that FIG. 31A and the like use shoulder joint center 233, which is shoulder position 233, instead of viewpoint position 232. Finger direction 2303 is considered to be roughly the same as direction 2304 from shoulder position 233. FIG. 31A shows the change and error in azimuth angle θ' (similar to angle θ' in FIG. 27B) corresponding to direction 2304 from shoulder position 233 in the x direction of the plane of Floating in Space Image 3. FIG. 31B shows the change and error in elevation / depression angle φ' (similar to angle φ' in FIG. 25B) corresponding to the direction from shoulder position 233 in the y direction of the plane of Floating in Space Image 3, for the same concept as FIG. 31A.

[0362] In the case of FIG. 31A etc., a deviation of the detection point from the operation target point on the screen may occur depending on the finger direction 2303, that is, the direction 2304 from the shoulder position 233 to the operation target point of the fingertip. Different deviations may occur depending on the operation target point. This deviation mainly depends on the position coordinates of the shoulder position 233. In FIG. 31A, it is assumed that the user 230 observes each operation target point from an observation reference point 232 on a normal line 2305. In other words, it is assumed that the user 230 views the screen as shown in FIG. 28A from a front view corresponding to the z-axis direction. Then, consider a case where each operation target point is touch-operated with a finger direction 2303 corresponding to the direction 2304 of a straight line from the shoulder position 233, as in the above-mentioned FIGS. 25B and 27B. In this case, the angle that the finger direction 2303 of the fingertip makes with the x direction of the screen of the floating-in-space image 3 is defined as angle θ', and the angle that the finger direction 2303 makes with the y direction is defined as angle φ'. As in the cases of FIGS. 30A and 30B, error angles Δθ' and Δφ' can occur in each axial direction depending on the operation target point. In the example of FIG. 31A, error angles Δθ' occur for operation target points P1 and P2, and deviations occur between detection points Q1 and Q2 and operation target points P1 and P2. Note that the error angles Δθ' and the like in FIG. 31A are different from the values ​​of error angles Δθ and the like in FIG. 30A.

[0363] [Displacement between the operation point and the detection point] As in the examples of Fig. 30A and Fig. 31A, there may be an error angle in the angle (θ, φ) corresponding to the finger direction 2303 of the fingertip approaching the operation target point on the plane of the floating image 3. In this case, a deviation / error occurs in the position coordinates of the detection point when the fingertip actually first contacts the plane with respect to the operation target point / apparent coordinates. This deviation / error is the object of correction in this embodiment.

[0364] FIG. 32 shows an example of the deviation between the operation target point and the detection point on the xy plane of the screen (display range 3R) of the floating-in-space image 3. The origin O is the center point O of the screen, and its position coordinates are (x0, y0). As an example of the operation target point 2301, point P1 is an arbitrary point within the screen, and its position coordinates are (x1, y1). In this example, point P1 is within the (+x, +y) region (first quadrant). As an example of the detection point 2306, point Q1 is a point located at a position deviated from point P1, and its position coordinates are (x2, y2). Note that the coordinate of the right edge of the screen is +lx, the coordinate of the left edge is -lx, the coordinate of the top edge is +ly, and the coordinate of the bottom edge is -ly.

[0365] In this example, the operation target point 2301 (P1) and the detection point 2306 (Q1) have a deviation (in other words, error) 3201 of (Δx, Δy). The deviation Δx is the difference between x1 and x2. The deviation Δy is the difference between y1 and y2. The correction function 1500 ( FIG. 15 ) of this embodiment calculates the amount of correction for correcting such deviation 3201 (Δx, Δy).

[0366] 32 illustrates a case where the operation point 2301 and the detection point 2306 are separated to some extent for ease of understanding, but in reality, small deviations and errors on a pixel-by-pixel basis may occur. The correction function 1500 of this embodiment can correct even such small deviations and errors.

[0367] Regarding the deviation 3201 (denoted as Δ) in FIG. 32, in FIG. 29, for example, the deviation between point P0 and detection point Q0 occurs as Δx. In FIG. 30A, for example, the deviation between point P1 and detection point Q1 occurs as Δx. In FIG. 31A, for example, the deviation between point P1 and detection point Q1 occurs as Δx. The magnitude of each deviation can be different.

[0368] In order to determine such a deviation, in this embodiment (for example, method 1), the shape of the fingertip and the angle θ corresponding to the finger direction 2303 are determined as shown in FIG. 23, and the correction amount corresponding to the deviation Δ is determined from them.

[0369] [Angle (θ, φ)] The calculation of the angle (θ, φ) will be described. Here, for example, as shown in the lower part of FIG. 32, the angle (azimuth angle) θ formed by the finger direction 2303 corresponding to the straight line connecting the observation reference point 232 fixed at a specific position coordinate to the center point O of the screen of the floating image in space 3 and, for example, the x-axis direction of the screen is defined as θ0, and the angle (elevation / depression angle) φ formed by the finger direction 2303 and the y-axis direction is defined as φ0. The position coordinates of an arbitrary point P in the xy plane of the screen are defined as (x, y), and the x coordinate of the observation reference point 232 on the xy plane is defined as xe and the y coordinate is defined as ye. When the user 230 looks at an arbitrary point P(x, y), for example, an operation target point P1(x1, y1), from the observation reference point 232, the angle (θ, φ) formed by the finger direction 2303 corresponding to the straight line from the observation reference point 232 to the operation target point P1 and the plane is defined as θx, the azimuth angle on the x-axis, and φy, the elevation / depression angle on the y-axis. The lower part of Figure 32 shows φ0, φy, and ye in the yz plane view, but the same applies to θ0, θx, and xe on the x axis.

[0370] At this time, the angles (θ, φ), particularly the angles θ0, θx, φ0, and φy, can be expressed by equation A shown in FIG. 33. The angles θx and φy can be obtained based on equation A. The position coordinates of detection point Q are now set as (x, y). The corrected position coordinates for detection point Q (the position coordinates of the detection point after correction) can be found by inputting the position of observation reference point 232 and the coordinates of detection point Q, using the estimated values ​​of (θ, φ) obtained from equation A.

[0371] [Method 1: Details of ellipse approximation] FIG. 34 is a detailed explanatory diagram of the ellipse approximation in FIG. 23. In FIG. 34, as in FIG. 23, the horizontal direction within the screen of the floating-in-space image 3 is the x-axis, the vertical direction within the screen is the y-axis, and the depth direction relative to the screen is the z-axis. Here, the explanation will be given using angle θ on the x-axis, but the same applies to angle φ on the y-axis. FIG. 34 shows an xz plane view, similar to the enlarged view of the fingertip portion 2300 in FIG. 23, of the displacement (displacement amount) Δx in the x-axis direction in FIG. 32. The difference is that FIG. 34 shows a case where the operation target point P is the origin O, which is the center point O of the screen. Depending on the finger direction 2303 and the like described above, a detection point Q corresponding to the point where the fingertip contacts the plane is displaced Δx from the operation target point P. As described above, the shape of the fingertip 231A of the finger 231 is obtained as an approximate ellipse 501 by scanning and fitting using a camera, and parameter values ​​such as the ellipse center 502, major axis a, minor axis b, and foci F1 and F2 are obtained.

[0372] Consider a straight line 3410 that passes through the ellipse center 502 and focal point F1 of the approximated ellipse 501, in other words, a straight line along the major axis a of the approximated ellipse 501. The straight line 3410 roughly coincides with the finger direction 2303. This straight line 3410 is extended beyond point F (the intersection of the straight line 3410 and the approximated ellipse 501), which is the center of the fingertip, and there is an intersection where this extension line intersects with the xy plane of the floating-in-space image 3. The intersection of this straight line 3410 corresponds to the operation target point P. As shown in the figure, when the fingertip moves along the direction of the straight line 3410 (finger direction 2303), the point where the fingertip first contacts the xy plane of the floating-in-space image 3 depending on the shape of the fingertip, etc., is detection point Q. In other words, detection point Q is the point of contact between the outline of the fingertip and the xy plane.

[0373] 34, a right-angled triangle 3400 is considered, which is formed by a center point O, which is the operation target point P, a detection point Q, and a focus F1 of the approximated ellipse 501. The bottom of FIG. 34 shows an enlarged view of the right-angled triangle 3400. The right-angled triangle 3400 has a side 3401 in the x-axis direction, a side 3402 in the z-axis direction, and a diagonal side 3403. Side 3401 corresponds to the shift Δx. Side 3402 corresponds to the length (perpendicular line) from the focus F1 to the detection point Q. Side 3403 corresponds to the length from the focus F1 to the operation target point P. In the illustrated example, when the angle θ with the x-axis is used, the interior angle of the right-angled triangle 3400 at the detection point Q is 90°, the interior angle at point P1 is 180°-θ, and the interior angle at the focus F1 is θ-90°.

[0374] From the above calculation, we can determine the angle θ (or angle θ') and angle φ (or angle φ') according to the finger direction 2303. In the xz plane as shown in Figure 34, a line 3410 representing the extension of the fingertip corresponding to the finger direction 2303 can be expressed as a function z = (tan θx)x (Equation A in Figure 35A).

[0375] Equation A in Figure 35A indicates a mathematical formula for approximating ellipse 501. Here, the ellipse center 502 of approximating ellipse 501 is set as (xd, zd) = (xd, (tan θx)xd) at point D, as in equation B in Figure 35A. In this case, the ellipse equation for approximating ellipse 501 can be expressed as equation C.

[0376] Here, the detection point Q, which is the contact point between the fingertip 231A and the xy plane, is set as (x2, z2) = (xP, 0). The intersection of the normal line passing through the contact point Q and the line 3410 is the focus F1, which is one of the ellipse foci in the approximated ellipse 501. Therefore, xd, which constitutes the x coordinate of point D at the ellipse center 502, is expressed as xd = xP + √(a 2 -b 2 )cosθ.

[0377] Furthermore, the detection point Q, which is the contact point, is a point on the approximate ellipse 501. Therefore, the detection point Q can be expressed by the formula D shown in Fig. 35B. Note that although the angle θ and the deviation Δx in the x-axis direction have been described above, the angle φ and the deviation Δy in the y-axis direction can also be calculated in the same way.

[0378] The number of pixels per unit length on the screen of the display unit 1902 (having, for example, a liquid crystal display panel) in Fig. 19, which corresponds to the screen of the floating image 3 in space, is set to n. The correction amount converted / expressed in pixels is obtained as correction amount nΔx by equation E in Fig. 35C using the number of pixels n and the deviation (deviation amount) Δx. This correction amount nΔx is the product of the number of pixels n and the deviation Δx. This correction amount nΔx is obtained as an appropriate correction amount (offset value) that corresponds to the shape and finger direction 2303 (angle θx) of the user 230's finger 231, etc., regarding the deviation / error between the detection point Q and the operation target point P (origin O).

[0379] The correction unit 1900 (FIGS. 19 and 20A) of the space floating image display device 1000 simply performs a calculation to reflect the correction amount nΔx for this pixel on the position coordinates (1914) of the detection point Q as a correction calculation. In the example of FIG. 34, the correction amount nΔx corresponding to the deviation Δx according to the angle θx is added as an offset value in the positive direction to the coordinates (x2, z2) of the detection point Q. This allows the position coordinates of the corrected detection point, which corresponds to the operation target point P1 (x1, z1), to be obtained. In other words, the position coordinates of the point that is estimated to be the point that the user 230 is attempting to touch as a target are obtained.

[0380] The space floating image display device 1000 may calculate the above correction amount every time a user performs an aerial operation, but if there is a correction amount obtained by one calculation for a reference point such as the center point O of the screen, that correction amount may be set as a standard correction amount / offset value for that user. Then, by applying that correction amount / offset value every time the same user performs an aerial operation, it is possible to omit the calculation every time.

[0381] [Method X / Method Y] Methods X and Y in FIG. 18 will be described.

[0382] FIG. 36 is an explanatory diagram regarding method X and method Y. As in FIG. 28A and FIG. 32, it is assumed that there is a screen of the floating image in space 3. As described above, in method X, the origin O, which is the center point O of the screen, is used as the reference point, and is regarded as the operation target point P, and the correction amount according to the finger direction 2303 and angle (θ, φ) is calculated. FIG. 36 shows the concept of the correction amount in method X on the xy plane of the screen. In response to a touch operation by the finger 231 of the user 230 relative to the origin O, a correction amount according to the way the finger 231 is moved, etc. is obtained. This correction amount is referred to as correction amount D0. This correction amount at the origin O may be obtained, for example, as calibration / adjustment based on a user interface such as that shown in FIG. 22 described above.

[0383] It is assumed that the operation target point P moves outward from the origin O in the x-axis direction and the y-axis direction on the screen. FIG. 36 shows, as an example, a certain operation target point 3601 (denoted as P1) on a line 3600 in a certain direction 3600. This operation target point 3601 is conceptually similar to the operation target point P1 (x1, y1) in FIG. 32 described above. The correction unit 1900 ( FIG. 19 ) calculates a correction amount (denoted as D1) for the point P1, which is the operation target point 3601, according to the finger direction 2303 and angle (θ, φ) described above. In this case, in method X, the finger direction 2303 and angle (θ, φ) for point P1 are considered to be approximately the same as the finger direction 2303 and angle (θ, φ) at the origin O, and the correction amount D0 at the origin O is similarly applied to the correction amount D1 at point P1 (D1=D0). This application is also the same when points at other position coordinates on the screen are operation target points. The same correction amount D0 is also applied to other points (indicated by diamond-shaped points) on the straight line 3600 in the direction 3600.

[0384] In the same figure, in method Y, a correction amount is calculated according to the finger direction 2303 and angle (θ, φ) for each point of arbitrary position coordinates on the screen. For example, a correction amount D0 at the origin O and a correction amount D1 at point P1 are calculated separately. For each point on the screen, depending on differences in the finger direction 2303 of the finger 231, that is, differences in characteristics such as how the user moves the finger 231, the calculated correction amount may be a value different from the correction amount D0 at the origin O.

[0385] [Modification of Method X (1)] In the above method X, the origin O is used as the reference point for correction, but this is not limiting and any predetermined point within the screen may be set as the reference point for correction. For example, four quadrants (FIG. 32) may be set within the screen, and a reference point for correction may be set for each quadrant. Then, the same amount of correction based on the reference point may be applied to each quadrant.

[0386] [Modification of Method X (2)] The following modification of method X is also possible. FIG. 37 shows this modification. Consider a point 3701 on the screen farthest from the origin O, for example, the top-left point PL on the outermost periphery. A line 3700 is shown extending in a direction 3700 from the origin O to point PL. The correction unit 1900 calculates a correction amount according to the finger direction 2303 and angle (θ, φ) for not only the origin O but also the point PL. This correction amount is referred to as the correction amount DL. Consider each operation point P on the line 3700 between the origin O and the outermost point PL. For example, assume there is a point PM. In this modification, the correction amount (referred to as DM) for point PM is calculated as a correction amount within the range between the correction amount DO for the origin O and the correction amount DL for point PL. This calculation can be performed using a simple calculation, such as using a ratio according to the approximate position of point PM. For example, if point PM is approximately midway between the origin O and point PL, the correction amount DM may be an intermediate value between the correction amount DO and the correction amount DL.

[0387] [Dominant hand when operating in the air] 38A and 38B show xz plane views of the floating-in-space image 3 as explanatory diagrams of the dominant hand during mid-air operation. FIG. 38A shows a case where the same operation target point P within the xy plane screen of the floating-in-space image 3 is operated with the fingertip of the right hand, for example, at the center point O, and FIG. 38B shows a case where the same operation target point P is operated with the fingertip of the left hand, and examples of the difference in correction amount and correction direction are shown. In this embodiment, as described above, based on the camera image of the imaging unit 1180, the viewpoint position 232, finger direction 2303, etc. are grasped, and it is also possible to grasp whether the user 230 is using their right hand or left hand during mid-air operation. That is, the floating-in-space image display device 1000 determines the dominant hand (right hand or left hand) for each user during mid-air operation. The floating-in-space image display device 1000 may then determine the direction of correction, etc., depending on the dominant hand of the user during mid-air operation.

[0388] FIG. 38A shows finger direction 2303 when the tip of finger 231a of the right hand approaches operation target point P (origin O). The actual detection point is designated as QR. Finger direction 2303 of the fingertip has angle α with respect to normal 2305 (z direction) of the screen, and angle θ with respect to the x direction. Here, in the x direction, the right side of the drawing is the positive direction (+x) and the left side of the drawing is the negative direction (-x). There is a deviation ΔxR between detection point QR and operation target point P. The position coordinates (xR1, yR1) of detection point QR are located to the right of the position coordinates (x0, y0) of point P (origin O) in the +x direction, a distance corresponding to the deviation ΔxR. In this case, the correction amount 3801 is calculated as an amount that includes the -x direction as the correction direction, and as shown in the enlarged view, by adding the correction amount 3801 corresponding to the deviation ΔxR to the position coordinates (xR1, yR1) of the detection point QR, the position coordinates of the corrected detection point 3802 corresponding to the operation target point P (origin O) are obtained.

[0389] FIG. 38B similarly shows finger direction 2303 when the tip of finger 231b of the left hand approaches operation target point P. The actual detection point is designated QL. There is a deviation ΔxL between detection point QL and operation target point P. The position coordinates (xL1, yL1) of detection point QL are located to the left of the position coordinates (x0, y0) of point P (origin O) in the -x direction by a distance corresponding to the deviation ΔxL. In this case, correction amount 3803 is calculated as an amount that includes the +x direction as the correction direction, and as shown in the enlarged view, by adding correction amount 3803 corresponding to the deviation ΔxL to the position coordinates (xL1, yL1) of detection point QL, the position coordinates of corrected detection point 3804 corresponding to operation target point P (origin O) are obtained.

[0390] The above correction can be similarly applied to other points on the screen. Note that the correction amount calculated by the correction unit 1900 may be an amount that does not include a positive or negative direction. In this case, the correction unit 1900 calculates not only the correction amount but also the positive / negative direction as the correction direction. When performing a calculation to reflect the correction amount on an actual detection point, the correction unit 1900 or the image processing unit 1901 performs a calculation to reflect the positive / negative direction as the correction direction along with the correction amount. For example, in the case of a situation like that shown in FIG. 38A, the correction direction is set to the negative direction, and the calculation is performed by subtracting the correction amount ΔxR from the position coordinates of the detection point QR, thereby obtaining the position coordinates of the detection point after correction.

[0391] [When the observation reference point is shifted] In the above embodiments, the case where the user's viewpoint position 232 is at the position of the observation reference point 232 on the observation reference direction, which is the normal direction of the space floating image 3, has been described (FIGS. 15, 26, etc.). Even if the user's viewpoint position 232 is deviated from this observation reference point 232, the concept of this correction can be considered in the same way, and each embodiment can be applied in the same way.

[0392] FIG. 39 shows an xz plane view as an explanatory diagram for a case where the user's viewpoint position 232 is shifted from the position of the preferred observation reference point 232 on the observation reference direction corresponding to the normal 2305 of the screen of the Floating in Space Image 3. The illustrated viewpoint position 232A is an example of a position shifted to the left in the x direction from the position of the preferred observation reference point 3901 on the observation reference direction corresponding to the normal 2305. Shoulder position 233A is an example of a shoulder position corresponding to viewpoint position 232A. This shows a case where the finger direction 2303 when the fingertip approaches an operation target point P1 (e.g., center point O) on the screen is approximately the same as the line of sight direction 2302. The angle θ corresponding to the finger direction 2303 is angle θ1. An example of an actual detection point is point Q1. Similarly, as another example, a case where the operation target point is point P2 is shown. Corresponding to point P2, there is angle θ2 of the finger direction 2303 and detection point Q2. In this example, as in the above, the amount of correction for the deviation between the operation target point and the detection point can be calculated using the viewpoint position 232A or shoulder position 233A, the elliptical approximation of the fingertip, the finger direction 2303, etc. Therefore, this example is preferable to one that does not consider the viewpoint position.

[0393] [User operation detection mechanism: mid-air operation detection sensor] FIG. 40 shows an example of the configuration of an aerial operation detection sensor 1351 applicable to each embodiment, as an example of the user operation detection mechanism 1904 (FIG. 19) for detecting an aerial operation by the finger 231 on the screen of the floating-in-space image 3. In this example, an xy plan view shows an example of the configuration when the aerial operation detection sensor 1351 is arranged at an upper position on the y-axis with respect to the coordinate system (x, y, z) of the floating-in-space image 3 in the arrangement of the floating-in-space image display device 1000 as shown in FIG. 26. The aerial operation detection sensor 1351 has multiple optical elements 1351c arranged in the x-direction. The optical elements 1351c are pairs of a light-emitting element 1351a and a light-receiving element 1351b. The light-emitting element 1351a is composed of, for example, an infrared element. The light-emitting element 1351a emits light a1, for example, infrared light, downward in the y-direction. If the light a1 is not blocked by an object, it passes through the display range 3R. When the light a1 is blocked by an object, it is reflected by the object and returns as reflected light a2, which is received by the light receiving element 1351b.

[0394] For example, if a contact point a3 made by a user's finger is within the xy plane of the display range 3R, light a1 is reflected from the contact point a3 and returns as reflected light a2. A light receiving element 1351b at a certain x-direction position detects the reflected light a2. As a result, based on the detection signal from the optical element 1351c, the aerial operation detection unit 1350 determines that the contact point a3 is located at that x-direction position. Furthermore, the distance can be calculated using the TOF (Time Of Flight) method from the time it takes for the light a1 to return as reflected light a2. For example, the distance a4 to the contact point a3 can be calculated. This also determines the position coordinates of the contact point a3 on the xy plane of the display range 3R.

[0395] Not limited to this example, the aerial operation detection sensor 1351 may be arranged above, below, or to the left or right of the xy plane of the screen (display range 3R). Furthermore, the aerial operation detection sensor 1351 may be arranged at a position shifted in the front-to-back direction, i.e., the z direction, with respect to the plane, or multiple aerial operation detection sensors 1351 may be arranged at multiple positions in the front-to-back direction.

[0396] In the third embodiment and the like, a sensor capable of two-dimensional (x, y) detection corresponding to the xy plane of the floating in space image 3 is applied as the user operation detection mechanism 1904 for the mid-air operation detection sensor 1351, as in the example of Fig. 40. In other words, this sensor is a sensor capable of detecting two-dimensional (x, y) position coordinates on the xy plane of the floating in space image 3 where the fingertip is in contact, that is, as the position coordinates where the fingertip is placed in space.

[0397] [User operation detection mechanism: 3D sensor] 41 shows another example of the user operation detection mechanism 1904, which uses a TOF three-dimensional sensor, in other words, a camera with a distance measurement function or an aerial operation detection sensor. The sensor 4101 in FIG. 41 is arranged, for example, at a rear position in the front-to-rear direction (z-axis) with respect to the xy plane of the screen of the floating in space image 3 (for example, camera 1180d in FIG. 26). The detection range of this sensor 4101 covers the xy plane of the floating in space image 3 as shown in the figure. As shown in the enlarged view, this sensor 4101 has an array of, for example, multiple sensor elements 4102. The xy plane of the floating in space image 3 can be considered as being divided into multiple regions 4103 on the x-axis and y-axis as shown in the figure. The sensor elements 4102 emit and receive light (for example, infrared light) for the regions associated with them in the array. As a result, for example, when a certain area 4103a is touched by a finger 231, the distance to the area 4103a can be measured by the TOF method based on the light reflected from the area 4103a. Furthermore, if the multiple areas 4103 shown in the figure are provided at a higher density, detection with higher accuracy is possible.

[0398] As in the example of FIG. 41, even in the case of a sensor 4101 that performs detection in three dimensions (x, y, z), the detection of two-dimensional (x, y) position coordinates corresponding to the screen of the floating image 3 in space is included, so the correction method in this embodiment can be similarly applied.

[0399] [Variation: Calibration] FIG. 42 shows a modified example of calibration. In this modified example, the space-floating image display device 1000, during calibration, has the user touch multiple points / areas within a plane on the screen of the space-floating image 3 with a manipulation finger, and performs calibration, i.e., correction, based on the detection information at those multiple points / areas. The screen example of FIG. 42 is a variation of the screen examples of FIGS. 22A and 22B. This screen displays a total of five positions (in other words, areas): position 4200 corresponding to the origin O, which is the center point O; position 4201 near the left end on the x-axis; position 4202 near the right end; position 4203 near the top end on the y-axis; and position 4204 near the bottom end. Touch operations at each position / area are sequentially prompted as a predetermined guide or guidance. The correction unit 1900 calculates the amount of correction at each position / area. The correction unit 1900 can use information such as the amount of correction for each of these five positions / areas as a basis to more precisely grasp the characteristics of how the user 230 moves their fingers when performing aerial operations, which are individual to the user, and can make more detailed corrections.

[0400] [Variation: Combination of Method A and Method B] As a modified example of Example 3, etc., it is also possible to use both Method A and Method B. This modified example uses Method A, which uses the viewpoint position 232 (angles θ, φ), and Method B, which uses the shoulder position 233 (angles θ', φ'), as the reference position for the finger direction 2303.

[0401] 43 shows an example configuration of the correction unit 1900 in a modified example. The correction unit 1900 includes a fitting processing unit 2001, a parameter calculation unit 2002, a user position grasping unit 2021, a user attribute / characteristic recognition unit 2022, a correction amount calculation unit 4301, and a correction amount selection unit 4302. The user attribute / characteristic recognition unit 2022 in the modified example recognizes the attributes / characteristics of the target user 230 from an image 1910 (e.g., a face image 2012) captured by the camera of the imaging unit 1180.

[0402] The correction amount calculation unit 4301 includes a method A correction amount calculation unit and a method B correction amount calculation unit. The method A correction amount calculation unit is a part that calculates the correction amount in method A (a method that uses a line of sight direction 2302 from a viewpoint position 232 as a finger direction 2303). The method B correction amount calculation unit is a part that calculates the correction amount in method B (a method that uses a direction 2304 from a shoulder position 233 as a f...

Claims

1. A floating-in-the-air image display device, a video processing unit that performs video processing; a display unit that displays the image that has been processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; a detection mechanism for detecting a user's mid-air operation on the screen of the floating image; a correction unit that corrects a deviation between an operation target point and a detection point in a plane direction of the screen, regarding an aerial operation by the user on a screen of the floating-in-the-air image; Equipped with The correction unit corrects the deviation based on a shape of a fingertip of the user's operating finger when performing an air operation. A floating video display device.

2. 2. The floating-in-the-air image display device according to claim 1, the correction unit corrects the deviation based on a direction in which a fingertip of a finger that is an operating finger of the user during an aerial operation approaches a plane of the screen. A floating video display device.

3. 2. The floating-in-the-air image display device according to claim 1, the correction unit detects a viewpoint position as a reference position of the user during the user's aerial operation, and corrects the deviation based on the viewpoint position. A floating video display device.

4. 2. The floating-in-the-air image display device according to claim 1, the correction unit calculates a correction amount for correcting a deviation between the operation target point and the detection point in an in-plane direction of the screen, and obtains the position coordinates of the detection point after the correction by a calculation that reflects the correction amount in the position coordinates of the detection point detected by the detection mechanism. A floating video display device.

5. 2. The floating-in-the-air image display device according to claim 1, the correction unit performs geometrically approximating fitting on a shape of a fingertip of the user's operating finger during mid-air operation, and corrects the deviation based on a parameter value of the fitting. A floating video display device.

6. 2. The floating-in-the-air image display device according to claim 1, the correction unit detects a viewpoint position as a reference position of the user when the user performs an aerial operation, calculates a finger direction as a direction from the viewpoint position to the operation target point as a direction when a fingertip of the operating finger approaches a plane of the screen, and corrects the deviation based on the finger direction. A floating video display device.

7. 2. The floating-in-the-air image display device according to claim 1, the correction unit detects a shoulder position as a reference position of the user when the user performs an aerial operation, calculates a finger direction as a direction from the shoulder position to the operation target point as a direction when a fingertip of the operating finger approaches a plane of the screen, and corrects the deviation based on the finger direction. A floating video display device.

8. 2. The floating-in-the-air image display device according to claim 1, the correction unit calculates a correction amount for correcting the deviation based on an aerial operation performed by the fingertip of the operating finger relative to a predetermined reference point within the screen, and similarly applies the correction amount at the reference point to aerial operations performed relative to other points within the screen. A floating video display device.

9. 2. The floating-in-the-air image display device according to claim 1, the correction unit calculates, for each point on the screen, a correction amount for correcting the deviation based on an air operation performed by the fingertip of the operating finger. A floating video display device.

10. 2. The floating-in-the-air image display device according to claim 1, the correction unit sets, in advance, statistical information about the shape of the fingertip of the operating finger of the user when performing the aerial operation, and a correction amount for correcting the deviation associated with the statistical information, in a correspondence table, and obtains the correction amount by referring to the correspondence table when the user performs the aerial operation. A floating video display device.

11. 2. The floating-in-the-air image display device according to claim 1, the correction unit guides the user to place the tip of the operating finger at a predetermined position on the screen, and performs the correction based on an image of the fingertip placed in accordance with the guidance. A floating video display device.

12. 6. The airborne image display device according to claim 5, the correction unit performs elliptical approximation as the fitting for the shape of the fingertip. A floating video display device.

13. 13. The airborne image display device according to claim 12, the correction unit uses parameter values ​​of the approximate ellipse obtained by the ellipse approximation to determine an intersection point where an extension line of the center of the fingertip intersects with the plane of the screen as the operation point and a point where the approximate ellipse is tangent to the plane of the screen as the detection point, and calculates a correction amount for correcting a deviation between the operation point and the detection point. A floating video display device.

14. 8. The airborne image display device according to claim 7, the correction unit detects a viewpoint position of the user when the user performs an aerial operation, and detects the shoulder position based on a positional relationship relative to the viewpoint position. A floating video display device.

15. 2. The floating-in-the-air image display device according to claim 1, the correction unit detects whether the user's right hand or left hand is a dominant hand for the operation finger during the aerial operation, and calculates a correction amount for correcting the deviation depending on the dominant hand. A floating video display device.

16. 2. The floating-in-the-air image display device according to claim 1, The correction unit a first correction unit that detects a viewpoint position as a reference position of the user when the user performs an aerial operation, calculates a finger direction that is a direction from the viewpoint position to the operation target point as a direction when a fingertip of the operating finger approaches a plane of the screen, and corrects the deviation based on the finger direction; a second correction unit that detects a shoulder position as a reference position of the user when the user performs an aerial operation, calculates a finger direction that is a direction from the shoulder position to the operation target point as a direction when a fingertip of the operating finger approaches a plane of the screen, and corrects the deviation based on the finger direction; applying a correction amount selected from a correction amount range including the correction amount by the first correction unit and the correction amount by the second correction unit, when the user performs an aerial operation; A floating video display device.

17. 2. The floating-in-the-air image display device according to claim 1, The correspondence table has set therein statistical information regarding a direction in which a fingertip of an operating finger of the user approaches a plane of the screen during an air operation, and the correction unit searches the correspondence table using, as an input value, information regarding a direction when the fingertip of the operating finger approaches a plane of the screen during the user's mid-air operation, and obtains the correction amount. A floating video display device.

18. 2. The floating-in-the-air image display device according to claim 1, The correction unit monitors a situation during an aerial operation by the user on the screen, and corrects the deviation when it determines that an erroneous operation has been repeated. A floating video display device.

19. 19. The airborne image display device according to claim 18, the correction unit controls a plurality of button areas provided on a first screen as the screen to transition to a second screen associated with the button area in accordance with the aerial operation for each button area, controls to enable return from the second screen to the first screen in accordance with the operation of the user, and determines that the erroneous operation has been repeated when the number of repetitions of the operation of returning from the second screen to the first screen is equal to or greater than a threshold value; A floating video display device.

20. A correction method for a floating-in-the-air image display device, comprising: The floating image display device includes: a video processing unit that performs video processing; a display unit that displays the image that has been processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; a detection mechanism for detecting a user's mid-air operation on the screen of the floating image; a correction unit that corrects a deviation between an operation target point and a detection point in a plane direction of the screen, regarding an aerial operation by the user on a screen of the floating-in-the-air image; Equipped with Steps executed by the floating-in-the-air image display device include: a step of correcting the deviation based on a shape of a fingertip of a finger that is an operating finger of the user when performing an air operation, by the correction unit; The correction method includes:

Images