Floating image display device

Abstract

The present invention provides a more suitable floating image display device. This floating image display device comprises: an image processing unit; a display unit that displays an image resulting from image processing performed by the image processing unit; an optical system that generates a floating image on the basis of the image displayed by the display unit; and a detection mechanism that detects an aerial operation by a user with respect to the floating image, wherein a 2D operation mode is provided to accept a planar operation in the plane of the floating image as the aerial operation, and in the 2D operation mode, an image prompting a planar operation is displayed in the floating image.

Description

Floating image display device

[0001] The present invention relates to a floating-in-the-air image display device.

[0002] The floating information display technology is disclosed in, for example, Patent Document 1.

[0003] Japanese Patent Application Laid-Open No. 2019-128722

[0004] However, the disclosure of Patent Document 1 does not sufficiently consider configurations for obtaining practical brightness and quality for the floating image, or configurations for allowing users to view the floating image more enjoyably.

[0005] An object of the present invention is to provide a more suitable floating-in-the-air image display device.

[0006] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes a plurality of means for solving the above problems, but to cite one example, a floating-in-the-air image display device includes an image processing unit, a display unit that displays an image that has been image-processed by the image processing unit, an optical system that generates a floating-in-the-air image based on the image displayed by the display unit, and a detection mechanism that detects aerial operations by a user on the floating-in-the-air image, and is provided with a 2D operation mode that accepts planar operations on the plane of the floating-in-the-air image as aerial operations, and in the 2D operation mode, displays an image on the floating-in-the-air image that prompts the planar operation.

[0007] According to the present invention, a more suitable floating-in-the-air image display device can be realized. Other problems, configurations, and effects will become clear in the following description of the embodiments.

[0008] FIG. 1 is a diagram showing an example of a usage form of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a projection diagram of a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 7 is a top view of a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 8 is a perspective view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 9 is a top view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 10 is a side view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of a configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 1 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention.FIG. 1 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 3 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 4 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 5 is a layout diagram showing main parts of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. FIG. 7 is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. FIG. 8 is an explanatory diagram for explaining light source diffusion characteristics of an image display device according to an embodiment of the present invention. FIG. 9 is an explanatory diagram for explaining diffusion characteristics of an image display device according to an embodiment of the present invention. FIG. 10 is an explanatory diagram for explaining an example of a problem solved by image processing according to an embodiment of the present invention. FIG. 11 is an explanatory diagram for explaining an example of image processing according to an embodiment of the present invention. FIG. 12 is an explanatory diagram for explaining an example of image display processing according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of the configuration of a main part and a retro-reflection part of a space-floating image display device according to an embodiment of the present invention. FIG. 14 is an explanatory diagram for explaining an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 1 is an explanatory diagram showing an implementation example of an aerial operation detection sensor according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing a modified example of the aerial operation detection sensor according to an embodiment of the present invention. FIG. 3 is an explanatory diagram showing a mode according to a state of a user operation detection mechanism according to an embodiment of the present invention. FIG. 4 is an explanatory diagram showing a mode according to a state of a user operation detection mechanism according to an embodiment of the present invention. FIG. 5 is an explanatory diagram of a space-floating image display device in which a 2D operation mode is implemented as implementation example 1 according to an embodiment of the present invention. FIG. 6 is an explanatory diagram of a space-floating image display device in which a 3D operation mode is implemented as implementation example 2 according to an embodiment of the present invention. FIG. 7 is an explanatory diagram of a space-floating image display device in which a 2D operation mode and a 3D operation mode are implemented as implementation example 3 according to an embodiment of the present invention. FIG. 8 is an explanatory diagram showing a trigger for determining a mode according to an embodiment of the present invention. FIG. 9 is an explanatory diagram showing an example of mode selection by a user according to an embodiment of the present invention.FIG. 1 is an explanatory diagram related to control of the correspondence between mid-air operation detection and space-floating image display, according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example of the configuration of a space-floating image display system in the case where information related to a mode is attached to image data, according to an embodiment of the present invention. FIG. 3 is an explanatory diagram showing an example of a 2D operation and a 3D operation, according to an embodiment of the present invention. FIG. 4 is an explanatory diagram showing an example of a display control process executed in correspondence with a mid-air operation, according to an embodiment of the present invention. FIG. 5 is an explanatory diagram showing an example of a 2D display prompting a 2D operation and an example of a 3D display prompting a 3D operation, according to an embodiment of the present invention. FIG. 6 is an explanatory diagram showing a first example of 2D / 3D display of a space-floating image, according to an embodiment of the present invention. FIG. 7 is an explanatory diagram showing a second example of 2D / 3D display of a space-floating image, according to an embodiment of the present invention. FIG. 8 is an explanatory diagram showing a third example of 2D / 3D display of a space-floating image, according to an embodiment of the present invention. FIG. 9 is an explanatory diagram showing a fourth example of 2D / 3D display of a space-floating image, according to an embodiment of the present invention. FIG. 10 is an explanatory diagram showing an example of conversion from original image data to display data for a space-floating image, according to an embodiment of the present invention. FIG. 11 is an explanatory diagram showing examples of conversion patterns, according to an embodiment of the present invention. FIG. 1 is an explanatory diagram showing an example of a 2D operation determination table according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example of a 3D operation determination table according to an embodiment of the present invention. FIG. 3 is an explanatory diagram showing a processing flow for switching between multiple modes according to an embodiment of the present invention. FIG. 4 is an explanatory diagram showing an example of 2D operation and 2D display according to an embodiment of the present invention. FIG. 5 is an explanatory diagram showing an example of 3D operation and 3D display according to an embodiment of the present invention. FIG. 6 is an explanatory diagram showing an example of a feedback display according to an embodiment of the present invention. FIG. 7 is an explanatory diagram showing an example of a configuration related to a mode information output function according to an embodiment of the present invention. FIG. 8 is an explanatory diagram showing an example of an indicator display image according to an embodiment of the present invention. FIG. 9 is an explanatory diagram showing an example of an audio output according to an embodiment of the present invention. FIG. 10 is an explanatory diagram showing another example of an indicator display according to an embodiment of the present invention. FIG. 11 is an explanatory diagram showing another example of an indicator display according to an embodiment of the present invention. FIG. 12 is an explanatory diagram showing an example of a mode selection display as a modified example according to an embodiment of the present invention. FIG. 13 is an explanatory diagram showing an example of a configuration of modes prepared as options in a space-floating image display device of Example 4 according to an embodiment of the present invention.FIG. 10 is an explanatory diagram showing an example of the configuration of modes prepared as options in a space-floating image display device of Example 4 according to an embodiment of the present invention. FIG. 11 is an explanatory diagram showing a processing flow in a space-floating image display device of Example 4 according to an embodiment of the present invention. FIG. 12 is an explanatory diagram showing a specific example of a switch determination from 2D operation mode B to 3D operation mode B according to an embodiment of the present invention. FIG. 13 is an explanatory diagram showing a specific example of a switch determination from 3D operation mode B to 2D operation mode B according to an embodiment of the present invention. FIG. 14 is an explanatory diagram showing an example of 2D / 3D display switch control in a modified example of Example 4 according to an embodiment of the present invention.

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, components having the same functions are given the same reference numerals, and repeated explanations thereof may be omitted.

[0010] The following examples relate to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member. In the following explanation of the examples, the image floating in space is expressed using the term "floating image in space." Instead of this term, it is also acceptable to express it as "aerial image," "spatial image," "floating image in space," "floating optical image of displayed image," "floating optical image of displayed image," etc. The term "floating image in space," which is mainly used in the explanation of the examples, is used as a representative example of these terms.

[0011] According to the following embodiments, a suitable image display device can be realized for, for example, bank ATMs, train station ticket machines, digital signage, and the like. For example, currently, bank ATMs, train station ticket machines, and the like typically use touch panels, but by using a transparent glass surface or a light-transmitting plate, high-resolution image information can be displayed in a floating state on the glass surface or light-transmitting plate. In this case, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflector, resulting in high light utilization efficiency and suppressing the ghost images that occur in addition to the main floating image, which is a problem with conventional retroreflection systems, thereby achieving a clear floating image. Furthermore, a device including the light source of this embodiment can provide a novel, highly usable floating image display device (floating image display system) that can significantly reduce power consumption. Furthermore, a floating image display device for a vehicle can be provided that can display a so-called unidirectional floating image that can be viewed inside and / or outside the vehicle. Example 1

[0012] <Example of Usage of the Space-Floating Image Display Device> Figure 1 is a diagram showing an example of usage of a space-floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using Figure 2 and other figures. Light with a narrow-angle directional characteristic and specific polarization is emitted from the image display device 1 as an image light beam, reflected by the optical system within the space-floating image display device, and then incident on the retroreflector 2, retroreflected and transmitted through a transparent member 100 (glass, etc.), forming a real aerial image (space-floating image 3) on the outside of the glass surface. In the following examples, the retroreflector 2 (retroreflector) will be used as an example of the retroreflector. However, the retroreflector 2 of the present invention is not limited to a planar plate, and is used as an example of a concept including a sheet-like retroreflector attached to a planar or non-planar member, or an entire assembly in which a sheet-like retroreflector is attached to a planar or non-planar member. Furthermore, since the light rays reflected by the retroreflector 2 have the optical property of forming an image, the retroreflector 2 may also be expressed as an imaging optical member or an imaging optical plate.

[0013] In addition, in a store or the like, a space is partitioned by a show window (also called "window glass") 105, which is a transparent member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit such a transparent member and display a floating image in one direction to the outside and / or inside of the store (space).

[0014] 1, the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing a means for reflecting specific polarized waves in the window glass 105, it is possible to form an aerial image at a desired position inside the store by reflecting the specific polarized waves.

[0015] <Configuration Example of Optical System of Space-Floating Image Display Device> Figure 2A is a diagram showing an example of the configuration of an optical system of a space-floating image display device according to one embodiment of the present invention. The configuration of the space-floating image display device will be described in more detail using Figure 2A. As shown in Figure 2A (1), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light with narrow-angle diffusion characteristics.

[0016] Image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 (in the figure, the polarization separation member 101 is formed in a sheet shape and adhered to the transparent member 100) provided on a transparent member 100 and has a film that selectively reflects image light of a specific polarization, and then enters the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector 2. The image light is polarized and converted from the specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when entering the retroreflector 2 and once when exiting. Here, the polarization separation member 101, which selectively reflects image light of a specific polarization, has the property of transmitting the polarized light of the other polarization that has been polarization-converted, so the image light of the specific polarization after polarization conversion passes through the polarization separation member 101. The image light that has passed through the polarization separation member 101 forms a spatially floating image 3, which is a real image, outside the transparent member 100. 2A shows an example in which the chief ray of the image light incident on the retroreflector 2 is incident at an angle of 90° to the retroreflector 2. However, the incident angle of the chief ray of the image light on the retroreflector 2 is not limited to 90°, and an angle of, for example, 90°±15° can also be used.

[0017] Here, a first example of polarization design for the optical system of FIG. 2A will be described. For example, S-polarized image light may be emitted from the display device 1 to the polarization separation member 101, which may have the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the S-polarized image light reaching the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and travels toward the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the S-polarized light to P-polarized light. The P-polarized image light then travels back toward the polarization separation member 101. Here, the polarization separation member 101 has the property of reflecting S-polarized light and transmitting P-polarized light, so the P-polarized image light passes through the polarization separation member 101 and then through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0018] Next, a second example of polarization design for the optical system of FIG. 2A will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from the display device 1 to the polarization separation member 101, and the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the P-polarized image light that reaches the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and proceeds toward the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the image light from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again toward the polarization separation member 101. Here, the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light, so the S-polarized image light passes through the polarization separation member 101 and then through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0019] The light forming the space-floating image 3 is a collection of light rays that converge from the retroreflector 2 to the optical image of the space-floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the space-floating image 3. Therefore, the space-floating image 3 is an image with high directionality, unlike the diffuse image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2A, when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in systems that display images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0020] Depending on the performance of the retroreflector 2, the polarization axis of the reflected image light may become irregular. The reflection angle may also become irregular. Such irregular light may not maintain the polarization state and propagation angle assumed in the design. For example, light with an unintended polarization state and propagation angle may re-enter the image display surface of the liquid crystal display panel 11 directly from the position of the retroreflector 2 without passing through the polarization separation member. Such light with an unintended polarization state and propagation angle may be reflected by components within the space-floating image display device and then re-enter the image display surface of the liquid crystal display panel 11. Such light re-entering the image display surface of the liquid crystal display panel 11 may be re-reflected by the image display surface of the liquid crystal display panel 11 constituting the display device 1, potentially generating ghost images and degrading the image quality of the space-floating image. Therefore, in this embodiment, an absorbing polarizer 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorbing polarizer 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorbing polarizer 12, thereby suppressing the re-reflection. This makes it possible to prevent degradation of image quality due to ghost images of spatially floating images. Specifically, if the display device 1 is configured to emit S-polarized image light to the polarization separation member 101, the absorbing polarizer 12 may be a polarizer that absorbs P-polarized light. Furthermore, if the display device 1 is configured to emit P-polarized image light to the polarization separation member 101, the absorbing polarizer 12 may be a polarizer that absorbs S-polarized light.

[0021] The polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

[0022] 2A(2) shows an example of the surface shape of a typical retroreflector 2. Light rays incident on the interior of the regularly arranged hexagonal prisms are reflected by the walls and bottoms of the hexagonal prisms and emitted as retroreflected light in a direction corresponding to the incident light, and a real floating image is displayed on the display device 1 based on the image displayed.

[0023] The resolution of this floating image in space depends not only on the resolution of the liquid crystal display panel 11, but also on the outer diameter D and pitch P of the retroreflective portion of the retroreflector 2 shown in Figure 2A (2). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, one pixel of the floating image in space will be equivalent to 300 μm. As a result, the effective resolution of the floating image in space will be reduced to about one-third.

[0024] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Furthermore, it is advisable to arrange the shape so that none of the sides of the retroreflective portion overlaps any of the sides of one pixel of the liquid crystal display panel.

[0025] The surface shape of the retroreflector according to this embodiment is not limited to the above example. Various surface shapes that achieve retroreflection may be used. Specifically, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which triangular pyramidal prisms, hexagonal pyramidal prisms, other polygonal prisms, or a combination thereof are periodically arranged. Alternatively, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which these prisms are periodically arranged to form cube corners. These may also be referred to as corner reflector arrays or polyhedral reflector arrays. Alternatively, the surface of the retroreflector according to this embodiment may be provided with capsule lens-type retroreflection elements in which glass beads are periodically arranged. The detailed configuration of these retroreflection elements can be achieved using existing technology, so a detailed description will be omitted. Specifically, the techniques disclosed in Japanese Patent Laid-Open Nos. 2001-33609, 2001-264525, 2005-181555, 2008-70898, and 2009-229942 may be used.

[0026] <Another configuration example 1 of the optical system of the space-floating image display device> Another configuration example of the optical system of the space-floating image display device will be described using Fig. 2B. In Fig. 2B, components with the same reference numerals as Fig. 2A have the same functions and configurations as Fig. 2A. For such components, repeated explanations will be omitted to simplify the explanation.

[0027] In the optical system of FIG. 2B , similar to FIG. 2A , image light of a specific polarization is output from the display device 1. The image light of a specific polarization output from the display device 1 is input to a polarization separation member 101B. The polarization separation member 101B is a member that selectively transmits image light of a specific polarization. Unlike the polarization separation member 101 of FIG. 2A , the polarization separation member 101B is not integrated with the transparent member 100 but has an independent plate-like shape. Therefore, the polarization separation member 101B may also be referred to as a polarization separation plate. The polarization separation member 101B may be configured as a reflective polarizing plate configured by attaching a polarization separation sheet to a transparent member. Alternatively, the transparent member may be formed of a metal multilayer film that selectively transmits specific polarization and reflects polarization of other specific polarizations. In FIG. 2B , the polarization separation member 101B is configured to transmit image light of a specific polarization output from the display device 1.

[0028] The image light that has passed through the polarization separation member 101B is incident on the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector. The image light is polarized and converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when it enters the retroreflector and once when it leaves. Here, the polarization separation member 101B has the property of reflecting the polarized light of the other polarization that has been polarized and converted by the λ / 4 plate 21, so the image light after polarization conversion is reflected by the polarization separation member 101B. The image light reflected by the polarization separation member 101B passes through the transparent member 100 and forms a space-floating image 3, which is a real image, outside the transparent member 100.

[0029] Here, a first example of polarization design for the optical system of FIG. 2B will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from the display device 1 to the polarization separation member 101B, and the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized image light that reaches the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and proceeds to the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, and the image light is converted from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again to the polarization separation member 101B. Here, the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light, so the S-polarized image light is reflected by the polarization separation member 101 and passes through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the floating image 3 to be formed optimally.

[0030] Next, a second example of polarization design for the optical system of FIG. 2B will be described. For example, S-polarized image light may be emitted from the display device 1 to the polarization separator 101B, which may have the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized image light reaching the polarization separator 101B from the display device 1 passes through the polarization separator 101B and proceeds to the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the S-polarized image light to P-polarized light. The P-polarized image light then proceeds again to the polarization separator 101B. Here, the polarization separator 101B has the property of reflecting P-polarized light and transmitting S-polarized light, so the P-polarized image light is reflected by the polarization separator 101 and passes through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the floating image 3 to be formed optimally.

[0031] In FIG. 2B , the image display surface of the display device 1 and the surface of the retroreflector 2 are arranged parallel to each other. The polarization separation member 101B is arranged at an angle α (e.g., 30°) with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. Then, upon reflection by the polarization separation member 101B, the direction of travel of the image light reflected by the polarization separation member 101B (the direction of the chief ray of the image light) differs by an angle β (e.g., 60°) from the direction of travel of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light). With this configuration, in the optical system of FIG. 2B , the image light is output at a predetermined angle shown toward the outside of the transparent member 100, forming a real image, the floating-in-space image 3. In the configuration of FIG. 2B , when viewed by a user from the direction of arrow A, the floating-in-space image 3 is perceived as a bright image. However, when viewed by another person from the direction of arrow B, the floating-in-space image 3 cannot be perceived as an image at all. This characteristic is extremely suitable for use in a system that displays images that require high security or highly confidential images that should be concealed from people directly facing the user.

[0032] As described above, the optical system of FIG. 2B has a different configuration from the optical system of FIG. 2A, but can form a suitable floating image in space, similar to the optical system of FIG. 2A.

[0033] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized light of the image light from the polarization separation member 101B and absorb the polarized light that is 90° out of phase with the polarized light of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2B due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0034] <Another configuration example 2 of the optical system of the space-floating image display device> Another configuration example of the optical system of the space-floating image display device will be described using Fig. 2C. Note that in Fig. 2C, components with the same reference numerals as Fig. 2B have the same functions and configurations as Fig. 2B. For the sake of simplicity, such components will not be described repeatedly.

[0035] The only difference between the optical system of Fig. 2B and the optical system of Fig. 2C is the arrangement angle of the polarization separation member 101B with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. All other configurations are the same as those of the optical system of Fig. 2B, so repeated explanations will be omitted. The polarization design of the optical system of Fig. 2C is also the same as that of the optical system of Fig. 2B, so repeated explanations will be omitted.

[0036] In the optical system of FIG. 2C , the polarization separation member 101B is tilted at an angle α with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. In FIG. 2C , the angle α is 45°. With this configuration, when the polarization separation member 101B reflects, the angle β between the direction of propagation of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light) and the direction of propagation of the image light reflected by the polarization separation member 101B (the direction of the chief ray of the image light) is 90°. With this configuration, the image display surface of the display device 1 and the surface of the retroreflector 2 are perpendicular to the direction of propagation of the image light reflected by the polarization separation member 101B, thereby simplifying the angular relationship between the surfaces that make up the optical system. If the surface of the transparent member 100 is positioned perpendicular to the direction of propagation of the image light reflected by the polarization separation member 101B, the angular relationship between the surfaces that make up the optical system can be further simplified. In the configuration of Figure 2C, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0037] As described above, the optical system of Fig. 2C has a different configuration from the optical systems of Fig. 2A and Fig. 2B, but can form a suitable floating image in space similar to the optical systems of Fig. 2A and Fig. 2B. In addition, the angles of the surfaces constituting the optical system can be made simpler.

[0038] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized light of the image light from the polarization separation member 101B and absorb the polarized light that is 90° out of phase with the polarized light of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2C due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0039] <Another Configuration Example 3 of the Optical System of the Space-Floating Image Display Device> Another configuration example of the optical system of the space-floating image display device will be described using FIG. 2D. The optical system of FIG. 2D is an optical system that uses a retroreflector 5 that is different from the retroreflector 2 used in FIGS. 2A to 2C. Hereinafter, Another Configuration Example 3 of the optical system will be described in more detail using FIGS. 2D to 2I. In FIG. 2D, components that are assigned the same reference numerals as those in FIGS. 2A to 2C have the same functions and configurations as those in FIGS. 2A to 2C. Such components will not be described repeatedly in order to simplify the explanation.

[0040] 2D is a diagram showing an example of the main components and retroreflection components of a space-floating image display device according to an embodiment of the present invention. A display device 10 that emits image light is provided obliquely on a transparent member 100 such as glass. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light.

[0041] A chief ray 9020 representing the light beam emitted from the display device 10 travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α may be, for example, 45°. However, the incident angle α is not limited to 45°, and may also be, for example, 45°±15°.

[0042] The retroreflector 5 is an optical element having the optical property of retroreflecting light rays in at least some directions. Furthermore, since the reflected light rays have the optical property of forming an image, the retroreflector 5 may also be referred to as an imaging optical element or an imaging optical plate.

[0043] 2E, 2F, etc., the principal ray 9020 travels in the z direction and is retroreflected in the x and y directions by the retroreflector 5. As a result, the reflected light ray 9021 travels in a direction away from the retroreflector 5 along an optical path that is mirror-symmetrical with respect to the principal ray 9020 with the retroreflector 5 as the reference, passes through the transparent member 100, and forms the floating image 3 in space as a real image on the imaging plane.

[0044] The light beam forming the space-floating image 3 is a collection of light rays that converge from the retroreflector 5 to the optical image of the space-floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the space-floating image 3. Therefore, the space-floating image 3 is an image with high directionality, unlike a diffuse image formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2, when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be perceived as an image at all. This characteristic is suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0045] An example of the configuration of the retroreflector 5 will be described using Figures 2E and 2F. The retroreflector 5 has a configuration in which a plurality of corner reflectors 9040 are arranged in an array on the surface of a transparent member 50. This may also be called a corner reflector array or a polyhedral reflector array. The specific configuration of the corner reflector 9040 will be described in detail using Figures 2G, 2H, and 2I. Light rays 9111, 9112, 9113, and 9114 emitted from a light source 9110 are reflected twice by two mirror surfaces 9041 and 9042 of the corner reflector 9040, becoming reflected light rays 9121, 9122, 9123, and 9124. This double reflection is retroreflection in the x and y directions, where the light is reflected back in the same direction as the incident direction (traveling in a direction rotated 180 degrees), and in the z direction, where the angle of incidence and the angle of reflection match due to total reflection.

[0046] That is, the light rays 9111 to 9114 generate reflected light rays 9121 to 9124 on straight lines symmetrical in the z direction with respect to the corner reflector 9040, forming an aerial real image 9120. Note that the light rays 9111 to 9114 emitted from the light source 9110 are four light rays representing the diffused light from the light source 9110, and although the light rays incident on the retroreflector 5 are not limited to these, depending on the diffusion characteristics of the light source 9110, all incident light rays cause similar reflections and form an aerial real image 9120. Note that to make the drawing easier to see, the position of the light source 9110 and the position of the aerial real image 9120 are shown shifted in the x direction, but in reality, the position of the light source 9110 and the position of the aerial real image 9120 in the x direction are the same, and are overlapping when viewed from the z direction.

[0047] 2G, 2H, and 2I, the configuration and effects of the corner reflector 9040 that constitutes the retroreflector 5 will be described. The corner reflector 9040 is a rectangular parallelepiped with only two specific surfaces being mirror surfaces 9041 and 9042, and the other four surfaces being made of transparent materials. The retroreflector 5 has a configuration in which these corner reflectors 9040 are arrayed so that the corresponding mirror surfaces face in the same direction.

[0048] When viewed from the top (+z direction), a light ray 9111 emitted from the light source 9110 enters the mirror surface 9041 (or the mirror surface 9042) at a specific angle of incidence, is totally reflected at the reflection point 9130, and then is totally reflected again at the reflection point 9132 on the mirror surface 9042 (or the mirror surface 9041).

[0049] If the angle of incidence of light ray 9111 with respect to mirror surface 9041 (or mirror surface 9042) is θ, then the angle of incidence of first reflected light ray 9131 reflected by mirror surface 9041 (or mirror surface 9042) with respect to mirror surface 9042 (or mirror surface 9041) can be expressed as 90°-θ. Therefore, with respect to light ray 9111, second reflected light ray 9121 undergoes a rotation of 2θ after the first reflection and 2×(90°-θ) after the second reflection, resulting in a total reversal optical path of 180°. On the other hand, when viewed from the side (the direction halfway between -x and -y), total reflection in the z direction occurs only once. Therefore, if the angle of incidence with respect to mirror surface 9041 or mirror surface 9042 is φ, then reflected light ray 9121 undergoes a rotation of 2×φ after one reflection with respect to light ray 9111.

[0050] As described above, the light rays incident on the corner reflector 9040 undergo retroreflection, which results in an inverted optical path in the x and y directions, and specular reflection due to total reflection in the z direction. Considering the retroreflector 5, similar reflection occurs in each optical path, so that an image is formed at a point symmetrical with respect to the z axis direction by an inverted optical path that is convergent in the x and y directions.

[0051] 2A to 2C, the retroreflector 2 has retroreflection properties in three axes. As a result, when a diffusive incident light beam is incident on the retroreflector 2, a convergent reflected light beam travels toward the side of the retroreflector 2 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air, forming a floating image 3. The traveling direction of the chief ray of the convergent reflected light beam reflected from the retroreflector 2 is opposite to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 2.

[0052] 2D, the retroreflector 5 has retroreflection properties in two axial directions and specular reflection in the other axial direction. As a result, when a diffusive incident light beam is incident on the retroreflector 5, the convergent reflected light beam is reflected by the corner reflector array and travels in the direction opposite to the side of the retroreflector 5 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air and forms the space floating image 3.

[0053] The traveling direction of the chief ray of the convergent reflected light beam reflected by the corner reflector array of the retroreflector 5 is not the opposite direction to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5. The normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5 and the normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray after being reflected by the retroreflector 5 and becoming a convergent reflected light beam, travel in a straight line, unchanged before and after reflection by the corner reflector array.

[0054] That is, the diffusive incident light beam is converted into a convergent reflected light beam by reflection on the retroreflector 5, but in the normal direction to the plate-shaped surface of the retroreflector 5, the light beam travels as if passing through the retroreflector 5. Here, the diffusive incident light beam incident on the retroreflector 5 and the convergent reflected light beam emerging from the retroreflector 5 are in a geometrically symmetrical relationship with respect to the plate-shaped surface of the retroreflector 5.

[0055] The resolution of the space-floating image formed by the light beams from the video output unit 10 depends not only on the resolution of the liquid crystal display panel 11, but also on the diameter D and pitch P (not shown) of the retroreflective portion of the retroreflector 5 shown in Figures 2E and 2F. For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch P is 300 μm, one pixel of the space-floating image will be equivalent to 300 μm. As a result, the effective resolution of the space-floating image will be reduced to about one-third.

[0056] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 10, it is desirable to make the diameter D and pitch P of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire due to the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Furthermore, it is advisable to arrange the shape of the retroreflective portion so that none of its sides overlaps any of the sides of one pixel of the liquid crystal display panel.

[0057] The shape of the retroreflector (imaging optical plate) according to this embodiment is not limited to the above example. It may have various shapes that achieve retroreflection. Specifically, it may be a variety of cubic corner bodies, corner reflector arrays, slit mirror arrays, dihedral corner reflector arrays, polyhedral reflector arrays, or shapes in which a combination of these reflective surfaces is periodically arranged. Alternatively, capsule lens-type retroreflecting elements in which glass beads are periodically arranged may be provided on the surface of the retroreflector according to this embodiment. Since existing technology can be used for the detailed configuration of these retroreflecting elements, a detailed description will be omitted. Specifically, the technology disclosed in JP 2017-33005 A, JP 2019-133110 A, JP 2017-67933 A, WO 2009 / 131128 A, etc. may be used.

[0058] 2D, the image light emitted from the display device 10 may be in any polarization state, either S-polarized or P-polarized.

[0059] As described above, the optical system of FIG. 2D is an optical system that uses a retroreflector different from the optical systems of FIGS. 2A to 2C, but it can form a more suitable floating image in space, similar to the optical systems of FIGS. 2A to 2C.

[0060] According to the optical systems of FIGS. 2A, 2B, 2C, and 2D described above, it is possible to provide brighter and higher quality floating images in space.

[0061] <<Block diagram of the internal configuration of the space floating image display device>>

[0062] Next, a description will be given of a block diagram of the internal configuration of the space-floating image display device 1000. Fig. 3 is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

[0063] The space-floating image display device 1000 includes a retroreflection unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power supply input interface 1111, an operation input unit 1107, a nonvolatile memory 1108, a memory 1109, a control unit 1110, a video signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, a microphone 1139, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc. In addition, the space-floating image display device 1000 may also include a removable media interface 1134, an attitude sensor 1113, a transmissive self-luminous image display device 1650, a second display device 1680, or a secondary battery 1112.

[0064] Each component of the space floating image display device 1000 is disposed in a housing 1190. Note that the imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0065] The retroreflecting portion 1101 in Fig. 3 corresponds to the retroreflector 2 in Fig. 2A, Fig. 2B, and Fig. 2C. The retroreflecting portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected from the retroreflecting portion 1101, the light output to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0066] The image display unit 1102 in Fig. 3 corresponds to the liquid crystal display panel 11 in Fig. 2A, 2B, and 2C. The light source 1105 in Fig. 3 corresponds to the light source device 13 in Fig. 2A, 2B, and 2C. The image display unit 1102, the light guide 1104, and the light source 1105 in Fig. 3 correspond to the display device 1 in Fig. 2A, 2B, and 2C.

[0067] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 (described later). The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0068] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power source 1106 converts AC current input from the outside via the external power input interface 1111 into DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each component within the space-floating image display device 1000. The secondary battery 1112 stores the power supplied from the power source 1106. The secondary battery 1112 also supplies power to the light source 1105 and other components requiring power via the external power input interface 1111 when power is not supplied from the outside. In other words, when the space-floating image display device 1000 is equipped with the secondary battery 1112, the user can use the space-floating image display device 1000 even when power is not supplied from the outside.

[0069] The light guide 1104 guides light generated by the light source 1105 and irradiates it onto the video display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called a backlight for the video display unit 1102. The light guide 1104 may be configured mainly using glass. The light guide 1104 may be configured mainly using plastic. The light guide 1104 may be configured using a mirror. Various methods are possible for combining the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0070] The aerial operation detection sensor 1351 is a sensor that detects an operation of the floating in space image 3 by the finger of the user 230. The aerial operation detection sensor 1351 senses, for example, an area that overlaps with the entire display area of ​​the floating in space image 3. Note that the aerial operation detection sensor 1351 may only sense an area that overlaps with at least a portion of the display area of ​​the floating in space image 3.

[0071] Specific examples of the aerial operation detection sensor 1351 include a distance sensor that uses invisible light such as infrared light, an invisible light laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may also be configured to detect coordinates on a two-dimensional plane by combining multiple sensors. The aerial operation detection sensor 1351 may also be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) sensor or an image sensor.

[0072] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect touch operations, etc., made by the user with their finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0073] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and determines whether or not the finger of the user 230 has made contact with an object in the floating-in-space image 3 based on the sensing signal, and calculates the position (contact position) where the finger of the user 230 has made contact with the object. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array), for example. Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0074] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally as separate entities from the space-floating image display device 1000. When provided as separate entities from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to be able to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0075] Furthermore, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the space-floating image display device 1000, which does not have an aerial operation detection function, is used as the main body, and only the aerial operation detection function can be added as an option. Furthermore, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the space-floating image display device 1000 may be used. In cases where it is desired to more freely position the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0076] The imaging unit 1180 is a camera with an image sensor, and captures images of the space near the floating-in-space image 3 and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180, or by using an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in detecting the touch operation of the floating-in-space image 3 by the user 230. The imaging unit 1180 may be provided separately from the floating-in-space image display device 1000. When the imaging unit 1180 is provided separately from the floating-in-space image display device 1000, it is sufficient to configure it so that an imaging signal can be transmitted to the floating-in-space image display device 1000 via a wired or wireless communication connection path, etc.

[0077] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that targets a plane (intrusion detection plane) including the display surface of the spatial floating image 3 and detects whether or not an object has intruded into this intrusion detection plane, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object that has not intruded into the intrusion detection plane (for example, a user's finger) is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0078] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the images captured by the multiple image capturing units 1180 and object depth information from the depth sensor. These pieces of information and various other information such as the distance between the object and the intrusion detection plane are then used for various display controls for the floating in space image 3.

[0079] Furthermore, without using the aerial operation detection sensor 1351 , the aerial operation detection unit 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging unit 1180 .

[0080] Furthermore, the imaging unit 1180 may capture an image of the face of the user 230 operating the space-floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not there is another person standing around or behind the user 230 operating the space-floating image 3 and peeking at the operation of the user 230 on the space-floating image 3, the imaging unit 1180 may capture an image of the user 230 operating the space-floating image 3 and a range including the area surrounding the user 230.

[0081] The operation input unit 1107 is, for example, an operation button, a signal receiving unit such as a remote controller, or an infrared light receiving unit, and inputs signals for operations different from the aerial operation (touch operation) by the user 230. Apart from the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may also be used by, for example, an administrator to operate the space floating image display device 1000.

[0082] The video signal input unit 1131 is connected to an external video output device and inputs video data. Various digital video input interfaces are possible for the video signal input unit 1131. For example, the video signal input unit 1131 may be configured as a video input interface conforming to the HDMI (registered trademark) standard (High-Definition Multimedia Interface), a video input interface conforming to the DVI (Digital Visual Interface) standard, or a video input interface conforming to the DisplayPort standard.

[0083] Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio signal input unit 1133 may be configured as an HDMI-standard audio input interface, an optical digital terminal interface, a coaxial digital terminal interface, or the like. In the case of an HDMI-standard interface, the video signal input unit 1131 and the audio signal input unit 1133 may be configured as an interface in which a terminal and a cable are integrated. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may be configured as a speaker.

[0084] The audio output unit 1140 may also output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 1140 may be configured to output a digital signal to an external device, like the Audio Return Channel function defined in the HDMI standard. The microphone 1139 is a microphone that collects sounds around the space-floating image display device 1000, converts them into signals, and generates audio signals. The microphone may record a person's voice, such as a user's voice, and the control unit 1110, described later, may perform voice recognition processing on the generated audio signal to obtain text information from the audio signal.

[0085] The non-volatile memory 1108 stores various data used by the space floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations to be displayed on the space floating image 3, display icons, data and layout information for objects to be operated by user operations, etc. The memory 1109 stores image data to be displayed as the space floating image 3, data for controlling the device, etc.

[0086] The control unit 1110 controls the operation of each connected unit. In addition, the control unit 1110 may cooperate with a program stored in the memory 1109 to perform calculations based on information acquired from each unit in the space floating image display device 1000.

[0087] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. If the communication unit 1132 has a wired communication interface, the wired communication interface may be configured, for example, by an Ethernet-standard LAN interface. If the communication unit 1132 has a wireless communication interface, the interface may be configured, for example, by a Wi-Fi communication interface, a Bluetooth communication interface, or a mobile communication interface such as 4G or 5G. Various types of data, such as video data, image data, and audio data, are transmitted and received through communication via the communication unit 1132.

[0088] The removable media interface 1134 is an interface for connecting a removable recording medium (removable media). The removable recording medium (removable media) may be composed of a semiconductor device memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disk. The removable media interface 1134 can read various information such as video data, image data, and audio data recorded on the removable recording medium. The video data, image data, etc. recorded on the removable recording medium are output as the floating image 3 via the video display unit 1102 and the retroreflection unit 1101.

[0089] The storage unit 1170 is a storage device that records various types of information, such as video data, image data, and audio data. The storage unit 1170 may be configured with a magnetic recording medium recording device, such as a hard disk drive (HDD), or a semiconductor element memory, such as a solid state drive (SSD). For example, various types of information, such as video data, image data, and audio data, may be recorded in advance in the storage unit 1170 at the time of product shipment. The storage unit 1170 may also record various types of information, such as video data, image data, and audio data, acquired from an external device, an external server, or the like via the communication unit 1132.

[0090] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflection unit 1101. The video data, image data, etc. of display icons, objects for the user to operate, etc., displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0091] Layout information of display icons, objects, etc. displayed as the spatial floating image 3, and various metadata information related to the objects, etc. are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0092] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. The video control unit 1160 may be referred to as a video processing circuit, and may be configured with hardware such as an ASIC, FPGA, or video processor. The video control unit 1160 may also be referred to as a video processing unit or an image processing unit. The video control unit 1160 controls video switching, such as determining which video signal to input to the video display unit 1102, between the video signal to be stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131, for example.

[0093] In addition, the image control unit 1160 may generate a superimposed image signal by superimposing the image signal to be stored in the memory 1109 and the image signal input from the image signal input unit 1131, and input the superimposed image signal to the image display unit 1102, thereby controlling the formation of a composite image as a floating image in space 3.

[0094] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing that enlarges, reduces, or deforms the image, brightness adjustment processing that changes the brightness, contrast adjustment processing that changes the contrast curve of the image, and Retinex processing that decomposes the image into light components and changes the weighting of each component.

[0095] Furthermore, the video control unit 1160 may perform special effect video processing or the like on the video signal input to the video display unit 1102 to assist the aerial operation (touch operation) of the user 230. The special effect video processing is performed based on, for example, the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 or the image of the user 230 captured by the imaging unit 1180.

[0096] The attitude sensor 1113 is a sensor configured with a gravity sensor, an acceleration sensor, or a combination of these, and can detect the attitude in which the space-floating image display device 1000 is installed. Based on the attitude detection result of the attitude sensor 1113, the control unit 1110 may control the operation of each connected unit. For example, if an undesirable attitude is detected as the user's usage state, the control unit 1110 may perform control such that the image displayed on the image display unit 1102 is stopped and an error message is displayed to the user. Alternatively, if the attitude sensor 1113 detects a change in the installation attitude of the space-floating image display device 1000, the control unit 1110 may perform control such that the display orientation of the image displayed on the image display unit 1102 is rotated.

[0097] As explained above, various functions are installed in the space floating image display device 1000. However, the space floating image display device 1000 does not need to have all of these functions, and any configuration is acceptable as long as it has the function of forming the space floating image 3.

[0098] <Configuration Example of Space-Floating Image Display Device> Next, a configuration example of the space-floating image display device will be described. The layout of the components of the space-floating image display device according to this embodiment can be various depending on the usage form. Below, the layouts of each of Figs. 4A to 4M will be described. Note that in each of the examples of Figs. 4A to 4M, the thick line surrounding the space-floating image display device 1000 indicates an example of the housing structure of the space-floating image display device 1000.

[0099] FIG. 4A is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4A is equipped with an optical system corresponding to the optical system shown in FIG. 2A. The space-floating image display device 1000 shown in FIG. 4A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4A, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user's finger. Note that the x direction is the left-right direction as seen from the user, the y direction is the front-back direction (depth direction) as seen from the user, and the z direction is the up-down direction (vertical direction). Hereinafter, the definitions of the x direction, y direction, and z direction are the same in each drawing of FIG. 4, so repeated explanations will be omitted.

[0100] FIG. 4B is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4B is equipped with an optical system corresponding to the optical system of FIG. 2A. The space-floating image display device 1000 shown in FIG. 4B is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4B, the space-floating image display device is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user 230's finger. 4B , the aerial operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so that this configuration can improve the accuracy of touch detection.

[0101] FIG. 4C is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4C is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4C is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4C, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user's 230 finger.

[0102] FIG. 4D is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4D is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4D is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4D, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger. 4D , the aerial operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so that this configuration can improve the accuracy of touch detection.

[0103] FIG. 4E is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4E is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4E is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4E, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels directly upward. When the mid-air operation detection sensor 1351 is provided as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user's 230 finger.

[0104] FIG. 4F is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4F is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4F is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4F, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels toward the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger.

[0105] FIG. 4G is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4G is equipped with an optical system corresponding to the optical system shown in FIG. 2C. In the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the central optical path of the image light emitted from the display device 1 was on the yz plane. That is, in the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the image light traveled in the front-to-back and up-to-down directions as seen from the user. In contrast, in the optical system of the space-floating image display device shown in FIG. 4G, the central optical path of the image light emitted from the display device 1 is on the xy plane. That is, in the optical system of the space-floating image display device shown in FIG. 4G, the image light travels in the left-to-right and front-to-back directions as seen from the user. In the space-floating image display device 1000 shown in FIG. 4G, the surface on which the space-floating image 3 is formed is installed so that it faces the front of the device (toward the user 230). 4G, the space-floating image display device 1000 has the transparent member 100 installed on the front side of the device (toward the user 230). The space-floating image 3 is formed on the user side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels towards the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0106] 4H is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4H differs from the space-floating image display device of FIG. 4G in that it has a window with a transparent plate 100B made of glass or plastic on the back of the device (opposite the position where the user 230 views the space-floating image 3, i.e., opposite the traveling direction of the image light of the space-floating image 3 toward the user 230). The rest of the configuration is the same as the space-floating image display device of FIG. 4G , so repeated explanations will be omitted. The space-floating image display device 1000 of FIG. 4H has a window with a transparent plate 100B on the opposite side of the traveling direction of the image light of the space-floating image 3 from the space-floating image 3. Therefore, when the user 230 views the space-floating image 3, they can recognize the scenery behind the space-floating image display device 1000 as the background of the space-floating image 3. Therefore, the user 230 can perceive the space floating image 3 as floating in the air in front of the scenery behind the space floating image display device 1000. This can further emphasize the floating feeling of the space floating image 3.

[0107] Depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and head toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, in order to prevent this stray light, the transparent plate 100B may not be provided in the window on the back of the space-floating image display device 1000.

[0108] Fig. 4I is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4I differs from the space-floating image display device of Fig. 4H in that a light-blocking door 1410 is provided in the window of the transparent plate 100B located on the back of the device (opposite the position where the user 230 views the space-floating image 3). The rest of the configuration is the same as that of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted.

[0109] The opening and closing door 1410 of the space-floating image display device 1000 in FIG. 4I has, for example, a light blocking plate and is equipped with a mechanism for moving (sliding), rotating, or attaching / detaching the light blocking plate, thereby switching between an open state and a light blocking state for the window (rear window) of the transparent plate 100B located at the back of the space-floating image display device 1000. The movement (sliding) and rotation of the light blocking plate by the opening and closing door 1410 may be electrically driven by a motor (not shown). The motor may be controlled by the control unit 1110 in FIG. 3. Note that the example in FIG. 4I discloses an example in which the opening and closing door 1410 has two light blocking plates. In contrast, the opening and closing door 1410 may have only one light blocking plate.

[0110] For example, if the view seen through the window of the transparent plate 100B of the space-floating image display device 1000 is outdoors, the brightness of sunlight varies depending on the weather. When the outdoor sunlight is strong, the background of the space-floating image 3 may become too bright, reducing the user 230's visibility of the space-floating image 3. In such a case, by moving (sliding), rotating, or attaching the light-shielding plate of the opening / closing door 1410 to block the light from the rear window, the background of the space-floating image 3 becomes dark, thereby relatively increasing the visibility of the space-floating image 3. Such a blocking operation by the light-shielding plate of the opening / closing door 1410 may be performed directly by the force of the user 230's hand. In response to an operation input via the operation input unit 1107 of FIG. 3 , the control unit 1110 may control a motor (not shown) to perform the blocking operation by the light-shielding plate of the opening / closing door 1410.

[0111] An illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 may control a motor (not shown) to perform the opening and closing operation of the light blocking plate of the opening and closing door 1410 according to the detection result of the illuminance sensor. By controlling the opening and closing operation of the light blocking plate of the opening and closing door 1410 in this way, it is possible to more suitably maintain the visibility of the space-floating image 3, even if the user 230 does not manually open or close the light blocking plate of the opening and closing door 1410.

[0112] Furthermore, the light blocking plate provided by the opening and closing door 1410 may be manually detachable. Depending on the intended use and installation environment of the space floating image display device 1000, the user can select whether to leave the rear window open or in a light blocking state. If it is planned to use the rear window in a light blocking state for a long period of time, the detachable light blocking plate can be fixed in the light blocking state. Furthermore, if it is planned to use the rear window in an open state for a long period of time, the detachable light blocking plate can be left detached. The light blocking plate may be attached and detached using screws, a hook structure, or a fitting structure.

[0113] Even in the example of the space-floating image display device 1000 shown in FIG. 4I, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and directed toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, to prevent this stray light, the window on the back of the space-floating image display device 1000 may be configured without the transparent plate 100B. The above-described opening / closing door 1410 may be provided in a window that does not have the transparent plate 100B. To prevent this stray light, it is desirable that the inner surface of the housing of the light-shielding plate of the above-described opening / closing door 1410 have a coating or material with low light reflectance.

[0114] Fig. 4J is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4J differs from the space-floating image display device of Fig. 4H in that instead of arranging a transparent plate 100B made of glass or plastic on the rear side window, an electronically controlled transmittance variable device 1620 is arranged. The other configurations are the same as those of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted. An example of the electronically controlled transmittance variable device 1620 is a liquid crystal shutter.

[0115] That is, the liquid crystal shutter can control the amount of light transmitted by controlling the voltage of the liquid crystal element sandwiched between two polarizing plates. Therefore, by controlling the liquid crystal shutter to increase the transmittance, the scenery through the rear window can be seen through the background of the space-floating image 3. Furthermore, by controlling the liquid crystal shutter to increase the transmittance, the scenery through the rear window can be hidden as the background of the space-floating image 3. Furthermore, since the liquid crystal shutter can control the intermediate length, it can also be set to a transmittance of 50% or more. For example, the control unit 1110 can control the transmittance of the electronically controlled transmittance variable device 1620 in response to an operation input via the operation input unit 1107 of FIG. 3 . With this configuration, in cases where a view through the rear window is desired as the background of the space-floating image 3, but the view through the background rear window is too bright and reduces the visibility of the space-floating image 3, the visibility of the space-floating image 3 can be adjusted by adjusting the transmittance of the electronically controlled transmittance variable device 1620.

[0116] In addition, an illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 controls the transmittance of the electronically controlled transmittance variable device 1620 according to the detection result of the illuminance sensor. In this way, even if the user 230 does not perform an operation input via the operation input unit 1107 of Fig. 3, the transmittance of the electronically controlled transmittance variable device 1620 can be adjusted according to the brightness of the space beyond the rear window, making it possible to more suitably maintain the visibility of the space-floating image 3.

[0117] In the above example, a liquid crystal shutter has been described as an example of the electronically controlled transmittance varying device 1620. However, electronic paper may be used as another example of the electronically controlled transmittance varying device 1620. The same effect as described above can be obtained when electronic paper is used. Furthermore, electronic paper consumes very little power to maintain a halftone state. Therefore, a space floating image display device with lower power consumption can be realized compared to when a liquid crystal shutter is used.

[0118] Fig. 4K is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4K differs from the space-floating image display device of Fig. 4G in that it has a transmissive self-luminous image display device 1650 instead of the transparent member 100. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0119] In the space-floating image display device 1000 of FIG. 4K, an image luminous flux passes through the display surface of the transmissive self-luminous image display device 1650, and then a space-floating image 3 is formed outside the space-floating image display device 1000. That is, when an image is displayed on the transmissive self-luminous image display device 1650, which is a two-dimensional flat display, the space-floating image 3 can be displayed as a pop-up image further in front of the image displayed on the transmissive self-luminous image display device 1650. In this case, the user 230 can simultaneously view two images at different depth positions. The transmissive self-luminous image display device 1650 may be configured using existing technology, such as a transmissive organic EL panel, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2014-216761. Although the transmissive self-luminous image display device 1650 is not shown in FIG. 3, it may be configured to be connected to other processing units, such as the control unit 1110, as a component of the space-floating image display device 1000 of FIG. 3.

[0120] Here, if the transmissive self-luminous image display device 1650 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise image experience.

[0121] Furthermore, if the interior of the space-floating image display device 1000 is kept in a light-blocking state, the background of the transmissive self-luminous image display device 1650 will be sufficiently dark. Therefore, when no image is displayed on the display device 1 or the light source of the display device 1 is turned off and an image is displayed only on the transmissive self-luminous image display device 1650, the transmissive self-luminous image display device 1650 appears to the user 230 as a normal two-dimensional flat display rather than a transmissive display (since the space-floating image 3 in the embodiment of the present invention is displayed as a real optical image in a space without a screen, if the light source of the display device 1 is turned off, the intended display position of the space-floating image 3 will become empty space). Therefore, when the transmissive self-luminous image display device 1650 is used to display an image as if it were a normal two-dimensional flat display, characters, objects, etc. can suddenly be displayed in the air as the space-floating image 3, thereby providing the user 230 with a more effective surprise video experience.

[0122] Note that the darker the interior of the space-floating image display device 1000, the more the transmissive self-luminous image display device 1650 appears like a two-dimensional flat display. Therefore, an absorptive polarizer (not shown) that transmits the polarized light of the image light reflected by the polarization separation member 101B and absorbs polarized light that is 90° out of phase with the polarized light may be provided on the interior surface of the transmissive self-luminous image display device 1650 (the surface on which the image light reflected by the polarization separation member 101B enters the transmissive self-luminous image display device 1650, i.e., the surface of the transmissive self-luminous image display device 1650 opposite the space-floating image 3). While this does not have a significant effect on the image light that forms the space-floating image 3, it can significantly reduce the light that enters the space-floating image display device 1000 from the outside through the transmissive self-luminous image display device 1650, making the interior of the space-floating image display device 1000 darker, which is preferable.

[0123] 4L is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4L is a modified example of the space-floating image display device of FIG. 4K. The orientation of the components in the space-floating image display device 1000 is different from that of the space-floating image display device of FIG. 4K, and is closer to the arrangement of the space-floating image display device of FIG. 4F. The functions and operations of each component are the same as those of the space-floating image display device of FIG. 4K, so repeated explanations will be omitted.

[0124] In the space-floating image display device of FIG. 4L, after the luminous flux of image light passes through the transmissive self-luminous image display device 1650, a space-floating image 3 is formed on the user 230 side of the transmissive self-luminous image display device 1650.

[0125] In both the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, the space-floating image 3 is displayed superimposed on the image of the transmissive self-luminous image display device 1650 from the user 230. Here, the position of the space-floating image 3 and the position of the image of the transmissive self-luminous image display device 1650 are configured to have a difference in the depth direction. Therefore, when the user moves their head (the position of the viewpoint), they can recognize the depth of the two images due to parallax. Therefore, by displaying two images at different depth positions, it is possible to provide the user with a more suitable three-dimensional image experience with the naked eye without the need for stereoscopic glasses or the like.

[0126] Fig. 4M is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4M is provided with a second display device 1680 on the rear side as seen from the user relative to the polarization separation member 101B of the space-floating image display device of Fig. 4G. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0127] In the configuration example shown in Fig. 4M, the second display device 1680 is provided behind the display position of the space-floating image 3, and its image display surface faces the space-floating image 3. With this configuration, from the user 230's perspective, the image of the second display device 1680 and the space-floating image 3, which are displayed at two different depth positions, can be viewed superimposed on each other. In other words, the second display device 1680 is positioned so as to display an image in the direction of the user 230 viewing the space-floating image 3. Although the second display device 1680 is not shown in Fig. 3, it may be configured to be connected to other processing units such as the control unit 1110 as one component of the space-floating image display device 1000 of Fig. 3.

[0128] Note that the image light of the second display device 1680 of the space-floating image display device 1000 of FIG. 4M is viewed by the user 230 after passing through the polarization separation member 101B. Therefore, in order for the image light of the second display device 1680 to more suitably pass through the polarization separation member 101B, it is desirable that the image light output from the second display device 1680 be polarized with a polarization in a vibration direction that the polarization separation member 101B more suitably transmits. In other words, it is desirable that the image light be polarized with a polarization in the same vibration direction as the polarization of the image light output from the display device 1. For example, if the image light output from the display device 1 is S-polarized, it is desirable that the image light output from the second display device 1680 is also S-polarized. Furthermore, if the image light output from the display device 1 is P-polarized, it is desirable that the image light output from the second display device 1680 is also P-polarized.

[0129] The example of the space-floating image display device of FIG. 4M also has the same effect as the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L in that a second image is displayed behind the space-floating image 3. However, unlike the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, in the example of the space-floating image display device of FIG. 4M, the luminous flux of image light for forming the space-floating image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-luminous image display device, but may be a liquid crystal display, which is a two-dimensional flat display. The second display device 1680 may also be an organic EL display. Therefore, in the example of the space-floating image display device of FIG. 4M, it is possible to realize the space-floating image display device 1000 at a lower cost than the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L.

[0130] Here, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, there is a possibility that part of the image light output from the display device 1 will be reflected by the polarization separation member 101B and head toward the second display device 1680. This light (part of the image light) may be reflected again by the surface of the second display device 1680 and may be visually recognized by the user as stray light.

[0131] Therefore, to prevent this stray light, an absorptive polarizer may be provided on the surface of the second display device 1680. In this case, the absorptive polarizer may be an absorptive polarizer that transmits the polarized waves of the image light output from the second display device 1680 and absorbs polarized waves that are 90° out of phase with the polarized waves of the image light output from the second display device 1680. If the second display device 1680 is a liquid crystal display, an absorptive polarizer is also present on the image output side inside the liquid crystal display. However, if a cover glass (cover glass on the image display surface side) is further provided on the output surface of the absorptive polarizer on the image output side inside the liquid crystal display, it is not possible to prevent stray light caused by reflection of the cover glass by light from outside the liquid crystal display. Therefore, it is necessary to separately provide the above-mentioned absorptive polarizer on the surface of the cover glass.

[0132] When an image is displayed on the second display device 1680, which is a two-dimensional flat display, the floating-in-space image 3 can be displayed as an image further in front of the user in relation to the image on the second display device 1680. In this case, the user 230 can simultaneously view two images at different depth positions. By displaying a character on the floating-in-space image 3 and a background on the second display device 1680, it is possible to provide the effect that the user 230 is viewing the space in which the character exists in three dimensions.

[0133] Furthermore, if the second display device 1680 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise visual experience.

[0134] Next, Fig. 4N is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4N is a space-floating image display device that employs the optical system of Fig. 2D. As with the examples of space-floating image display devices that employ the optical systems of Figs. 2A to 2C, an image is formed in the air as a space-floating image 3 by image light that has passed through a transparent member 100. Furthermore, using sensing light from an aerial operation detection sensor 1351 that is located on the back side of the transparent member 100 as seen from the user, it is possible to detect operation of the space-floating image 3 by the user's finger 9004.

[0135] 2A to 2C, and in the example of the space-floating image display device employing the optical system of Fig. 2D, the space-floating image 3 is formed in front of the transparent member 100, and the operation of the space-floating image 3 by the user's finger can be detected using the sensing light of the mid-air operation detection sensor 1351 arranged on the back side of the transparent member 100 as seen from the user. Therefore, the space-floating image display device employing the optical system of Fig. 2D has a different optical system from the space-floating image display device in which the optical system of Fig. 2A to 2C is arranged on the back side of the transparent member 100 as seen from the user.

[0136] However, from the user's perspective, the usability of the space-floating image display device employing the optical system of FIG. 2D is almost the same as that of the space-floating image display device employing the optical system of FIGS. 2A to 2C.

[0137] Next, Fig. 4O is a diagram showing an example of the configuration of a space-floating image display device. Fig. 4O is a diagram showing the configuration of the internal optical system of the space-floating image display device 1000 of Fig. 4N. The space-floating image display device 1000 shown in Fig. 4O is equipped with an optical system corresponding to the optical system of Fig. 2D. The space-floating image display device 1000 shown in Fig. 4O is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward.

[0138] 4O, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0139] Here, the configuration of Fig. 4O will be compared with the configuration of Fig. 4A to confirm the differences. In Fig. 4A, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the polarization separation member 101. In contrast, in Fig. 4O, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the retroreflector 5. Furthermore, the configuration of Fig. 4A includes the retroreflector 2 and the λ / 4 plate 21, but these are not present in Fig. 4O. Furthermore, while the presence of the absorbing polarizer 12 is more preferable in Fig. 4A, the absorbing polarizer 12 is not particularly necessary in Fig. 4O.

[0140] That is, to replace the optical system of FIG. 2A in the configuration of FIG. 4A with the optical system of FIG. 2D and to replace it with the configuration of FIG. 4O, the following can be done. That is, the polarization separation member 101 in the configuration of FIG. 4A is replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 plate 21 are removed from the configuration of FIG. 4A. The absorptive polarizer 12 is optional. By performing a replacement based on this idea, the optical system of FIGS. 2A to 2C mounted in the configuration of the space-floating image display device of FIGS. 4A to 4G can be replaced with the optical system of FIG. 2D, and the space-floating image display device can be replaced with the optical system of FIG. 2D. In this case, the polarization separation member 101 in FIGS. 4A and 4B is replaced with the retroreflector 5, and the polarization separation member 101B in FIGS. 4C to 4G is replaced with the retroreflector 5.

[0141] In this way, it is possible to realize a space-floating image display device in which the optical system in the configuration of the space-floating image display device of Figures 4A to 4G is replaced with the optical system of Figure 2D. Even in these space-floating image display devices in which the optical system of Figure 2D is replaced, it is possible to realize a space-floating image display device that is almost as easy to use as the space-floating image display device of Figures 4A to 4G.

[0142] <Display Device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes the light source of the image display element 11. Fig. 5 shows the light source device 13 together with the liquid crystal display panel as an exploded perspective view.

[0143] As shown by arrow 30 in Figure 5, this liquid crystal display panel (image display element 11) receives an illumination light beam from light source device 13, which is a backlight device, that has narrow-angle diffusion characteristics, i.e., has strong directivity (linearity) and characteristics similar to laser light with a polarization plane aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam in accordance with an input video signal. The modulated image light is reflected by retroreflector 2 and passes through transparent member 100 to form a real image, a floating image in space (see Figure 1).

[0144] 5 also shows a configuration including a liquid crystal display panel 11 constituting the display device 1, a light redirection panel 54 for controlling the directional characteristics of the light beam emitted from the light source device 13, and, if necessary, a narrow-angle diffuser (not shown). Specifically, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarization is emitted with its intensity modulated by a video signal (see arrow 30 in FIG. 5). This allows the desired image to be projected as highly directional (linear) light of a specific polarization via the light redirection panel 54 toward the retroreflector 2, where it is reflected by the retroreflector 2 and transmitted to the eyes of a monitor outside the store (space), forming a floating image 3. A protective cover 50 (see FIGS. 6 and 7) may be provided on the surface of the light redirection panel 54.

[0145] <Display Device Example 1> FIG. 6 shows an example of a specific configuration of the display device 1. In FIG. 6, a liquid crystal display panel 11 and a light direction conversion panel 54 are disposed on the light source device 13 shown in FIG. 5. The light source device 13 is configured on a case shown in FIG. 5, formed of, for example, plastic, and contains LED elements 201 and a light guide 203 therein. As shown in FIG. 5 and other figures, the end surface of the light guide 203 has a lens-like shape whose cross-sectional area gradually increases toward the light receiving section in order to convert the divergent light from each LED element 201 into a substantially parallel beam. The lens-like shape gradually reduces the divergence angle by multiple total reflections during propagation within the light guide 203. The liquid crystal display panel 11 constituting the display device 1 is attached to the top surface of the display device 1. Furthermore, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202 on which their control circuits are mounted are attached to one side of the case of the light source device 13 (the left end surface in this example). A heat sink, which is a member for cooling the heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED board 202 .

[0146] Furthermore, a frame (not shown) for the liquid crystal display panel is attached to the top surface of the case of the light source device 13. The frame includes the liquid crystal display panel 11, which is attached to the frame, and a flexible printed circuit (FPC) (not shown) electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is an image display element, generates a display image by modulating the intensity of transmitted light in conjunction with the LED elements 201, which are solid-state light sources, based on a control signal from a control circuit (image control unit 1160 in FIG. 3 ) constituting the electronic device. The generated image light has a narrow diffusion angle and contains only specific polarization components, resulting in a novel image display device similar to a surface-emitting laser image source driven by a video signal. Currently, it is technically and safety-wise impossible to obtain a laser beam of the same size as the image obtained by the display device 1 described above using a laser device. Therefore, in this embodiment, light similar to the surface-emitting laser image light described above is obtained from a beam of light from a general light source, for example, an LED element.

[0147] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 7 as well as FIG.

[0148] 6 and 7 are cross-sectional views, and only one of the LED elements 201 constituting the light source is shown, and this light is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion on the end surface of the light guide and the LED element are attached while maintaining a predetermined positional relationship.

[0149] Each light guide 203 is formed of a translucent resin such as acrylic. The LED light-receiving surface at the end of the light guide 203 has a cone-shaped outer periphery obtained by rotating a parabolic cross section, and the top of the light guide 203 has a concave portion with a convex portion (i.e., a convex lens surface) formed in the center, and the center of the flat portion has a convex lens surface (or a concave lens surface) that protrudes outward (not shown). The outer shape of the light-receiving portion of the light guide to which the LED element 201 is attached is a parabolic shape that forms a cone-shaped outer periphery, and is set within an angle range that allows total reflection of the light emitted from the LED element toward the periphery within the parabolic shape, or a reflective surface is formed.

[0150] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed to the LED collimator (light-receiving end surface 203 a) so that the LED elements 201 on the surface are positioned in the center of the recessed portion described above.

[0151] With this configuration, the shape of the light-receiving end surface 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as approximately parallel light, thereby improving the utilization efficiency of the generated light.

[0152] As described above, the light source device 13 is configured by attaching a light source unit having a plurality of LED elements 201 arranged as light sources to the light-receiving end surface 203a, which is a light-receiving section provided on the end surface of the light guide 203, and the divergent light beams from the LED elements 201 are converted into approximately parallel light by the lens shape of the light-receiving end surface 203a of the light guide end surface, which is then guided inside the light guide 203 as shown by the arrow (in a direction parallel to the drawing), and emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11, which is disposed approximately parallel to the light guide 203 (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of the light beam direction conversion means 204 depending on the shape inside or on the surface of the light guide.

[0153] The light beam direction conversion means 204 described above emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 (in a direction perpendicular to the front of the drawing) that is disposed substantially parallel to the light guide 203, by changing the shape of the surface of the light guide or by providing a portion with a different refractive index inside the light guide. In this case, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is positioned at the same position as the diagonal dimension of the screen, if the relative brightness ratio between the center of the screen and the periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

[0154] 6 is a cross-sectional layout diagram illustrating the configuration and operation of the light source of this embodiment that performs polarization conversion in light source device 13 including the above-mentioned light guide 203 and LED element 201. In Fig. 6, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface thereof is attached liquid crystal display panel 11 that has polarizing plates on the light source light entrance surface and the image light exit surface.

[0155] In addition, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., P wave) 212 of the natural light beam 210 emitted from the LED element 201. The reflected light is reflected again by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and directed toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice to convert the reflected light beam from P polarized light to S polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam, the light intensity of which is modulated by a video signal in the liquid crystal display panel 11 (arrow 213 in Figure 6), enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0156] 7 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in a light source device 13 including a light guide 203 and an LED element 201, similar to Fig. 6. The light source device 13 is similarly configured with a light guide 203 formed of, for example, plastic, on the surface of which or inside which a light beam direction conversion means 204 is provided, an LED element 201 as a light source, a reflective sheet 205, a retardation plate 206, a lenticular lens, etc. A liquid crystal display panel 11 is attached to the top surface of the light source device 13 as an image display element, and the panel has polarizing plates on the light source light entrance surface and the image light exit surface.

[0157] A film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., S wave) 211 of the natural light beam 210 emitted from the LED element 201. That is, in the example of FIG. 7 , the selective reflection characteristics of the reflective polarizing plate 49 differ from those in FIG. 7 . The reflected light is reflected by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and returns to the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting the reflected light beam from S-polarized light to P-polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam intensity-modulated by the image signal in the liquid crystal display panel 11 (arrow 214 in FIG. 7 ) enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0158] In the light source devices shown in Figures 6 and 7, in addition to the function of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, the reflective polarizer reflects the polarized light component on one side. Therefore, the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizer multiplied by the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In fact, experiments have confirmed that the contrast performance of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those of self-luminous organic EL displays are obtained.

[0159] <Display Device Example 2> Figure 8 shows another example of the specific configuration of the display device 1. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, etc., in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. Also, an LED board on which LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and their control circuits are mounted are attached to one side of the case of the light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer surface of the LED board.

[0160] The liquid crystal display panel frame attached to the top surface of the case is configured to have attached thereto the liquid crystal display panel 11 attached to the frame, and further to have attached thereto an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements 14a and 14b, which are solid-state light sources, based on a control signal from a control circuit (not shown here) that constitutes the electronic device.

[0161] <Display Device Example 3> Next, another example of the specific configuration of the display device 1 (Display Device Example 3) will be described with reference to Fig. 9 . The light source device of this display device 1 converts a divergent beam of light (a mixture of P-polarized and S-polarized light) from an LED into a substantially parallel beam by a collimator 18, and reflects the parallel beam toward the liquid crystal display panel 11 by the reflective surface of a reflective light guide 304. The reflected light is incident on a reflective polarizer 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizer 49 transmits light of a specific polarization (e.g., P-polarized light) and causes the transmitted polarized light to be incident on the liquid crystal display panel 11. Here, light polarized other than the specific polarization (e.g., S-polarized light) is reflected by the reflective polarizer 49 and directed again toward the reflective light guide 304.

[0162] The reflective polarizing plate 49 is installed at an angle with respect to the liquid crystal display panel 11 so that the reflective polarizing plate 49 is not perpendicular to the chief ray of light from the reflective surface of the reflective light guide 304. The chief ray of light reflected by the reflective polarizing plate 49 is incident on the transmission surface of the reflective light guide 304. The light that has entered the transmission surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 which is a retardation plate, and is reflected by the reflector 271. The light reflected by the reflector 271 passes through the λ / 4 plate 270 again, and passes through the transmission surface of the reflective light guide 304. The light that has passed through the transmission surface of the reflective light guide 304 is incident on the reflective polarizing plate 49 again.

[0163] At this time, the light that re-enters the reflective polarizing plate 49 has passed through the λ / 4 plate 270 twice, and therefore its polarization has been converted to a polarization (for example, P-polarized light) that is transmitted through the reflective polarizing plate 49. Therefore, the light whose polarization has been converted passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11. Note that, with regard to the polarization design related to the polarization conversion, the polarization may be configured in reverse from the above explanation (S-polarized light and P-polarized light may be reversed).

[0164] As a result, the light from the LED is aligned to a specific polarization (for example, P polarization), enters the liquid crystal display panel 11, and is brightness-modulated in accordance with the video signal to display an image on the panel surface. As in the above example, multiple LEDs that make up the light source are shown (however, because this is a vertical cross section, only one is shown in Figure 9), and these are attached at predetermined positions relative to the collimator 18.

[0165] Each collimator 18 is formed of a translucent resin such as acrylic or glass. The collimator 18 may have a cone-shaped outer circumferential surface obtained by rotating a parabolic cross section. The collimator 18 may have a concave portion with a convex portion (i.e., a convex lens surface) formed in the center of the apex (the side facing the LED substrate 102). The collimator 18 may have a convex lens surface (or a concave lens surface) protruding outward in the center of the flat surface (the side opposite the apex). The parabolic surface forming the cone-shaped outer circumferential surface of the collimator 18 is set within an angle range that allows total reflection of the light emitted from the LED toward the periphery, or a reflective surface is formed therein.

[0166] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED board 102. The LED board 102 is arranged and fixed to the collimator 18 so that the LEDs on the surface are positioned at the center of the apex of the convex cone shape (or in the recess if the apex has a recess).

[0167] With this configuration, the collimator 18 focuses light emitted from the LED, particularly light emitted from the central portion, into parallel light by the convex lens surface that forms the outer shape of the collimator 18. Light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the conical shape of the collimator 18, and is similarly focused into parallel light. In other words, the collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, makes it possible to extract almost all of the light generated by the LED as parallel light, thereby improving the utilization efficiency of the generated light.

[0168] Furthermore, the light converted into approximately parallel light by the collimator 18 shown in FIG. 9 is reflected by the reflective light guide 304. Of this light, light of a specific polarization passes through the reflective polarizer 49 due to the action of the reflective polarizer 49, while light of the other polarization reflected by the action of the reflective polarizer 49 passes through the light guide 304 again. The light is reflected by the reflector 271 located opposite the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarized and converted by passing twice through the λ / 4 plate 270, which is a retardation plate. The light reflected by the reflector 271 passes through the light guide 304 again and enters the reflective polarizer 49 provided on the opposite surface. Since the incident light has been polarized and converted, it passes through the reflective polarizer 49 and enters the liquid crystal display panel 11 with its polarization direction aligned. As a result, all of the light from the light source can be used, doubling the geometrical optical utilization efficiency of light. Furthermore, since the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, the use of the light source device of this embodiment significantly improves the contrast ratio of the entire display device. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 allows adjustment of the angle of light reflection and diffusion at each reflective surface. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 for each design can improve the uniformity of the light incident on the liquid crystal display panel 11.

[0169] It should be noted that the λ / 4 plate 270, which is the retardation plate in Fig. 9, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of Fig. 9, any retardation plate may be used as long as the phase changes by 90° (λ / 2) when polarized light passes through it twice. The thickness of the retardation plate may be adjusted according to the incident angle distribution of the polarized light.

[0170] <Display Device Example 4> Another example (Display Device Example 4) of the configuration of an optical system such as a light source device of a display device will be described with reference to Fig. 10. This is a configuration example in which a diffusion sheet is used instead of the reflective light guide 304 in the light source device of Display Device Example 3. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) that convert diffusion characteristics in the vertical and horizontal directions of the drawing (front-to-back directions not shown in the drawing) are used on the light emission side of collimator 18, and light from collimator 18 is made to enter between the two optical sheets (diffusion sheets).

[0171] The optical sheet may be a single sheet instead of a two-sheet configuration. In a single-sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes on the front and back surfaces of the single optical sheet. Alternatively, multiple diffusion sheets may be used to share the functions. In the example of FIG. 10 , the reflection and diffusion characteristics due to the front and back shapes of optical sheets 207A and 207B can be optimally designed using the number of LEDs, the divergence angle from LED substrate (optical element) 102, and the optical specifications of collimator 18 as design parameters so that the surface density of the light beam emitted from the liquid crystal display panel 11 is uniform. In other words, the diffusion characteristics are adjusted by the surface shapes of multiple diffusion sheets instead of light guides.

[0172] In the example of FIG. 10 , polarization conversion is performed in the same manner as in the display device example 3 described above. That is, in the example of FIG. 10 , the reflective polarizing plate 49 may be configured to have the property of reflecting S-polarized light (transmitting P-polarized light). In this case, the reflective polarizing plate 49 transmits P-polarized light from the light source LED, and the transmitted light enters the liquid crystal display panel 11. The reflective polarizing plate 49 reflects S-polarized light from the light source LED, and the reflected light passes through the retardation plate 270 shown in FIG. 10 . The light that passes through the retardation plate 270 is reflected by the reflector 271. The light reflected by the reflector 271 passes through the retardation plate 270 again and is converted to P-polarized light. The polarization-converted light passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11.

[0173] It should be noted that the λ / 4 plate 270, which is the retarder in FIG. 10, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of FIG. 10, any retarder that changes the phase by 90° (λ / 2) when polarized light passes through it twice may be used. The thickness of the retarder may be adjusted according to the distribution of incident angles of the polarized light. It should be noted that, in FIG. 10 as well, the polarization design for polarization conversion may be configured in reverse (reversing the S-polarized light and P-polarized light) from the above explanation.

[0174] In a typical TV device, the light emitted from the liquid crystal display panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (shown on the X-axis in FIG. 12(a)) and the vertical direction of the screen (shown on the Y-axis in FIG. 12(b)). In contrast, the diffusion characteristics of the light beam emitted from the liquid crystal display panel of this embodiment are, for example, as shown in Example 1 of FIG. 12, when the viewing angle at which the luminance is 50% of that at a front view (angle of 0 degrees) is set to 13 degrees, this is 1 / 5 of the 62 degrees of a typical TV device. Similarly, the vertical viewing angle is set asymmetrically between the top and bottom, and the reflection angle and the area of ​​the reflective surface of the reflective light guide are optimized to keep the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness being more than 50 times higher.

[0175] Furthermore, assuming the viewing angle characteristics shown in Example 2 of Figure 12, if the viewing angle at which brightness is 50% of that at a front view (angle of 0 degrees) is set to 5 degrees, this is 1 / 12 of the 62 degrees of devices used for general TV applications. Similarly, the vertical viewing angle is set equal at the top and bottom, and the reflection angle and area of ​​the reflective surface of the reflective light guide are optimized to keep the viewing angle to about 1 / 12 of that of devices used for general TV applications. As a result, the amount of image light directed in the monitoring direction is significantly improved compared to conventional LCD TVs, and brightness is more than 100 times higher.

[0176] As described above, by setting the viewing angle to a narrow angle, the amount of luminous flux directed in the monitoring direction can be concentrated, significantly improving the efficiency of light utilization. As a result, even when using a liquid crystal display panel for general TV applications, by controlling the light diffusion characteristics of the light source device, it is possible to achieve a significant improvement in brightness with similar power consumption, making it possible to create a video display device that is compatible with information display systems aimed at bright outdoor areas.

[0177] When using a large LCD panel, the brightness of the screen can be improved by directing the light from the periphery of the screen inward so that it is directed toward the observer when the observer is facing the center of the screen. Figure 11 shows the convergence angle between the long and short sides of the panel when the observer's distance from the panel, L, and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in portrait orientation, the convergence angle can be set to match the short side. For example, when using a 22-inch panel in portrait orientation and the monitoring distance is 0.8 m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the observer.

[0178] Similarly, when monitoring with a 15-inch panel in portrait orientation and a monitoring distance of 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the monitor. As described above, depending on the size of the liquid crystal display panel and whether it is used portrait or landscape, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the monitor who is in the optimum position to monitor the centre of the screen.

[0179] As shown in Figure 9, the basic configuration involves a light source device directing a light beam with a narrow angle of directionality to a liquid crystal display panel 11, which is then luminance-modulated according to a video signal. The video information displayed on the screen of the liquid crystal display panel 11 is then reflected by a retroreflector, and the resulting spatially floating image is displayed indoors or outdoors via a transparent member 100.

[0180] By using the display device and light source device according to the embodiment of the present invention described above, it is possible to realize a space floating image display device with higher light utilization efficiency.

[0181] <Example of Image Display Processing in Space-Floating Image Display Device> Next, an example of a problem solved by the image processing of this embodiment will be described with reference to Fig. 13A. In the space-floating image display device 1000, when the far side of the space-floating image 3 as seen from the user is inside the housing of the space-floating image display device 1000 and is sufficiently dark, the user will visually recognize that the background of the space-floating image 3 is black.

[0182] Here, an example of displaying a character "panda" 1525 in a floating-in-space image 3 will be described with reference to Fig. 13A. First, for an image including a pixel region for drawing the image of the character "panda" 1525 and a transparent information region 1520 that is a background image, as shown in Fig. 13A(1), the image control unit 1160 in Fig. 3 distinguishes and recognizes the pixel region for drawing the image of the character "panda" 1525 from the transparent information region 1520 that is a background image.

[0183] A method for distinguishing and recognizing the character image from the background image is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the character image layer in front of the background image layer can be processed as separate layers, and the character image and background image can be distinguished and recognized based on the superimposition relationship when these layers are combined.

[0184] Here, the image control unit 1160 recognizes black pixels that depict objects such as character images and transparent information pixels as different information. However, it is assumed that both the black pixels that depict objects and the transparent information pixels have a luminance of 0. In this case, when the Space Floating Image 3 is displayed, there is no difference in luminance between the pixels that depict black in the image of the character "panda" 1525 and the pixels of the transparent information region 1520, which is the background image. Therefore, in the Space Floating Image 3, as shown in FIG. 13A (2), neither the pixels that depict black in the image of the character "panda" 1525 nor the pixels of the transparent information region 1520 have luminance, and they are visually perceived by the user as the same optically black space. In other words, the black portions of the image of the character "panda" 1525, which is an object, blend into the background, and only the non-black portions of the character "panda" 1525 are perceived as floating in the display region of the Space Floating Image 3.

[0185] An example of image processing according to this embodiment will be described with reference to FIG. 13B. FIG. 13B is a diagram illustrating an example of image processing that more effectively resolves the issue of the black image region of the object blending into the background, as described in FIG. 13A. In FIGS. 13B(1) and 13B(2), the upper side shows the display state of the floating-in-space image 3, and the lower side shows the input / output characteristics of the image processing of the image of the object. Note that the image of the object (character "panda" 1525) and the corresponding data may be read from the storage unit 1170 or memory 1109 in FIG. 3, or may be input from the video signal input unit 1131, or may be acquired via the communication unit 1132.

[0186] In the state shown in Figure 13B(1), the input / output characteristics of the image processing of the object image are in a linear state with no particular adjustment. In this case, the display state is the same as in Figure 13A(2), and the black image area of ​​the object blends into the background. In contrast, in Figure 13B(2), the video control unit 1160 of this embodiment adjusts the input / output characteristics of the image processing of the image of the object (character "panda" 1525) to the input / output characteristics shown in the lower row.

[0187] That is, the video control unit 1160 performs image processing with input / output characteristics on the image of the object (character "panda" 1525), which has a characteristic of converting pixels in low-brightness areas of the input image into output pixels with increased brightness values. After the image of the object (character "panda" 1525) has been subjected to the image processing with input / output characteristics, a video including the image of the object (character "panda" 1525) is input to the display device 1 and displayed. Then, as shown in the upper part of FIG. 13B (2), the display state of the floating in space image 3 is such that the brightness of pixel areas depicting black in the image of the character "panda" 1525 increases. This allows the user to distinguish the areas depicting black among the areas depicting the image of the character "panda" 1525 without them blending into the black background, making it possible to display the object more appropriately.

[0188] 13B(2), the area displaying the image of the object character "panda" 1525 can be distinguished from the black background inside the housing of the space-floating image display device 1000 through the window, improving the visibility of the object. Therefore, for example, even if the object includes pixels with a brightness value of 0 among the pixels constituting the object before the image processing (i.e., at the time when the image of the object and the corresponding data are read from the storage unit 1170 or memory 1109 in FIG. 3, or at the time when the image of the object is input from the video signal input unit 1131, or at the time when the data of the object is acquired via the communication unit 1132, etc.), the image processing of the input / output characteristics by the video control unit 1160 converts the object into an object with increased brightness values ​​of pixels in low-brightness areas, and then the object is displayed on the display device 1 and converted into a space-floating image 3 by the optical system of the space-floating image display device 1000.

[0189] That is, the pixels that make up the object after image processing of the input / output characteristics are converted to a state in which they do not include pixels with a brightness value of 0, and then they are displayed on the display device 1 and converted into a floating image 3 by the optical system of the floating image display device 1000.

[0190] In the image processing of Figure 13B(2), a method for applying image processing with the input / output characteristics of Figure 13B(2) only to the image area of ​​the object (character "panda" 1525) is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the layer of the character image in front of the background image layer can be processed as separate layers, and the image processing with the input / output characteristics of Figure 13B(2) is applied to the character image layer while not applying this image processing to the background image layer.

[0191] Then, by combining these layers, image processing with a characteristic of raising the low-brightness areas of the input image is performed only on the character image, as shown in Fig. 13B (2). Alternatively, after the character image layer and background image layer are combined, image processing with the input / output characteristics of Fig. 13B (2) may be performed only on the character image area.

[0192] Furthermore, the input / output image characteristics used in the image processing for enhancing low-luminance regions of the input / output characteristics for the input image are not limited to the example shown in FIG. 13B(2). Any image processing for enhancing low luminance may be used, including so-called brightness adjustment. Alternatively, image processing for improving visibility by controlling the gain that changes the weighting of Retinex processing, as disclosed in International Publication WO 2014 / 162533, may be performed.

[0193] According to the image processing of FIG. 13B(2) described above, it is possible to make the user aware of areas where black is drawn among areas where images of characters, objects, etc. are drawn without blending into the black background, thereby realizing a more suitable display.

[0194] 13A and 13B, the problems and more suitable image processing were explained using a space-floating image display device in which the background appears black (for example, the space-floating image display device 1000 in FIGS. 4A to 4G, or the space-floating image display device 1000 in a state where the rear side window is shaded in FIGS. 4I and 4J). However, this image processing is also effective for devices other than these space-floating image display devices.

[0195] Specifically, in the space-floating image display device 1000 of Fig. 4H and the space-floating image display device 1000 in Fig. 4I and Fig. 4J where the rear window is not shaded, the background of the space-floating image 3 is not black, but the scenery behind the space-floating image display device 1000 through the window. In this case, the problems explained in Fig. 13A and Fig. 13B also exist.

[0196] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the scenery behind the space-floating image display device 1000 through the window. In this case, by using the image processing of Fig. 13B (2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognized as being distinct from the scenery behind the space-floating image display device 1000 through the window, improving the visibility of the object.

[0197] That is, by using the image processing of FIG. 13B(2), the area displaying the image of the character "panda" 1525, which is an object, can be recognized as distinct from the scenery behind the space floating image display device 1000 through the window, and it can be more easily recognized that the character "panda" 1525, which is the object, is in front of the scenery, improving the visibility of the object.

[0198] 4K, 4L, and 4M, as described above, when another image (such as an image from the transmissive self-luminous image display device 1650 or an image from the second display device 1680) is displayed at a position different in depth from the space-floating image 3, the background of the space-floating image 3 is not black, but the other image. In this case, the problems described in FIGS. 13A and 13B also exist.

[0199] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the other image that is displayed at a different depth from the floating image in space 3. In this case as well, by using the image processing of Fig. 13B(2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognizable as being distinct from the other image, improving the visibility of the object.

[0200] In other words, by using the image processing of Figure 13B (2), the area displaying the image of the object character "panda" 1525 can be recognized as distinct from the other image, and it can be more easily recognized that the object character "panda" 1525 is in front of the other image, improving the visibility of the object.

[0201] An example of the image display process of this embodiment will be described with reference to Fig. 13C. Fig. 13C shows an example of the image display process of this embodiment, in which the space floating image 3 and a second image 2050, which is another image, are simultaneously displayed. The second image 2050 may correspond to the image displayed by the transmissive self-luminous image display device 1650 of Fig. 4K or 4L. The second image 2050 may also correspond to the image displayed by the second display device 1680 of Fig. 4M.

[0202] That is, the example of the image display in Fig. 13C shows a specific example of the image display examples of the space-floating image display device 1000 in Figs. 4K, 4L, and 4M. In the example of this figure, a bear character is displayed in space-floating image 3. Areas other than the bear character in space-floating image 3 are displayed in black, and become transparent as a space-floating image. In addition, the second image 2050 is a background image in which a plain, a mountain, and a sun are drawn.

[0203] 13C , the floating-in-space image 3 and the second image 2050 are displayed at different depth positions. When the user 230 views the two images, the floating-in-space image 3 and the second image 2050, in the line of sight of the arrow 2040, the user 230 can view the two images superimposed on each other. Specifically, the bear character of the floating-in-space image 3 appears superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0204] Here, since the space floating image 3 is formed as a real image in the air, when the user 230 moves their viewpoint slightly, they can recognize the depth of the space floating image 3 and the second image 2050 due to parallax. Therefore, the user 230 can get a stronger sense of floating in space from the space floating image 3 while viewing the two images in an overlapping state.

[0205] An example of the image display process of this embodiment will be described with reference to Fig. 13D. Fig. 13D(1) is a diagram of the floating in space image 3, from the example of image display of this embodiment in Fig. 13C, as seen from the line of sight of the user 230. Here, a bear character is displayed in the floating in space image 3. The area other than the bear character in the floating in space image 3 is displayed in black, and the floating in space image is transparent.

[0206] 13D(2) is a diagram showing the second image 2050 in the example of the video display of this embodiment shown in FIG. 13C as viewed from the line of sight of the user 230. In the example shown in this figure, the second image 2050 is a background image in which a plain, a mountain, and a sun are depicted.

[0207] 13D(3) is a diagram showing the state in which the second image 2050 and the floating in space image 3 appear superimposed in the line of sight of the user 230, among the example of image display of this embodiment in Fig. 13C. Specifically, the bear character of the floating in space image 3 appears superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0208] Here, when the space-floating image 3 and the second image 2050 are displayed simultaneously, it is desirable to pay attention to the balance of brightness between the two images in order to more favorably ensure the visibility of the space-floating image 3. If the second image 2050 is too bright compared to the brightness of the space-floating image 3, the displayed image of the space-floating image 3 will be transparent, and the second image 2050, which is the background, will be strongly visible through it.

[0209] Therefore, the output of the light source of the spatially floating image 3 and the display image brightness of the display device 1, and the output of the light source of the display device displaying the second image 2050 and the display image brightness of the display device should be set so that at least the brightness per unit area of ​​the spatially floating image 3 at the display position of the spatially floating image 3 is greater than the brightness per unit area of ​​the image light that reaches the display position of the spatially floating image 3 from the second image 2050.

[0210] Note that, since this condition only needs to be satisfied when the space-floating image 3 and the second image 2050 are displayed simultaneously, when switching from the first display mode in which the space-floating image 3 is not displayed and only the second image 2050 is displayed to the second display mode in which the space-floating image 3 and the second image 2050 are displayed simultaneously, control may be performed to reduce the brightness of the second image 2050 by lowering the output of the light source of the display device that displays the second image 2050 and / or the display image brightness of the display device. These controls may be realized by the control unit 1110 in Fig. 3 controlling the display device 1 and the display device that displays the second image 2050 (the transmissive self-luminous image display device 1650 in Fig. 4K or 4L or the second display device 1680 in Fig. 4M).

[0211] Note that when switching from the above-described first display mode to the above-described second display mode, if control is performed to reduce the brightness of second image 2050, the brightness may be reduced uniformly across the entire screen of second image 2050. Alternatively, instead of reducing the brightness uniformly across the entire screen of second image 2050, the brightness reduction effect may be greatest in the portion where an object is displayed in space-floating image 3, and the brightness reduction effect may be gradually weakened around that portion. In other words, the visibility of space-floating image 3 can be sufficiently ensured by reducing the brightness of second image 2050 only in the portion where space-floating image 3 is visually recognized as being superimposed on second image 2050.

[0212] Here, since the space floating image 3 and the second image 2050 are displayed at positions with different depths, when the user 230 slightly changes their viewpoint, the parallax causes a change in the superimposed position of the space floating image 3 relative to the second image 2050. Therefore, when switching from the above-mentioned first display mode to the above-mentioned second display mode, if the brightness is to be reduced unevenly across the entire screen of the second image 2050, it is not desirable to reduce the brightness sharply based on the outline of the object displayed in the space floating image 3, and it is desirable to perform gradation processing of the brightness reduction effect, which changes the brightness reduction effect in stages depending on the position as described above.

[0213] In addition, in the space floating image display device 1000 where the position of the object displayed in the space floating image 3 is approximately at the center of the space floating image 3, the position where the brightness reduction effect of the gradation processing of the brightness reduction effect is greatest can be set to the center position of the space floating image 3.

[0214] According to the image display process of this embodiment described above, the user 230 can view the space floating image 3 and the second image 2050 more favorably.

[0215] Note that when displaying the space-floating image 3, control may be performed so as not to display the second image 2050. Since not displaying the second image 2050 increases the visibility of the space-floating image 3, this is suitable for applications such as the space-floating image display device 1000 in which the user must be able to reliably view the space-floating image 3 when it is displayed.

[0216] <Example 2> As Example 2 of the present invention, an example of another configuration example of a space-floating image display device will be described. Note that the space-floating image display device according to this example is obtained by changing the optical system stored in the space-floating image display device described in Example 1 to the optical system shown in Figure 14 (1) or Figure 14 (2). In this example, differences from Example 1 will be described, and repeated explanations of the same configuration as Example 1 will be omitted. Note that in the following description of this example, the predetermined polarized light and the other polarized light are polarized waves whose phases differ by 90° from each other.

[0217] Fig. 14(1) shows an example of an optical system and optical path according to this embodiment. The optical system shown in Fig. 14(1) is configured such that the display device 1 is closer to the polarization separation member 101B in the optical system of Fig. 2C, making the entire optical system more compact. In Fig. 14(1), components that are assigned the same reference numerals as those in Fig. 2C will not be described in detail again.

[0218] 14(1), similar to Fig. 2C, image light of a predetermined polarized light (P-polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, similar to Fig. 2C, polarization separation member 101B selectively transmits the predetermined polarized light (P-polarized light in the figure) emitted from display device 1 and reflects the other polarized light (S-polarized light in the figure).

[0219] Therefore, image light of a predetermined polarization (P-polarized light in the figure) traveling in the vertical direction from the image display surface of the display device 1 passes through the polarization separation member 101B and reaches the retroreflector 2 to which the λ / 4 plate 21 is attached. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separation member 101B has passed through the λ / 4 plate 21 twice, and is converted from the predetermined polarization (P-polarized light in the figure) at the time of emission from the display device 1 to the other polarization (S-polarized light in the figure). The image light that travels again toward the polarization separation member 101B is the other polarization (S-polarized light in the figure), and is therefore reflected by the polarization separation member 101B toward the position where the user should be. The traveling direction of the image reflected by the polarization separation member 101B is determined based on the angle at which the polarization separation member 101B is disposed.

[0220] In the example of Figure 14 (1), the image light traveling toward the polarization separation member 101B is reflected at a right angle by the polarization separation member 101B and travels as shown in the figure. The image light reflected by the polarization separation member 101B forms a floating image 3A in space. The floating image 3A can be viewed by the user from the direction of arrow A.

[0221] Here, due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the position where the space floating image 3A is formed. This relationship determines the position where the space floating image 3A is formed in the traveling direction of the image light reflected by the polarization separating member 101B.

[0222] In the example of Figure 14(1), the display device 1, the polarization separating member 101B, and the retroreflector 2 are arranged closer than in the example of Figure 2C. This allows the entire optical system to be configured more compactly. However, the amount by which the space-floating image 3A protrudes from the optical system of Figure 14(1) is not very large. For example, as an indicator of the amount by which the space-floating image 3A protrudes from the optical system, the figure shows the distance from the position where the central light beam of the image light is reflected by the polarization separating member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)).

[0223] 14(1), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0224] Next, Fig. 14(2) shows another example of an optical system and an optical path according to this embodiment. The optical system of Fig. 14(2) is a modified version of the optical system of Fig. 14(1) in order to increase the amount of the floating image projecting from the optical system while realizing the same compactness as the optical system of Fig. 14(1). In Fig. 14(2), the components with the same reference numerals as Fig. 14(1) will not be described in detail again.

[0225] 14(2), as in FIG. 14(1), image light of a predetermined polarized light (P-polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, the polarization characteristics of polarization separation member 101B are arranged 90 degrees differently from those in FIG. 14(1). Image light of a predetermined polarized light (P-polarized light in the figure) traveling in a direction perpendicular to the image display surface of display device 1 passes through polarization separation member 101B.

[0226] 14(1), the image light passing through the polarization separation member 101B is not faced with the retroreflector 2 having the λ / 4 plate 21 attached thereto, but with the specular reflector 4 having the λ / 4 plate 21B attached thereto. Here, the reflection at the specular reflector 4 is specular reflection (also called regular reflection), not retroreflection.

[0227] Therefore, the image light that has passed through polarization separation member 101B is specularly reflected by specular reflector 4 to which λ / 4 plate 21B is attached. The image light that has been specularly reflected by specular reflector 4 and travels again toward polarization separation member 101B has been converted from the predetermined polarization (P-polarized in the figure) at the time of emission from display device 1 to the other polarization (S-polarized in the figure) by having passed through λ / 4 plate 21 twice. The image light that has traveled again toward polarization separation member 101B is the other polarization (S-polarized in the figure), and is therefore reflected by polarization separation member 101B.

[0228] Here, because the orientation of the polarization separation member 101B in Figure 14(2) is different from that in Figure 14(1), the image light reflected by the polarization separation member 101B travels in the opposite direction from the position where the user should be. A retroreflector 2 with a λ / 4 plate 21C attached is disposed at the destination of the image light reflected by the polarization separation member 101B. The image light is retroreflected by the retroreflector 2. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separation member 101B has been converted from the other polarized light (S-polarized light in the figure) to the specified polarized light (P-polarized light in the figure) by passing through the λ / 4 plate 21C twice.

[0229] The image light that travels back toward the polarization separation member 101B is of a predetermined polarization (P polarization in the figure), so it passes through the polarization separation member 101B and continues toward the location where the user should be. The image light that has passed through the polarization separation member 101B forms a floating image 3B in space. The floating image 3B can be easily viewed by the user from the direction of arrow A.

[0230] 14(2), as in the case of Fig. 14(1), due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the formation position of the space floating image 3B. This relationship determines the formation position of the space floating image 3B in the traveling direction of the image light transmitted through the polarization separation member 101B.

[0231] 14(2), the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is longer than the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 in FIG. 14(1). This is because in the optical system of FIG. 14(2), an optical path going back and forth between the polarization separation member 101B and the specular reflector 4, which does not exist in the optical system of FIG. 14(1), is added to the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2.

[0232] As a result, in the optical system of Figure 14(2), the distance from the position where the central ray of the image light passes through the polarization separation member 101B to the position where the image light forms the space-floating image 3B (L2 in the example of Figure 14(2)) is significantly longer than the distance from the position where the central ray of the image light is reflected by the polarization separation member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)) in the optical system of Figure 14(1).

[0233] 14(2), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0234] As described above, according to the optical systems of Fig. 14(1) and Fig. 14(2) in the second embodiment of the present invention, a more compact optical system can be realized. In particular, according to the optical system of Fig. 14(2), it is possible to increase the amount of the floating image projecting from the optical system, despite the more compact optical system.

[0235] In addition, when the optical system of Fig. 14(1) or Fig. 14(2) is incorporated into a space-floating image display device, it can be realized by replacing the optical system in the space-floating image display device described in Example 1 with the optical system of Fig. 14(1) or Fig. 14(2). Specifically, the optical system of Fig. 14(1) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4H, Fig. 4I, Fig. 4J, Fig. 4K, Fig. 4L, or Fig. 4M. In this case, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0236] Specifically, the optical system of Fig. 14(2) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4K, or Fig. 4L. In this case, it is possible to increase the amount of the space-floating image projecting from the optical system. Also, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0237] [Comparative Example] Fig. 15 shows a case where, in a comparative example of a floating-in-space image display device 1500, there is a misalignment between a display range 1501 of the floating-in-space image 3 and an aerial operation detection range 1502 by the aerial operation detection sensor 1351. In the comparative example of Fig. 15, in a horizontally placed floating-in-space image display device 1500, there is a display range 1501 of the floating-in-space image 3 at a predetermined position, and there is an aerial operation detection range 1502 by the aerial operation detection sensor 1351 at a predetermined position further back in the Y direction from that position. There is a misalignment 1503 in terms of spatial position between the display range 1501 and the aerial operation detection range 1502.

[0238] The comparative example has the following problems. In the space-floating image display device 1500, the space-floating image 3 can be clearly seen from a predetermined viewpoint position of the user 230, but the aerial operation detection range 1502 is invisible. Therefore, it is unclear to the user 230 where in space and what kind of aerial operation should be performed with respect to the space-floating image display device 1500. Even if the aerial operation detection range 1502 is taught to the user 230 in advance, it may be difficult to perform aerial operations within the aerial operation detection range 1502 because the aerial operation detection range 1502 is invisible. Therefore, in the comparative example, the aerial operation performed by the user 230 on the space-floating image 3 may fall outside the aerial operation detection range 1502. In this case, the user's aerial operation may not be detected at all, or may be difficult to detect with high accuracy. The same problem exists even when the aerial operation detection range 1502 is configured by another sensor, such as a camera of the imaging unit 1180.

[0239] As a means for solving the above problem, as in one embodiment of the floating-in-space image display device 1000 shown in Fig. 16 etc., it is sufficient to configure the display range 1601 of the floating-in-space image 3 to be formed within a range where the aerial operation by the user 230 can be suitably detected, for example, within the aerial operation detection range 1602 of the aerial operation detection sensor 1351. With this configuration, there is no deviation between the display range 1601 and the aerial operation detection range 1602.

[0240] As a result, the user 230 can perform aerial operations on the visible floating in space image 3 without knowing the mechanism for detecting aerial operations or the spatial position of the aerial operation detection range 1602. If the user 230 operates by touching the plane of the floating in space image 3 with their fingers, this can be detected as an aerial operation by the aerial operation detection range 1602 that is superimposed on the display range 1601 of the floating in space image 3. Therefore, with this configuration, the user's operation can be suitably detected, and the user 230 is less likely to be confused by the aerial operation. Furthermore, it is even better if the display content of the floating in space image 3 is an image that encourages aerial operations, as will be described later.

[0241] <Embodiment 3> As embodiment 3 of the present invention, an example of the space-floating image display device 1000 will be described. As the basic configuration of the space-floating image display device 1000 of embodiment 3, any of the space-floating image display devices 1000 shown in the drawings of embodiment 1 or embodiment 2 can be applied. In this embodiment 3, differences from embodiment 1 or embodiment 2 will be mainly described, and repeated explanations of similar configurations will be omitted.

[0242] [Outline of Example 3] The floating-in-the-air image display device of Example 3 is premised on a configuration in which the display range of the floating-in-the-air image 3 is formed within the mid-air operation detection range as shown in Fig. 16 . In addition, the floating-in-the-air image display device of Example 3 has two modes: a mode (sometimes referred to as 2D operation mode) that accepts two-dimensional (abbreviated as 2D) operations on the floating-in-the-air image 3, and a mode (sometimes referred to as 3D operation mode) that accepts three-dimensional (abbreviated as 3D) operations on the floating-in-the-air image 3, and is configured to be able to select and switch between these modes as appropriate. The 2D operation mode is a mode that accepts 2D operations that do not include operations in the depth direction on the floating-in-the-air image 3, in other words, planar operations. The 3D operation mode is a mode that accepts not only 2D operations but also 3D operations that include operations in the depth direction on the floating-in-the-air image 3.

[0243] The space-floating image display device 1000 of the third embodiment controls the display of the space-floating image 3 in the 2D operation mode, when displaying images such as GUI or content on the space-floating image 3, to display a message indicating that 2D operations are accepted, or that operations in the depth direction are not accepted, or to display a message encouraging 2D operations. In the 3D operation mode, the space-floating image display device 1000 controls the display of the space-floating image 3 in the 3D operation mode, when displaying images such as GUI or content on the space-floating image 3, to display a message indicating that 3D operations are accepted, or to display a message encouraging 3D operations.

[0244] In addition, the control and processing corresponding to the characteristic functions in the following Example 3 and the like can be performed, for example, by the control unit 1110, the image control unit 1160, or the aerial operation detection unit 1350 in Figure 3, and these may also be referred to as image processing units.

[0245] [3-1. System capable of accepting 2D operation and 3D operation] Example 3 provides a system capable of accepting both 2D operation and 3D operation as an operation by the user 230 on the floating-in-space image 3. The 2D operation is an operation of moving an operation object such as a finger in a planar direction along the plane of the floating-in-space image 3. The 3D operation is an operation of moving an operation object such as a finger in a three-dimensional space that includes a dimension of the front-to-back direction, which is the depth direction relative to the plane of the floating-in-space image 3. An operation in the depth direction may be referred to as a depth operation.

[0246] FIG. 17 shows a configuration example of a space-floating image display device 1000 of Example 3, and is a YZ cross-sectional view of a vertically-mounted space-floating image display device 1000 and a user 230 viewed from the side. In the configuration example of FIG. 17, the space-floating image display device 1000 includes components such as the display device 1, the polarization separation member 101B, and the retroreflective member 2 with the λ / 4 plate 21, located in the rear of the housing 1190. The image light reflected from the polarization separation member 101B travels forward in the Y direction, i.e., toward the user 230, passes through the transparent member 100, and forms a space-floating image 3, which is a real image, at a predetermined external position. This space-floating image 3 is formed to stand in the vertical Z direction, and has a two-dimensional X-Z plane as a display range 3R.

[0247] A lower portion 1190A and an upper portion 1190B of the housing 1190 protrude forward in the Y direction relative to the transparent member 100 of the housing 1190. A user operation detection mechanism 1701 is provided in the lower portion 1190A. The user operation detection mechanism 1701 is a mechanism that can detect aerial operations in at least the left-right and up-down directions, which are the X-Z directions of the floating in space image 3, in other words, the position and movement of fingers. The above-mentioned aerial operation detection sensor 1351 can be used as the user operation detection mechanism 1701, for example. The aerial operation detection sensor 1351, which is the user operation detection mechanism 1701, is installed corresponding to the position of the floating in space image 3 in the Y direction, and emits light, for example, upward in the Z direction, to sense the plane of the floating in space image 3. The optical axis of detection is indicated by a dashed arrow.

[0248] Furthermore, a user operation detection mechanism 1702 is provided on the upper portion 1190B. The user operation detection mechanism 1702 is a mechanism capable of detecting at least an aerial operation in the Y direction, which is the depth direction relative to the floating-in-space image 3 (in other words, the position and movement of the fingers). The user operation detection mechanism 1702 can be, for example, a camera as the imaging unit 1180 ( FIG. 3 ) described above. The imaging unit 1180, which is the user operation detection mechanism 1702, is installed in a position near the floating-in-space image 3 in the Y direction, for example, with the optical axis of imaging pointing downward in the Z direction, and the imaging range includes the display range 3R of the floating-in-space image 3. In the configuration example of FIG. 17 , the user operation detection mechanism 1702 is a mechanism capable of detecting the position and movement of the fingers in the Y direction as a depth operation.

[0249] 3D operations can be detected by combining the detection results of the user operation detection mechanism 1701 and the user operation detection mechanism 1702. Alternatively, the user operation detection mechanism 1701 may be a mechanism capable of detecting operations in the Y direction as well, i.e., a mechanism capable of detecting 3D operations by itself. Furthermore, the user operation detection mechanism 1702 may be a mechanism capable of detecting operations in the X direction and Z direction as well, i.e., a mechanism capable of detecting 3D operations by itself, by arranging it in a different position and orientation, or by using a different type of sensor. For example, as the user operation detection mechanism 1702, the imaging unit 1180 may be installed near the transparent member 100, as shown by imaging unit 1180b, and the optical axis of the image may be oriented in the Y direction toward the plane of the floating image 3, thereby enabling detection of operations in the X and Z directions.

[0250] The detection of the aerial operation in the depth direction is performed by a user operation detection mechanism 1701 and a user operation detection mechanism 1702 provided inside and outside the space floating image display device 1000. These user operation detection mechanisms are not limited to the aerial operation detection sensor 1351 and the camera which is the imaging unit 1180, but may also use a LiDAR (Light Detection And Ranging) or TOF (Time of Flight) distance measurement sensor. The details of the user operation detection mechanism are not limited.

[0251] In Example 3, a configuration may be provided that includes a user operation detection mechanism that can detect 3D operations on the floating-in-space image 3, i.e., the position and movement of fingers in each three-dimensional direction. This is not limited to the configuration example that combines the user operation detection mechanism 1701 and the user operation detection mechanism 1702. For example, a configuration that is integrated into one user operation detection mechanism, or a configuration that combines three or more sensors, etc., may be used. Multiple mid-air operation detection sensors 1351 may be provided at multiple positions in the depth direction. Furthermore, in various configuration examples that form the floating-in-space image 3, not limited to the portrait type as shown in FIG. 17, a UI and display control corresponding to 2D operations and 3D operations are similarly provided.

[0252] The user 230 uses, for example, a finger as an operation object to operate the floating-in-space image 3. Finger position 1721 shown by a solid line is an example of the position of the user's 230's finger when operating in the X-Z direction, which is the planar direction corresponding to the plane of the floating-in-space image 3. For example, one finger touches the plane of the floating-in-space image 3, in other words, is placed within that plane, and as an example of operation, the finger moves in the X direction or Z direction along that plane. Such an operation / movement may be referred to as a planar operation or a 2D operation.

[0253] Furthermore, finger position 1722 shown by a dashed line is an example of the position of the fingers of the user 230 when operating in the Y direction, which is the depth direction relative to the plane of the floating-in-space image 3. For example, one finger moves from the front side relative to the plane of the floating-in-space image 3, touching the plane and invading the depth side, and the other finger is placed in a position on the depth side. Such a movement / operation may be referred to as a depth operation. Furthermore, a three-dimensional operation formed by combining a 2D operation and a depth operation may be referred to as a 3D operation.

[0254] 16 and 17 , the aerial operation detection sensor 1351 is installed and implemented so as to have an aerial operation detection range that covers sensing of aerial operations on the entire plane of the floating in space image 3. This makes it possible to first cover detection of 2D operations. Furthermore, in FIG. 17 and the implementation example described below, for example, a user operation detection mechanism 1702 is installed and implemented so as to have an aerial operation detection range that covers sensing of three-dimensional space including the depth direction of the floating in space image 3.

[0255] 16 and 17, the floating-in-space image 3 is displayed within an aerial operation detection range, which is a range in which the user operation detection mechanism can preferably detect an aerial operation. In other words, in these configurations, the aerial operation detection range of the user operation detection mechanism, for example, the aerial operation detection sensor 1351, is designed to match the formation position of the plane of the floating-in-space image 3 (display range 3R in FIG. 17) and to cover the sensing of that plane.

[0256] [2D Operation and 3D Operation] Figure 18 summarizes the concepts of 2D operation and 3D operation. It can be said that the concept of 2D operation is included in the concept of 3D operation, which may be confusing, but please note that these are different concepts. The 2D operation mode described below is a mode that can detect 2D operation. The 3D operation mode described below is a mode that can detect 3D operation, but since 2D operation is included in the 3D operation, it can also be said to be a mode that can detect 2D operation. In other words, the 3D operation mode described below is a mode that can detect both planar operation and depth operation.

[0257] As an overview of the third embodiment, as shown in Fig. 17, the space-floating image display device 1000 is equipped with a user operation detection mechanism 1701 that can detect aerial operations in the X-Z directions, which are planar directions, as well as a user operation detection mechanism 1702 that can detect aerial operations in the Y direction, which is the depth direction. These user operation detection mechanisms allow the space-floating image display device 1000 to detect both 2D operations and 3D operations performed by the user 230 on the space-floating image 3. In other words, the space-floating image display device 1000 in Fig. 17 is capable of receiving aerial operations, namely, 2D operations and 3D operations, and executes display control processing according to the received aerial operations.

[0258] Furthermore, the space-floating image display device 1000 of the third embodiment provides a user interface (UI) or man-machine interface that corresponds to the reception of these 2D operations and 3D operations. This UI includes the provision of a graphical user interface (GUI) for the space-floating image 3. At the same time, the space-floating image display device 1000 of the third embodiment controls the behavior of the displayed space-floating image 3 to change according to the 2D operation or 3D operation of the user 230. Specifically, as will be described later, in the case of 2D operation, the space-floating image display device 1000 provides a GUI or content display suitable for the 2D operation or a display that encourages the 2D operation, and in the case of 3D operation, it provides a GUI or content display suitable for the 3D operation or a display that encourages the 3D operation.

[0259] [Aerial Operation Detection Sensor] As a supplementary note, the configuration of the aerial operation detection sensor 1351, which is the user operation detection mechanism 1701, can be, for example, the configuration shown in Fig. 19. Fig. 19 shows an implementation example of the aerial operation detection sensor 1351 that can be applied in Example 3 and the like. The aerial operation detection unit 1350, control unit 1110, or image control unit 1160 in Fig. 3, in other words the image processing unit, uses the detection signal of the aerial operation detection sensor 1351 to determine and detect aerial operations within the display range of the floating-in-space image 3.

[0260] FIG. 19 shows an xy plane view aligned with the display range 3R of the floating-in-space image 3. The aerial operation detection sensor 1351 has multiple optical elements 1351c arranged in the x direction. The optical elements 1351c are pairs of a light-emitting element 1351a and a light-receiving element 1351b. The light-emitting element 1351a is configured, for example, as an infrared element. The emission surface of the light-emitting element 1351a of the aerial operation detection sensor 1351 coincides with, for example, the top surface of the housing 1190. The light-emitting element 1351a emits light a1, for example, infrared light, in the y direction. If the light a1 is not blocked by an object, it passes through the display range 3R. If the light a1 is blocked by an object, it is reflected by the object and returns as reflected light a2. The reflected light a2 is received by the light-receiving element 1351b.

[0261] For example, if a contact point 1901 made by the user 230's finger is located within the x-y plane of the display range 3R, light a1 is reflected from the contact point 1901 and returns as reflected light a2. A light receiving element 1351b at a certain x-direction position detects the reflected light a2. This allows the aerial operation detection unit 1350 to determine that the contact point 1901 is located at that x-direction position. Furthermore, the aerial operation detection unit 1350 can calculate the distance using the TOF method from the time it takes for the emitted light a1 to return as reflected light a2. For example, it can calculate a distance 1902 to the contact point 1901. This also allows the position coordinates of the contact point 1901 on the x-y plane of the display range 3R to be determined.

[0262] 19, there is nothing above the floating in space image 3, but part of the housing 1190 may also be provided above the floating in space image 3. In that case, light a1 from the aerial operation detection sensor 1351 is reflected by that part of the housing 1190 and returns as reflected light a2. Not limited to this example, the aerial operation detection sensor 1351 may be placed above the xy plane of the display range 3R, or may be placed in the left / right direction, or may be placed at a position shifted in the front / back direction, i.e., the z direction. Multiple aerial operation detection sensors 1351 may be placed at multiple positions in the front / back direction.

[0263] [Modification of Aerial Operation Detection Sensor] FIG. 20 shows a modification of the aerial operation detection sensor 1351, which is the user operation detection mechanism 1701. This modification is based on the same configuration as FIG. 17 , and is configured such that multiple aerial operation detection sensors 1351, for example, three aerial operation detection sensors 1351, A1, A2, and A3, are provided in the depth direction, or Y direction, from the front. The configuration of FIG. 19 , for example, can be applied to each of the aerial operation detection sensors 1351. In this example, the aerial operation detection sensor 1351 is provided on the upper portion 1190B of the housing 1190 and emits light downward. The aerial operation detection sensor A2 is provided to correspond to the position of the display range 3R of the floating-in-space image 3 in the Y direction, so as to cover the detection of the display range 3R. The aerial operation detection sensor A1 is provided on the front side of the display range 3R in the Y direction, and can detect the position of the user's 230 fingers even when they are positioned in front of the display range 3R. The aerial operation detection sensor A3 is provided on the rear side of the display range 3R in the Y direction, and can detect the position of the fingers of the user 230 when they are positioned further back than the display range 3R. The camera serving as the imaging unit 1180 is provided facing forward in the Y direction so as to be able to capture images of the display range 3R and the user 230. Even with this configuration, 3D operations can be detected.

[0264] [Control of the state of the user operation detection mechanism] For example, in the configuration of Fig. 17, the two user operation detection mechanisms, the user operation detection mechanism 1701 and the user operation detection mechanism 1702, can each be switched between an enabled state (also referred to as an ON state) in which they can accept operations, and an disabled state (also referred to as an OFF state) in which they cannot accept operations. The space floating image display device 1000 controls the ON / OFF of these user operation detection functions. An example of switching control will be described later. Depending on the combination of the states of each user detection mechanism, detectable operations differ. In the third embodiment, operation modes corresponding to detectable operations are provided.

[0265] The disabled state (OFF state) of the aerial operation detection sensor 1351 is, for example, the following state. Roughly speaking, this disabled state means that there is no reaction to an aerial operation by the fingers of the user 230, in other words, an operation input. More specifically, this disabled state can be any of the following states, and any of them may be used.

[0266] (1) When a finger is placed within the aerial operation detection range of the aerial operation detection sensor 1351, the aerial operation detection sensor 1351 does not perform detection. This means, for example, stopping the operation of the aerial operation detection sensor 1351 itself, for example, stopping the emission of light from the light-emitting element of the aerial operation detection sensor 1351.

[0267] (2) Detection by the aerial operation detection sensor 1351 or detection processing by the aerial operation detection unit 1350 is performed, but the detection results are not output. This means, for example, obtaining a detection signal from the aerial operation detection sensor 1351 but not outputting the detection signal, or outputting all detection signals as off values ​​(for example, 0 values). Alternatively, the aerial operation detection unit 1350 may not output a detection result based on the detection signal from the aerial operation detection sensor 1351, or may output a detection result with an off value. Alternatively, the video control unit 1160 or the like may not input the detection result from the aerial operation detection unit 1350, or may input it but not use it.

[0268] (3) The aerial operation detection sensor 1351 performs detection and the aerial operation detection unit 1350 performs detection processing, outputting the detection result of the aerial operation, but the image control unit 1160 or the like does not execute part or all of the image control processing that is normally executed for the aerial operation. For example, suppose the image control unit 1160 receives a value representing a touch operation on an object in the floating-in-space image 3 as the detection result. The image control unit 1160 normally executes a predetermined display control processing associated with the determination that the object has been touched. The display control processing is, for example, the selection or execution of the object. In contrast, in this embodiment, the image control unit 1160 determines that the object has not been touched, and does not execute the predetermined display control processing. In other words, this control disables the aerial operation on the object in the floating-in-space image 3, or disables the display control processing of the object.

[0269] In any of the above controls, from the user 230's perspective, the result is that the space floating image display device 1000 including the aerial operation detection sensor 1351 does not react to the aerial operation on the space floating image 3.

[0270] [Operation Modes According to the State of the User Operation Detection Mechanism] The table in FIG. 21A summarizes several possible states for accepting and detecting mid-air operations, based on combinations of the disabled (OFF) and enabled (ON) states of each user operation detection mechanism, in the case of the example configuration having the user operation detection mechanism 1701 and the user operation detection mechanism 1702 of FIG. 17 . These states may be referred to as modes. The table in FIG. 21A has, as rows, a user operation detection mechanism 1701 capable of detecting operations in the X-Z directions, a user operation detection mechanism 1702 capable of detecting operations in the Y direction, operation detection and mode, and GUI display in the floating-in-space image 3. The columns have, for example, four types of states, from state 1 to state 4. Other states are possible, which will be described later.

[0271] First, in state 1, both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are in the OFF state. In state 1, 2D and 3D mid-air operations are not detected and are not accepted. For the sake of explanation, state 1 is referred to as mode 1.

[0272] State 2 is a state in which the user operation detection mechanism 1701 is ON and the user operation detection mechanism 1702 is OFF. State 2 is a state in which only planar operations in the XZ directions, i.e., 2D operations, are detected and accepted. State 2 is referred to as Mode 2. Mode 2 is also referred to as 2D operation mode.

[0273] State 3 is when the user operation detection mechanism 1701 is in an OFF state and the user operation detection mechanism 1702 is in an ON state. State 3 is a state in which only depth operations can be detected and accepted. State 3 is referred to as Mode 3. For example, when a finger enters the plane of the floating image 3 from the front to the back, this state 3 can detect whether the finger is in front of the plane, within the plane, or behind the plane. In other words, this state 3 can detect the position of the finger in the Y direction, but cannot detect the position in the X direction or the Y direction within the plane.

[0274] State 4 is a state in which both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are in the ON state. State 4 is capable of detecting both planar operations in the XZ directions and depth operations in the Y direction. State 4 is referred to as mode 4. That is, mode 4 is a mode in which both 2D operations and 3D operations can be accepted. Mode 4 is also referred to as 3D operation mode. In mode 4, when a planar operation is detected, it is accepted as a 2D operation, and the corresponding display control processing is executed. In mode 4, when a depth operation is detected, it is accepted as a 3D operation, and the corresponding display control processing is executed.

[0275] Furthermore, the following modes are also possible as other states. The table in FIG. 21B similarly shows states 5 and 6. State 5, like state 4, is when both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are ON, and both planar and depth operations can be detected. State 5 is referred to as mode 5. In state 5, both planar and depth operations can be detected during processing, i.e., 3D operations can be determined and detected, but only planar operations are accepted as 2D operations as valid aerial operations. In mode 5, even if depth operations are detected, they are not accepted as 3D operations, and corresponding display control processing is not executed. For example, when a planar operation is detected for a button object that accepts planar operations, display control processing is executed. In state 5, the user operation detection mechanism 1702 is used as an auxiliary processing element.

[0276] State 6, like State 4, is a state in which both the user operation detection mechanism 1701 and the user operation detection mechanism 1702 are ON, and both planar operations and depth operations can be detected. State 6 is referred to as Mode 6. In State 6, both planar operations and depth operations can be detected during processing, i.e., 3D operations can be determined and detected, but only depth operations are accepted as 3D operations as valid aerial operations. In Mode 6, even if a planar operation is detected, it is not accepted as a 2D operation, and corresponding display control processing is not executed. For example, if a depth operation is detected for a button object that accepts depth operations, display control processing is executed. In State 6, the user operation detection mechanism 1701 is used as an auxiliary processing element.

[0277] The space floating image display device 1000 of the third embodiment implements at least the 2D operation mode, which is the above-mentioned mode 2, and the 3D operation mode, which is the above-mentioned mode 4. However, other modes may be implemented in a modified example.

[0278] The space-floating image display device 1000 of Example 3 further controls the display of the space-floating image 3 to provide a GUI display corresponding to each mode, as shown in the fourth row of the table. Not only GUIs, but content and the like can also be controlled in a similar manner. Examples of GUI displays for each mode are as follows: For the GUI display of Mode 1, a display indicating that aerial operations are not accepted is provided. For the GUI display of Mode 2, a display encouraging 2D operations (planar operations) is provided. Also, for example, a display indicating a 2D operation mode is provided. For the GUI display of Mode 3, a display encouraging depth operations is provided. For the GUI display of Mode 4, a display indicating that both 2D operations and 3D operations are accepted, or a display encouraging 3D operations, is provided. Also, for example, a display encouraging a 3D operation mode is provided. For the GUI display of Mode 5, a display encouraging planar operations (2D operations) is provided. For the GUI display of Mode 6, a display encouraging depth operations (3D operations) is provided.

[0279] [Reason for Providing Two Types of Modes: 2D Operation Mode and 3D Operation Mode] In Example 3, two types of modes, the 2D operation mode and the 3D operation mode, are distinguished and can be switched between as appropriate. The reasons for this configuration are as follows: Each of the above modes has its own advantages. In the 3D operation mode, which detects and processes 3D operations, the operational load on the hardware and software of the space-floating image display device 1000 is generally greater than in the 2D operation mode, which detects and processes 2D operations. Therefore, depending on the implementation, in the 3D operation mode, power consumption may be greater, detection accuracy may be more difficult to improve, and detection time may be longer. In other words, in the 2D operation mode, power consumption may be lower, detection accuracy may be higher, and detection time may be shorter. In other words, although the 2D operation mode limits the effective aerial operations available to the user 230, it has the advantage of providing faster response to user operations. Therefore, if 2D operation is sufficient as a UI, switching to the 2D operation mode can improve operability for the user 230 and reduce power consumption, etc. Conversely, if 3D operation is suitable for the UI, the 3D operation mode can be set to accept more complex 3D operations. These can be used depending on the application and situation.

[0280] [3-2. Changing the display between 2D mode and 3D mode] In Example 3, as described above, in a system capable of accepting both 2D and 3D operations, the display of the GUI etc. of the floating-in-space image 3 is changed in each of the 2D operation mode and the 3D operation mode.

[0281] As described above (FIG. 18, etc.), aerial operations as user operations can be broadly divided into 2D operations and 3D operations. 2D operations are planar operations, two-dimensional operations, performed within the aerial operation detection range in the planar direction, corresponding to the plane of the display range of the floating-in-space image 3. 3D operations are three-dimensional operations including depth operations, performed within the aerial operation detection range, including the planar direction and depth direction, corresponding to the plane of the display range of the floating-in-space image 3. In addition, the floating-in-space image display device 1000 changes the display content of the floating-in-space image 3 in a state where only 2D operations are accepted (mode 2, 2D operation mode in FIG. 21A) and a state where 3D operations are accepted in addition to 2D operations (mode 4, 3D operation mode in FIG. 21A).

[0282] The following issues can be cited. In a space-floating image display system capable of 2D and 3D operations, the user may be confused about the mid-air operation of the space-floating image, and may not be able to perform the mid-air operation expected by the system. The user may not know whether the space-floating image accepts 2D operations or 3D operations, and may not be able to determine whether they should perform 2D operations or 3D operations. For example, it may be difficult for the user to understand that the space-floating image accepts 3D operations when it is displaying 2D images. Furthermore, it may be difficult for the user to understand that the space-floating image 3 accepts 2D operations when it is displaying 3D images.

[0283] The third embodiment has the following solution. When a 2D operation is accepted, for example, in 2D operation mode, a GUI or content of the floating-in-space image 3 is displayed so as to induce a planar operation, i.e., a 2D operation, by the user 230. This display may be, for example, a browser screen or a planar figure, which does not give a sense of depth or makes it difficult to feel it. Alternatively, this display may be a display that has undergone image processing or the like so as to not give a sense of depth or makes it difficult to feel it. This display may typically be a 2D image display, but is not limited to this, and may be any display such as an image that does not give a sense of depth so as to encourage 2D operation.

[0284] Similarly, when a 3D operation is accepted, for example, in 3D operation mode, the GUI or content of the floating-in-space image 3 is displayed so as to induce 3D operations, including operations in the depth direction, by the user 230. This display may be, for example, a display that gives / makes easy to sense depth, such as a three-dimensional figure. Alternatively, this display may be a display that has undergone image processing or the like to give / make easy to sense depth. This display may typically be a 3D image display, but is not limited to this and may be any display such as an image that gives / makes easy to sense depth so as to encourage 3D operation.

[0285] In this way, in the third embodiment, the GUI of the floating-in-space image 3 and the display of the content are changed and controlled according to the 2D operation / 3D operation. When accepting a 2D operation, for example, if an input is desired for an object using a 2D operation, the object is displayed in a manner that encourages the 2D operation. This makes it difficult for the user to be aware of the 3D operation, and encourages the 2D operation. Furthermore, when accepting a 3D operation, for example, if an input is desired for an object using a 3D operation including a depth operation, the object is displayed in a manner that encourages the 3D operation. This makes it difficult for the user to be aware of the 2D operation, and encourages the 3D operation.

[0286] This reduces the user's confusion about the aerial operation between 2D and 3D operations. Also, the aerial operation that the floating-in-space image display system expects from the user can be achieved with less disturbance and higher detectability. Specific examples of displays that encourage 2D operations and 3D operations on the floating-in-space image 3 will be described later.

[0287] [Implementation Examples for Two Types of Operation Modes] Implementation examples of the space floating image display device 1000 for the above two types of operation modes, 2D operation mode and 3D operation mode, are as follows. That is, there are a system that implements only the 2D operation mode, a system that implements only the 3D operation mode, and a system that implements both the 2D operation mode and the 3D operation mode and is switchable between them. Any of the implementation examples is possible. Each implementation example will be explained in order below. Fig. 22 shows an implementation example that has only the 2D operation mode. Fig. 23 shows an implementation example that has only the 3D operation mode. Fig. 24 shows an implementation example that has both the 2D operation mode and the 3D operation mode.

[0288] [Implementation Example 1: 2D Operation Mode] FIG. 22 shows a Y-Z cross-sectional view of the space-floating image display device 1000 in which a 2D operation mode, a mode that accepts only 2D operations, is implemented as implementation example 1, and also shows the state of 2D operation as the mid-air operation of the user 230 in the 2D operation mode. In the example of FIG. 22, the same components as those shown in FIG. 16 are implemented in a horizontally placed housing 1190. Image light from the retroreflective member 2 travels diagonally upward, passes through the polarization separation member 101 and the transparent member 100, and forms a real image, a space-floating image 3, in a display range 2203 at a predetermined external position. This space-floating image 3 is a space-floating image for 2D operation. An aerial operation detection sensor 1351, for example, is implemented in the housing 1190 as a 2D operation detection device 2201, which is a user operation detection mechanism 2201. The aerial operation detection sensor 1351 can be configured as shown in FIG. 19, for example.

[0289] The aerial operation detection sensor 1351 is installed near the transparent member 100 on the top surface of the housing 1190, on the front side closer to the user 230 in the Y direction. The detection optical axis of this aerial operation detection sensor 1351 faces diagonally upward, i.e., in the y direction, as shown by the dashed-dotted arrow, and overlaps with the display range 2203 of the floating-in-space image 3 for 2D operation. The aerial operation detection sensor 1351 forms a 2D operation detection range 2202. The 2D operation detection range 2202 is a range that encompasses the display range 2203 of the floating-in-space image 3 for 2D operation. In other words, the display range 2203 of the floating-in-space image 3 for 2D operation is set within the 2D operation detection range 2202.

[0290] The coordinate system of the floating in space image 3 is represented by (x, y, z). The x direction is the horizontal or left-right direction on the screen of the floating in space image 3, the y direction is the vertical or up-down direction on the screen of the floating in space image 3, and the z direction is the depth or front-to-back direction relative to the floating in space image 3. In other words, the 2D operation detection range 2202 is a range along the xy plane, which is the display range 2203 of the floating in space image 3. The mid-air operation detection sensor 1351, which is the 2D operation detection device 2201, senses the 2D operation detection range 2202, which includes the xy plane of the floating in space image 3. This detects the 2D operation as a planar operation on the xy plane of the floating in space image 3. The user 230 can easily view the floating in space image 3 in the z direction.

[0291] The finger position 2211 of the user 230 is an example of a case where a finger, for example, one finger, is placed within the x-y plane of the floating-in-space image 3. The user 230 performs 2D operations, for example, by touching or tapping the finger on the x-y plane of the floating-in-space image 3, or by moving the finger in the x or y direction within the x-y plane of the floating-in-space image 3. Note that touch and tap operations are basic operations that can be determined by whether or not the finger is placed within the x-y plane, and are included in the concept of 2D operation. The floating-in-space image display device 1000 in 2D operation mode detects such 2D operations using the 2D operation detection device 2201, and executes display control processing as processing according to the detected 2D operation.

[0292] The space-floating image display device 1000 in 2D operation mode displays images such as a GUI for 2D operation and content on the space-floating image 3 for 2D operation. At the bottom of Fig. 22, an example of a 2D image display encouraging 2D operation is shown as an example of the space-floating image 3 for 2D operation. In this example, a character image 2231 and a button object 2232 are displayed. The character image 2231 is a character image that guides the slide operation, for example, "Please Slide!" The button object 2232 is a GUI that accepts slide operations in the x-y directions. These images are flat images that do not give the impression of depth.

[0293] [Implementation Example 2: 3D Operation Mode] Figure 23 shows, as implementation example 2, a Y-Z cross-sectional view of the space floating image display device 1000 in which a 3D operation mode that accepts both 2D operation and 3D operation simultaneously is implemented, and also shows the state of 3D operation as mid-air operation by the user 230 in the 3D operation mode.

[0294] In the example of FIG. 23 , similar to FIG. 22 , the above-described components are mounted inside a horizontally placed housing 1190. A real image, a floating-in-space image 3, is formed in a display range 2203 at a predetermined position outside the housing 1190. This floating-in-space image 3 is a floating-in-space image for 3D operation. Similar to FIG. 22 , an aerial operation detection sensor 1351, for example, is mounted on the housing 1190 as a 2D operation detection device 2201. The 2D operation detection device 2201 has a 2D operation detection range 2202 similar to that of FIG. 22 , and senses the xy plane of the display range 2203 of the floating-in-space image 3. In addition, a 3D operation detection device 2301 is mounted on the housing 1190. In this example, the 3D operation detection device 2301 is installed near the transparent member 100, which is the top surface of the housing 1190, at a position further back in the Y direction than the 2D operation detection device 2201.

[0295] The 3D operation detection range 2302 is a detection range for aerial operations by the 3D operation detection device 2301, and covers a spatial region in the z direction, which is the depth direction, relative to the 3D operation floating image 3 in space and the 2D operation detection range 2202 of the 2D operation detection device 2201. In this example, the 3D operation detection range 2302 covers a spatial region behind the floating image 3 in space, but is not limited to this, and may also cover a spatial region on the front side (user 230 side) in the z direction, which is the depth direction relative to the floating image 3 in space. The 3D operation detection device 2301 is a mechanism capable of detecting the position and movement of fingers in the z direction. For example, the camera of the imaging unit 1180 or other sensors can be used as the 3D operation detection device 2301. Without being limited to this, the 3D operation detection device 2301 may be installed in other positions.

[0296] A three-dimensional detection range 2303 in the 3D operation mode is formed by combining the 2D operation detection range 2202 of the 2D operation detection device 2201 and the 3D operation detection range 2302 of the 3D operation detection device 2301 .

[0297] As a variation of FIG. 23, the 2D operation detection device 2201 or the 3D operation detection device 2301 may be installed at a position shown as detection device 2350, and sensing may be performed in the xy direction or z direction from detection device 2350 toward the floating image 3 in space, which is located in front of it in the Y direction.

[0298] The finger position 2311 of the user 230 is an example of the position when the fingers, for example, the palm, are placed within the 3D operation detection range 2302 including the x-y plane of the floating-in-space image 3. The user 230 performs, for example, an operation of touching the x-y plane of the floating-in-space image 3 with the palm and moving the palm, or an operation of moving the palm to the back in the z direction, which is the depth direction from the x-y plane of the floating-in-space image 3. As another example, the finger position 2312 is an example of the position when a pressing operation is performed in which one finger is inserted to the back in the z direction, which is the depth direction from the x-y plane of the floating-in-space image 3. The floating-in-space image display device 1000 in the 3D operation mode detects such 2D or 3D operations using the 2D operation detection device 2201 and the 3D operation detection device 2301, and executes a display control process according to the detected mid-air operation.

[0299] The space-floating image display device 1000 in 3D operation mode displays images such as a GUI for 3D operation and content on the space-floating image 3 for 3D operation. An example of a 3D image display encouraging 3D operation is shown at the bottom of Fig. 23 as an example of the space-floating image 3 for 3D operation. In this example, a character image 2331 and a button object 2332 are displayed. The character image 2331 is a character image that guides a push operation, for example, "Please Push!" The button object 2332 is a GUI that accepts a push operation in the z direction. These images are three-dimensional images that easily give a sense of depth.

[0300] [Implementation Example 3: 2D Operation Mode and 3D Operation Mode] Figure 24 shows, as Implementation Example 3, a Y-Z cross-sectional view of the space-floating image display device 1000, which is implemented with two types of modes, 2D operation mode and 3D operation mode, and which can be selected and switched between them, and also shows the operation state of the user 230 in the selected mode (e.g., 3D operation mode). The configuration of Figure 24 is similar to the configuration of Figure 23, but differs in that the space-floating image display device 1000 has a control function for appropriately switching between the 2D operation mode and the 3D operation mode, and this control function can be realized by an image processing unit such as the control unit 1110 and image control unit 1160 in Figure 3. Furthermore, in Implementation Example 3, as shown in the figure, an indicator that displays the status of the current mode, etc., and an operation button 2309 (operation input unit 1107 in Figure 3) that allows the mode to be selected may be provided on the housing 1190.

[0301] In the 2D operation mode in which only 2D operations are accepted, the space-floating image display device 1000 uses the 2D operation detection device 2201 to detect planar operations on the xy plane of the space-floating image 3, and executes processing according to the detected planar operations. In the 3D operation mode in which both 2D operations and 3D operations are accepted simultaneously, the space-floating image display device 1000 uses the 2D operation detection device 2201 and the 3D operation detection device 2301 to detect planar operations and depth operations on the xy plane of the space-floating image 3, and executes processing according to the detected 2D operation or 3D operation.

[0302] In 2D operation mode, the space-floating image display device 1000 displays images such as a GUI for 2D operation and content on the space-floating image 3 for 2D operation. In 3D operation mode, the space-floating image display device 1000 displays images such as a GUI for 3D operation and content on the space-floating image 3 for 3D operation. The bottom of Fig. 24 shows an example of a space-floating image 3 for 2D operation in 2D operation mode, and an example of a space-floating image 3 for 3D operation in 3D operation mode. The space-floating image display device 1000 switches these displays depending on the mode.

[0303] The implementation is not limited to the above example, and a system may be implemented with modes other than those shown in Fig. 21A. For example, mode 5 may be implemented as a 2D operation mode, and mode 6 may be implemented as a 3D operation mode.

[0304] [Example of Control of Switching of User Operation Detection Mechanism] In the third embodiment, the control of the user operation detection mechanism may be as follows. The space-floating image display device 1000 sets the user operation detection mechanism in the planar direction, for example, the user operation detection mechanism 1701 in FIG. 17, to an enabled state (ON state) in which it is always able to accept operations, and sets the user operation detection mechanism in the depth direction, for example, the user operation detection mechanism 1702 in FIG. 17, to an disabled state (OFF state) in which it is always unable to accept operations. Then, the space-floating image display device 1000 switches the user operation detection mechanism in the depth direction 1702 to an enabled state (ON state) in which it is able to accept operations upon a predetermined trigger. Examples of the predetermined trigger include when a sensor such as the imaging unit 1180 detects the approach of the user 230, when the user operation detection mechanism 1701 detects an operation in the planar direction, or when a sensor such as the imaging unit 1180 detects that the user 230 is about to perform not only an operation in the planar direction but also an operation in the depth direction.

[0305] [Example of Operation Mode Switching Control] For example, in a configuration in which two types of operation modes, 2D operation mode and 3D operation mode, are switched as shown in FIG. 24, the following is an example of control for switching the operation mode.

[0306] The space-floating image display device 1000 initially starts in, for example, 2D operation mode. Examples of initial operation include when the space-floating image display device 1000 is started up, when an application is started up, or when a sensor such as the imaging unit 1180 detects the approach of the user 230. The 2D operation mode has a simpler mechanism than the 3D operation mode, and therefore has advantages in terms of processing efficiency and power consumption. When switching to the 2D operation mode, the space-floating image display device 1000 turns the 2D operation detection device 2201 ON and the 3D operation detection device 2301 OFF. This enables detection of 2D operations. In the 2D operation mode, the space-floating image display device 1000 displays a space-floating image 3 for 2D operation, for example, an image prompting 2D operation. When the space-floating image display device 1000 receives a 2D operation, it executes a corresponding display control process.

[0307] The space-floating image display device 1000 switches from 2D operation mode to 3D operation mode as appropriate based on a predetermined trigger or judgment. When switching to 3D operation mode, the space-floating image display device 1000 turns on the 2D operation detection device 2201 and the 3D operation detection device 2301. This makes it possible to detect 2D and 3D operations. In 3D operation mode, the space-floating image display device 1000 displays a space-floating image 3 for 3D operation, for example, an image that prompts 3D operation. When the space-floating image display device 1000 receives a 2D operation or a 3D operation, it executes the corresponding display control process.

[0308] Furthermore, thereafter, the space-floating image display device 1000 switches from 3D operation mode to 2D operation mode as appropriate based on a predetermined opportunity or judgment. When switching to 2D operation mode, the space-floating image display device 1000 turns the 2D operation detection device 2201 to the ON state and the 3D operation detection device 2301 to the OFF state. This makes it possible to detect 2D operations. In the 2D operation mode, the space-floating image display device 1000 displays a space-floating image 3 for 2D operation, for example, an image that prompts 2D operation. When the space-floating image display device 1000 receives a 2D operation, it executes the corresponding display control process.

[0309] [Triggers for determining a mode] There are various triggers and conditions for selection, switching, and determination regarding the above-mentioned 2D operation mode, 3D operation mode, etc. Fig. 25 shows a table summarizing triggers for determining a mode.

[0310] (1) Setting the mode on the space-floating image display device 1000. The mode to be used is set in advance for each space-floating image display device 1000. A person such as a business operator selects and sets the mode to be used when installing the device. The space-floating image display device 1000 also provides a setting screen for setting the mode (for example, similar to FIG. 26).

[0311] (2) Setting the mode in the application of the space floating image display device 1000. The mode to be used is set in advance for each application. For example, a 2D operation mode is used for a window information application, and a 3D operation mode is used for an application that involves conversation with a character.

[0312] (3) Mode designation operation input by user 230. The space floating image display device 1000 may be provided with an operation input device such as a physical button for selecting and switching modes. Alternatively, a display such as a button for selecting and switching modes may be provided on the display of the space floating image 3, etc. For example, see FIG. 26.

[0313] (4) Mode for each individual image to be displayed. For example, the mode to be used is selected for each GUI or content, in other words, for each type of image or ID. For example, the attribute information attached to the image data of the image to be displayed may describe 2D or 3D attributes or mode designation, and the space-floating image display device 1000 may determine and select the mode to be used based on that description (described later). The creator or provider of the image data of the image to be displayed may set information on at least one of 2D / 3D display and 2D / 3D operation. For example, if a GUI object accepts 2D operation, the 2D operation mode may be specified. In the space-floating image display device 1000, the correspondence between the type of image or ID, 2D display / 3D display, and 2D operation / 3D operation may be set in a correspondence table or the like.

[0314] (5) Automatic determination by the space-floating image display device 1000. For example, the space-floating image display device 1000 may determine whether to switch to 2D operation mode or 3D operation mode based on the situation of mid-air operation by the user 230 on the space-floating image 3. Details of this will be described in Example 4.

[0315] [Example of Mode Selection by User] Fig. 26 shows an example of an operation input device and a display example of the floating-in-space image 3 when the user 230 can select and specify the operation mode to use from two types of operation modes, for example, in a configuration like Fig. 24. As examples of indicators and operation buttons 2309 on the housing 1190, a physical 2D button 2601 and a 3D button 2602 are provided. The user 230 presses the 2D button 2601 when wanting to switch to 2D operation mode, and presses the 3D button 2602 when wanting to switch to 3D operation mode. The floating-in-space image display device 1000 switches to the operation mode corresponding to the pressed button.

[0316] In this example, the floating-in-space image 3 displays a message "Please select a mode" within an outer frame that indicates the display range, and a GUI is displayed that allows the user to select between 2D operation mode and 3D operation mode by mid-air operation. The user 230 touches the "2D operation mode" area if they want to switch to 2D operation mode, and touches the "3D operation mode" area if they want to switch to 3D operation mode. The floating-in-space image display device 1000 detects the touch operation and switches to the operation mode corresponding to the pressed area.

[0317] [Control of Correspondence Between Aerial Operation Detection and Space Floating Image Display] The above has described a control example in which the starting point is whether to accept 2D operation or 3D operation, i.e., whether to use 2D operation mode or 3D operation mode, and then accordingly decides whether to display 2D display (display for 2D operation, for example, display encouraging 2D operation) or 3D display (display for 3D operation, for example, display encouraging 3D operation) as the display content of the space floating image 3. Control of the correspondence between aerial operation detection and space floating image display is not limited to this.

[0318] FIG. 27 shows a table summarizing control examples of the correspondence between the detection of an aerial operation and the display of the floating-in-space image 3. In Example 3, as described above, the floating-in-space image display device 1000 may control whether the floating-in-space image 3 is displayed in 2D or 3D, starting from whether the aerial operation and its detection are 2D or 3D operations. The 2D or 3D operation is determined first, and then the corresponding correspondence is indicated by the arrows, and the display of the floating-in-space image 3 is determined in 2D or 3D. Rows #1 and #2 in the table show such control examples. #1 is a control that determines 2D display in response to 2D operation, and #2 is a control that determines 3D display in response to 3D operation. Note that here, 2D display or 3D display is also described as 2D display mode / 3D display mode. Control examples #1 and #2 correspond to, for example, selecting a display mode in response to the setting or specification of an operation mode, triggered by #1, #2, and #3 in FIG. 25.

[0319] Conversely, control may be performed so that the mid-air operation and its detection are determined as 2D or 3D operations, starting from whether the display of the floating image 3 in space is 2D or 3D. First, it is determined whether the display of the floating image 3 in space is 2D or 3D, and then the detection of the mid-air operation is determined as 2D or 3D operation, as indicated by the corresponding arrows. Rows #3 and #4 in the table show such control examples. #3 is control that determines 2D operation depending on the 2D display, and #4 is control that determines 3D operation depending on the 3D display. Control examples of #3 and #4 correspond to, for example, selecting an operation mode depending on whether the image to be displayed is 2D or 3D, triggered by #4 in FIG. 25.

[0320] The space floating image display device 1000 may apply any of the above control examples. In the above explanation, the operation mode and the display mode are defined separately, but the operation mode and the display mode may be defined as a single mode. Even when specifying a mode as in the example of Fig. 26, it may be possible to specify the display mode instead of the operation mode.

[0321] [Video Data and Mode Information] Figure 28 shows an example of the configuration of a floating-in-space video display system when video data is accompanied by information about the mode, also corresponding to the example of opportunity #4 in Figure 25. The mode here is the aforementioned operation mode or display mode as an example, but in other words, it is an aerial operation method, an aerial operation detection method, a display method, etc.

[0322] In the system of FIG. 28 , an external device 2800 is connected to the space-floating image display device 1000. The external device 2800 is an image provider for the image to be displayed on the space-floating image 3. The external device 2800 creates image data 2811 of the image to be displayed and provides it to the space-floating image display device 1000 via communication or the like. The space-floating image display device 1000 acquires the image data 2811 via communication or the like and stores it in memory. The image processing unit 2801 performs necessary image processing based on the image data 2811 to create display data (not shown), supplies this to the display unit 2802, and controls the display unit 2802. The display unit 2802 displays the image on a screen such as a liquid crystal display panel based on the display data. The image light of the image displayed on the screen is emitted via an optical system 2803, and the space-floating image 3 is formed. Furthermore, the image processing unit 2801 controls the user operation detection mechanism 2804, and detects an aerial operation by the user 230 on the floating in space image 3 by the user operation detection mechanism 2804. The image processing unit 2801 executes display control processing according to the detected aerial operation on objects in the image, etc.

[0323] Furthermore, in this embodiment, the external device 2800 creates or sets 2D or 3D GUI and content in the video data 2811, and also specifies and sets 2D / 3D operation or mode related to mid-air operation for the video in attribute information 2812 attached to the video data 2811. The space-floating image display device 1000 acquires such video data 2811 and attribute information 2812 from the external device 2800, and determines the above-mentioned 2D operation mode or 3D operation mode according to the operation or mode specification in the attribute information 2812. The video processing unit 2801 controls the display on the display unit 2802 and the detection by the user operation detection mechanism 2804 so as to detect 2D / 3D mid-air operation and display 2D / 3D floating-in-space image 3 corresponding to the operation mode.

[0324] An example of the description of attribute information 2812 is shown in a balloon at the bottom of Fig. 28. In example 1, the image type of video data 2811 is image type 1, the image is created as a 2D display (in other words, a 2D image), and 2D operation is specified as the operation. Instead of specifying 2D operation, 2D operation mode may be specified. In example 2, the image type of video data 2811 is image type 2, the image is created as a 3D display (in other words, a 3D image), and 3D operation is specified as the operation. Instead of specifying 3D operation, 3D operation mode may be specified.

[0325] In another embodiment, the space floating image display device 1000, rather than the external device 2800, may assign the attribute information 2812 to the image data 2811. For example, the image processing unit 2801 may determine a correspondence relationship between the 2D display / 3D display mode of the image data 2811 of the image to be displayed and the 2D operation / 3D operation of the aerial operation, and describe the correspondence relationship in the attribute information 2812.

[0326] For example, when the aerial operation modes of 2D operation and 3D operation are determined for video data 2811 of the video to be displayed, the video processing unit 2801 may determine the display modes of 2D display and 3D display to match the mode of the aerial operation, and then select and determine the operation mode corresponding to the mode of the aerial operation and the display mode corresponding to the display mode.Furthermore, when the display modes of 2D display and 3D display are determined for video data 2811 of the video to be displayed, the video processing unit 2801 may determine the aerial operation modes of 2D operation and 3D operation to match the display mode, and then select and determine the operation mode corresponding to the mode of the aerial operation and the display mode corresponding to the display mode.

[0327] The video processing unit 2801 may create display data by performing image processing or the like on the original video data 2811 to create 2D or 3D display in the space floating image 3. Alternatively, the video processing unit 2801 may create the video data 2811 itself, and determine the mode of 2D operation / 3D operation according to the mode of 2D display / 3D display in the created video data 2811.

[0328] [3-3. Examples of 2D and 3D operations] Next, specific examples of the above 2D and 3D operations are given below. In the table of Fig. 29A, examples of 2D operations are shown in the left column, and examples of 3D operations are shown in the right column. The space floating image display device 1000 of Example 3 is configured to detect and accept such 2D and 3D operations (the implementation example described above).

[0329] Examples of 2D operations include the following. Numbers in parentheses are identifiers for explanatory purposes. (1-1) Tap. Placing a finger on the plane of the floating-in-space image 3 and then lifting it towards you. (1-2) Swipe. In other words, slide. Placing a finger on the plane of the floating-in-space image 3 and moving it in a planar direction. (1-3) Flick. Placing a finger on the plane of the floating-in-space image 3 and moving it quickly in a planar direction. (1-4) Multi-swipe. Placing multiple points of your finger on the plane of the floating-in-space image 3 and moving it in a planar direction. (1-5) Hold. Placing your finger on the plane of the floating-in-space image 3 for a certain period of time or more. (1-6) Circular rotation after hold. Rotation within the plane of the floating-in-space image 3. (1-7) Pinch in / pinch out. Placing multiple points of your finger on the plane of the floating-in-space image 3 and changing the spacing between those multiple points.

[0330] Examples of 3D operations include the following: (2-1) Push: Moving your fingers in three dimensions, including the depth direction, relative to the plane of the floating image 3, passing through said plane and moving your fingers to the back of said plane. (2-2) Grab: Grabbing an object displayed on the plane of the floating image 3 with your fingers or palm. (2-3) Grab and rotate: Rotation in three dimensions. (2-4) Grab and move: Movement in three dimensions. (2-5) Swipe: For example, moving the palm of your hand left or right. (2-6) Pull: For example, moving the palm of your hand that is grasping an object towards you in the depth direction.

[0331] Fig. 29B shows an example of a display control process executed in association with the aerial operation shown in Fig. 29A. Each row corresponds to an example of 2D operation or 3D operation shown in Fig. 29A. The left column shows examples of 2D GUI and content for the floating-in-space image 3, and the right column shows examples of 3D GUI and content.

[0332] For example, a tap operation (1-1) as a 2D operation corresponds to object selection / execution as an example of a control process for 2D GUI / content. That is, when the image processing unit of the space-floating image display device 1000, for example, the image control unit 1160 of FIG. 3, detects a tap operation on an object displayed as a GUI or content using 2D images in the space-floating image 3, it executes a process to control the display of the space-floating image 3 of the object, etc., so that the object is selected or executed. Furthermore, when the image processing unit detects a tap operation on an object displayed as a GUI or content using 3D images in the space-floating image 3, it executes a display control process so that the object is selected or executed.

[0333] Furthermore, for example, when the video processing unit detects a swipe operation (1-2) as a 2D operation on an object in a 2D image, it executes a display control process to move the object by sliding it within the plane of the Floating in Space Image 3, or to transition the screen of a page or the like in the Floating in Space Image 3. Furthermore, when the video processing unit detects a swipe operation (1-2) as a 2D operation on an object in a 3D image, it executes a display control process to slide the object or transition the screen. Furthermore, in this case, an example of the display control process is to rotate the 3D object.

[0334] Furthermore, for example, when the video processing unit detects a hold operation (1-5) as a 2D operation on an object in a 2D video, it executes a display control process to fix, in other words, to make the object still.Furthermore, when the video processing unit detects a hold operation (1-5) as a 2D operation on an object in a 3D video, it similarly executes a display control process to fix, in other words, to make the object still.

[0335] Furthermore, when a 3D operation such as a push operation (2-1) is performed on an object in a 2D image, the image processing unit is associated with performing no operation as an example of display control processing. In other words, when a push operation is performed on an object in a 2D image, the image processing unit does not accept the push operation and performs control to maintain the original display of the object.

[0336] In addition, when a pressing operation (2-1) is performed as a 3D operation on an object in a 3D image, the image processing unit executes a display control process to move the object toward the back in the depth direction relative to the plane of the floating image 3.

[0337] As described above, in the control example of the mid-air operation detection and the display of the floating-in-space image 3 in the third embodiment, for the 2D display image, 2D operations are accepted to allow two-dimensional display changes, but 3D operations including depth operations are not accepted, and three-dimensional display changes including depth are not allowed. Also, for the 3D display image, both 2D operations and 3D operations are accepted to allow two-dimensional display changes and three-dimensional display changes. This makes it easier to understand the correspondence between 2D and 3D in the mid-air operation detection and the display of the floating-in-space image 3, and the user 230 is less likely to be confused by the mid-air operation.

[0338] The correspondence between 2D / 3D aerial operations and 2D / 3D displays, as in the above example, may be designed and set in advance by the business operator of the space-floating image display device 1000. It is preferable to set this correspondence depending on the space-floating image display device 1000 and the application, etc. For example, in the case of a window information application, it is specified whether the images of GUI, content, etc. displayed on the space-floating image 3 will be displayed in 2D or 3D, and it is also specified whether the aerial operations to be accepted for that image will be 2D operations or 3D operations, and the associated display control process.

[0339] [3-4. Examples of 2D and 3D Displays] Next, specific examples of 2D and 3D displays for prompting the above-mentioned 2D and 3D operations in images such as the GUI and content of the floating-in-space image 3 will be described. The table in Fig. 30A shows examples of 2D displays that prompt 2D operations in the left column, and examples of 3D displays that prompt 3D operations in the right column. Numbers in parentheses are identifiers.

[0340] The space floating image display device 1000 controls the 2D display and the 3D display to display the appropriate space floating image 3 in a 2D operation mode that accepts only 2D operations, and a 3D operation mode that accepts 3D operations in addition to 2D operations.

[0341] In Figure 30A, examples of 2D displays that prompt 2D operations include: (2D-1) No shading of the object, shading removed. (2D-2) Adding a light shading to the object. (2D-3) Adding a frame to the object. (2D-4) Representation of movement of the object in a planar direction. (2D-5) Planar figure of the object.

[0342] Examples of 3D displays that encourage 3D manipulation include: (3D-1) Adding deep shading to objects. (3D-2) Expressing movement of an object in directions other than a plane (especially in the depth direction). (3D-3) Processing the depth of an object. (3D-4) A three-dimensional model of the object, such as an avatar or character. (3D-5) A three-dimensional shape of the object. (3D-6) Glossing an object.

[0343] Fig. 30B shows a first example as a 2D / 3D display example of a floating in space image 3, corresponding to Fig. 30A. The floating in space image 3 on the left is a 2D displayed floating in space image 3001, which includes a 2D displayed object 3003. The 2D displayed object 3003 is an example of a text image "Please Push!", which is an example of a GUI that encourages a push operation.

[0344] The floating image 3 on the right is a floating image 3002 in 3D display, which includes a 3D object 3004. The 3D object 3004 is an example in which shading is added to the text image "Please Push!" This shading also exists in the z direction, which is the depth direction, and is a type of pseudo-3D expression.

[0345] The video processing unit may perform shading processing on video data or display data corresponding to 2D display space-floating image 3001, thereby creating video data or display data corresponding to 3D display space-floating image 3002. The video processing unit may perform shading-free display or shading removal processing on video data or display data corresponding to 3D display space-floating image 3002, thereby creating video data or display data corresponding to 2D display space-floating image 3001. Note that, for the sake of explanation, while in Fig. 30B the conversion between display examples of space-floating image 3 is shown with arrows, actual conversion is performed between video data or display data, as explained in Fig. 28 .

[0346] FIG. 30C shows a second example of a 2D / 3D display of a floating-in-space image 3 corresponding to FIG. 30A . The floating-in-space image 3 on the left is a floating-in-space image 3011 in 2D display, and includes a 2D object 3013. The 2D object 3013 is an example of a circular image as a planar figure. The floating-in-space image 3 on the right is a floating-in-space image 3012 in 3D display, and includes a 3D object 3014. The 3D object 3014 is an example of a spherical image as a three-dimensional figure. The object 3014 is given a shadow near the bottom right and a gloss near the top left. Similarly, the object 3015 on the left is an example of a concentric circular pattern as a planar figure. Correspondingly, the object 3016 on the right is an example of a pattern that has been subjected to depth processing, and is made darker toward the center and brighter toward the periphery, for example.

[0347] FIG. 30D shows a third example of a 2D / 3D display of a floating-in-space image 3 corresponding to FIG. 30A . The floating-in-space image 3 on the left side is a floating-in-space image 3021 in 2D display, and includes a 2D object 3023. The object 3023 is a triangular image, an example of a planar figure. When a swipe operation or the like is detected as a planar operation (2D operation), a display control process associated with the operation is performed on the object 3023, for example, a movement representation 3023a in a planar direction, and the display changes to show the object 3023b after the movement. Furthermore, when a circular rotation operation after the aforementioned hold operation or the like is detected as a planar operation (2D operation) for the object 3023, a rotation representation 3023c in a planar direction is performed to rotate the object 3023 within the x-y plane, for example.

[0348] On the other hand, the floating-in-space image 3 on the right side is a floating-in-space image 3022 in 3D display, and includes a 3D object 3024. When, for example, a swipe operation as the above-mentioned 2D operation or a grab and rotate operation as a 3D operation is detected for the 3D object 3024, a three-dimensional rotation representation 3024a including the depth direction (in this example, a rotation toward the back around the y-axis) is executed, and the display changes to show the object 3024b after rotation.

[0349] Fig. 30E shows a fourth example of a 2D / 3D display example of a floating-in-space image 3 corresponding to Fig. 30A. The floating-in-space image 3 of (1) shows a floating-in-space image 3031A in 3D or pseudo-3D display, and includes an object 3032A in 3D or pseudo-3D display. The object 3032A is an example of an avatar / character image, and is an animated image generated based on a 3D solid model. It is also possible to extract a still image from such a 3D / pseudo-3D display using image processing to create a 2D / pseudo-2D display.

[0350] (2) Space Floating Image 3 shows an example of the display state of Space Floating Image 3031B that similarly includes an object 3032B such as an avatar. Object 3032B moves, including a three-dimensional rotation 3033, relative to the xy plane.

[0351] (3) Space Floating Image 3 shows an example of the display state of Space Floating Image 3031C that similarly includes an object 3032C such as an avatar. Object 3032C moves in a manner that includes a three-dimensional movement representation 3034 that includes the depth direction relative to the xy plane.

[0352] As shown in FIG. 30A etc., the space-floating image display device 1000 controls the display of the space-floating image 3 so that the user 230 can intuitively operate the space-floating image 3 by appropriately displaying a 2D display that prompts 2D operation and a 3D display that prompts 3D operation. This makes it less likely that the user 230 will be confused between 2D operation and 3D operation. Note that in this embodiment, a control example has been described in which an object is displayed in 2D or pseudo-2D when a 2D operation of the object is prompted, and an object is displayed in 3D or pseudo-3D when a 3D operation of the object is prompted, but this is not limiting. Depending on the object, 3D operation may be accepted in the case of 2D display, or 2D operation may be accepted in the case of 3D display.

[0353] When switching the display content of the spatial floating image 3 in conjunction with a mode change, for example, when switching between 2D image display and 3D image display, in addition to abrupt switching with temporary erasure in between, the display may also be controlled to transition or change continuously from one display to the other over a predetermined period of time.

[0354] Regarding the above-mentioned 2D or 3D display, it is also possible to create something that is essentially 2D but appears pseudo-3D. Such a display is referred to as a pseudo-3D display. It is also possible to create something that is essentially 3D but appears pseudo-2D. Such a display is referred to as a pseudo-2D display. 2D display includes the concept of such pseudo-2D display, and 3D display includes the concept of such pseudo-3D display. For example, as shown in FIG. 30B, an object 3004 obtained by adding shading to a 2D object 3003 can be considered a pseudo-3D display. Also, as shown in FIG. 30C, an object 3016 generated from a planar object 3015 by depth processing appears as a pseudo-3D object, and can therefore be considered a pseudo-3D display. Furthermore, an image such as an animation generated based on a 3D model such as that shown in FIG. 30E, particularly one that has movement in the depth direction, can be considered a 3D display or pseudo-3D display.

[0355] [Regarding Generation of 2D Display and 3D Display] With reference to Figures 30A to 30E, when displaying the 2D or 3D floating image 3, the image processing unit generates display data for the 2D or 3D image by applying rendering processing and shading processing / shading removal processing to the image of the object or the like of the original image data. 2D display includes the concept of pseudo-2D display. 3D display includes the concept of pseudo-3D display. Pseudo-3D display includes, for example, the display of shaded objects and pseudo-3D objects. The shading processing may be a process of generating two-dimensional shading using two-dimensional image processing, or a process of generating three-dimensional shading using three-dimensional image processing.

[0356] As also shown in Fig. 28, the image processing unit of the space-floating image display device 1000 performs processing to create display data for displaying the space-floating image 3 based on the original image data. The display data is data for displaying an image on the screen of, for example, the display device 1 in Fig. 24, the liquid crystal display panel 11 in Fig. 4A, the image display unit 1102 in Fig. 3, or the display unit 2802 in Fig. 28. The image processing unit creates display data for 2D display or 3D display as the display data based on the 2D display or 3D display image in the original image data by rendering in a virtual three-dimensional space, in other words, by rendering-based processing.

[0357] In this rendering, all of the following conversions are possible based on known techniques: Conversion from a 2D display original image to a 2D display space floating image 3. Conversion from a 3D display original image to a 3D display space floating image 3. Conversion from a 2D display original image to a 3D display space floating image 3. Conversion from a 3D display original image to a 2D display space floating image 3.

[0358] An overview of the rendering-based processing is as follows. The video processing unit performs video processing on an image generated by rendering a virtual 3D space in which an object is placed. Rendering of the virtual 3D space is performed by capturing an image using a perspective projection method with a virtual 3D space camera set within the virtual 3D space. The image generated by rendering is an image rendered by capturing an image of the virtual 3D space in which an object is placed, with the angle of view of the virtual 3D space camera set to satisfy predetermined conditions based on the focal length of the lens of the virtual 3D space camera, the distance from which the user 230 views the screen corresponding to the display range of the floating-in-space image 3, the diagonal length of the screen, etc.

[0359] FIG. 31A is a table summarizing examples of conversion from original video data to display data for the floating-in-the-air image 3. The table in FIG. 31A has columns showing original video data, conversion, and display data for the floating-in-the-air image 3. The rows (1) through (4) show various conversion patterns as described above. Various processes are possible for generating display data for forming the floating-in-the-air image 3. For example, in pattern (1), when 2D display video data is used as the original video data, 2D display data for the floating-in-the-air image 3 is generated by rendering based on that 2D display video data. In addition, in pattern (2), when 3D display video data such as a 3D model is used as the original video data, 3D display data for the floating-in-the-air image 3 is generated by rendering based on that data. Conversion from one display data to another is also possible.

[0360] FIG. 31B shows examples of patterns (1) to (4) corresponding to FIG. 31A. The original video data is on the left, and the display data is on the right. Here, the text image "Please Push!" is shown as an example of a video object, and a case where shading is added to the text is shown as an example of 3D or pseudo-3D expression. The display data is an image adjusted to match the size of the display range of the Floating in Space Image 3, etc.

[0361] 3-5. Method for Determining and Detecting Aerial Operations Next, an example of a method for determining and detecting an aerial operation as a user operation will be described.

[0362] The floating-in-space image display device 1000 of the third embodiment prepares an algorithm or table for determining and detecting aerial operations according to the 2D operation mode and an algorithm or table for determining and detecting aerial operations according to the 3D operation mode, and applies the algorithm or table according to the operation mode. This algorithm or table corresponds to software or data for determining whether 2D or 3D operations are valid or invalid as aerial operations, in other words, an aerial operation determination and detection method. The processor performs processing based on cooperation between hardware such as memory and sensors and the software. Since each mode has its own advantages, the third embodiment (e.g., FIG. 24) is configured to appropriately switch between two types of operation modes and the algorithms or tables corresponding to those operation modes. By changing and applying the algorithm or table according to the state of the operation mode, 2D / 3D aerial operations can be suitably detected.

[0363] One problem is that when a state in which 3D operations are accepted is in place, for example, when the device has switched to 3D operation mode, if an algorithm that can only determine 2D operations is applied, the accuracy of detecting user operations will be low.

[0364] In this embodiment, the solution is as follows: (1) The space-floating image display device 1000 is provided with two types of aerial operation detection algorithms, for example, a 2D operation algorithm and a 3D operation algorithm. The image processing unit of the space-floating image display device 1000 switches between these two types of algorithms as appropriate, for example, depending on the operation mode.

[0365] (2) The space-floating image display device 1000 is basically provided with one common algorithm for determining and detecting mid-air operations, regardless of whether the operation is 2D or 3D. Furthermore, the space-floating image display device 1000 is provided with two types of tables, for example, a 2D operation determination table and a 3D operation determination table. The image processing unit of the space-floating image display device 1000 refers to the 2D operation determination table when accepting a 2D operation (for example, a planar operation), and refers to the 3D operation determination table when accepting a 3D operation (for example, a depth operation).

[0366] As described above, by determining and detecting an aerial operation by applying an algorithm or table appropriate for 2D or 3D operation, it is possible to improve the accuracy of detecting an aerial operation.

[0367] FIG. 32A shows an example of a 2D operation determination table as an example of a method for determining an aerial operation, which is a user operation in 2D operation mode. This example shows a case where operations using up to two fingers are supported. For example, x1 and y1 are the x and y coordinates of the position where a first finger is placed on the plane of the floating-in-space image 3 (here, the x-y plane). Similarly, x2 and y2 are the position coordinates of the position where a second finger is placed. "Hold" indicates whether the position at the position coordinates is maintained. "Move" indicates the direction and amount of movement from the position coordinates. "Operation determination" is a determination regarding the accepted aerial operation, such as that shown in FIG. 29A above. "Processing corresponding to the operation" is the display control processing described above, such as object selection, screen scrolling, and zooming in / out.

[0368] For example, the first line shows a determination regarding the tap operation described above (FIG. 29A). The position coordinates of one placement point are (10, 15), and there is no hold or movement. Based on this detection, a tap operation is determined. For example, the second line shows a determination regarding the swipe operation described above. The position coordinates of one placement point are (8, 12), and the amount of movement in the downward y-direction is 5. Based on this detection, a swipe operation, particularly a downward swipe operation, is determined.

[0369] As in the example of FIG. 32A, the image processing unit refers to the display data of the floating image 3, the detection results by the user operation detection mechanism described above, and the operation determination table, to determine and detect the operation of the fingers within the plane of the floating image 3, and executes display control processing associated with the detected operation, in other words, the operation to be accepted, and the object, etc.

[0370] Similarly, Figure 32B is an example of a 3D operation determination table. In this example, only the portion of the 3D operation determination table that shows 3D operations including the z direction, which is the depth direction, is shown, but 2D operations that do not include the depth direction can also be determined in the 3D operation determination table. "Initial x" and "Post-x movement" are the initial and post-movement coordinates in the x direction. The same applies to the y and z coordinates. "Shape" is the shape of the fingers used as the operation object.

[0371] For example, the first line indicates a determination regarding the pressing operation described above ( FIG. 29A ). For example, in the z direction, which is the depth direction, the movement amount in the z+ direction from front to back is 5. The shape is one finger, and there is a hold. Based on this detection, a pressing operation is determined, and in response to the determination of the pressing, an object is determined as a process associated with it.

[0372] In the second line, the shape is a palm, and the movement amount in the x-direction to the left is 6. Based on this detection, the operation is determined to be a "left sweep," and correspondingly, the "advance left / go back" process is executed. In the third line, the shape is a palm, there is a hold, and the movement amount in the z-direction from back to front is 8. Based on this detection, the operation is determined to be a "pull toward you," and correspondingly, the "enlarge / reduce" process is executed. In the fourth line, the shape is a palm, there is a hold, and there is no movement. Based on this detection, the operation is determined to be a "grasp," and correspondingly, the "end" process is executed. In the fifth line, the shape is a palm, there is a hold, and the movement includes rotation and a movement amount of 5 in the x-direction and a movement amount of 5 in the y+direction. Based on this detection, the operation is determined to be a "grab and release," and correspondingly, the "display update" process is executed.

[0373] As in the above example, when judging and detecting an operation using an algorithm / table for 2D operation in 2D operation mode, since the judgment of the depth direction is not included, judgment and detection can be performed faster and with less load than in 3D operation mode.

[0374] [3-6. Processing flow for switching between multiple modes] Next, Fig. 33 shows the processing flow of the space floating image display device 1000 when, for example, in a configuration like Fig. 24, the multiple modes are switched between 2D operation mode and 3D operation mode, and control is performed to change the display of the space floating image 3 for each operation mode.

[0375] In step S1, the image processor of the space floating image display device 1000 checks the current operation mode setting. If it is the 2D operation mode described above, the process proceeds to step S2, and if it is the 3D operation mode, the process proceeds to step S3. In step S1, the operation mode setting is checked to determine the mode to be used first. In step S1, the mode to be used may be determined by checking various triggers and conditions such as those shown in FIG. 25 described above.

[0376] In step S2, the image processing unit displays the space floating image 3 for 2D operation in accordance with the 2D operation mode. That is, the space floating image 3 for the 2D display described above is displayed as a display that encourages 2D operation, for example.

[0377] In step S3, the image processing unit displays the space floating image 3 for 3D operation in accordance with the 3D operation mode. That is, the space floating image 3 for the 3D display described above is displayed as, for example, a display that prompts 3D operation.

[0378] After step S2 or step S3, in step S4, the image processing unit uses the above-mentioned user operation detection mechanism to determine and detect an aerial operation by the user 230 on the floating-in-space image 3. For example, in the 2D operation mode, the image processing unit uses an algorithm / table corresponding to the 2D operation mode to determine and detect a 2D operation. In the 3D operation mode, the image processing unit uses an algorithm / table corresponding to the 3D operation mode to determine and detect a 2D operation or a 3D operation as the aerial operation that is the user operation. If the result of this determination / detection is a valid operation, in other words, an accepted operation or an allowed operation, the process proceeds to step S6, and if the result is an invalid operation, in other words, an unacceptable operation or an unallowable operation, the process proceeds to step S5.

[0379] In step S5, the image processing unit outputs feedback to the user 230 to the effect that the operation is invalid. This may be, for example, a feedback display on the screen of the floating-in-space image 3, or a feedback audio output. The process returns from step S5 to step S4.

[0380] In step S6, the image processing unit executes processing according to the valid user operation, that is, the display control processing set in association with the mid-air operation and object as described above. For example, if the 3D operation is a push operation on a button object, the display of the floating image 3 in space is updated so that the button object appears to be pushed in at a position further back in the depth direction (described later). As in the flow of steps S4 and S6, the basic principle is that processing according to the user operation is executed immediately.

[0381] Furthermore, in step S7, the image processing unit outputs feedback to the user 230 that the operation is valid. For example, the feedback may be displayed on the screen of the floating image 3 in space, or may be output as a feedback sound.

[0382] The feedback output in step S7 or step S5 may be an output defined for each object or aerial operation. Examples of feedback output include an effect display indicating that the aerial operation has been accepted as a valid aerial operation, or an effect display indicating that the aerial operation has not been accepted as an invalid aerial operation (described below). It is desirable that the processes of steps S6 and S7 are both executed promptly after it is determined in step S4 that a valid user operation has been performed. In this processing flow, step S7 is executed after step S6, but this is just an example, and the order of steps S6 and S7 may be reversed if priority is given to the feedback output to the user 230.

[0383] In step S8, the image processing unit checks and determines whether to switch the operation mode or change the operation mode settings while displaying the 2D or 3D space floating image 3. If it is determined that the operation mode should be changed, the process proceeds to step S9, and if it is determined that the operation mode should not be changed, the process proceeds to step S11.

[0384] Examples of this confirmation and judgment include the following: 1. As in #3 of Fig. 25, a mode designation operation input by the user 230 (described later). 2. As in #5 of Fig. 25, an automatic judgment by the space floating image display device 1000. For example, a judgment according to the user's operation situation (Example 4 described later).

[0385] In step S9, the video processing unit changes the operation mode setting: from 2D operation mode to 3D operation mode in the case of the flow from step S2, and from 3D operation mode to 2D operation mode in the case of the flow from step S3.

[0386] In the next step S10, the image processor displays the floating image 3 in space for 2D operation or 3D operation as a display mode corresponding to the changed operation mode. In step S11, the process waits for the next user operation, and the same process is repeated thereafter.

[0387] [Examples of 2D Operation and 2D Display] Fig. 34 shows a swipe operation as an example of a planar operation, which is a 2D operation on an object in the floating-in-space image 3, and an example of the corresponding 2D display. (1A) is a front view of the xy plane of the floating-in-space image 3, with a button object 3401 displayed in the center position (x coordinate is x1, y coordinate is y1). This button object 3401 is an object that accepts a swipe operation, in other words, a slide operation. For ease of explanation, a case is shown in which the text image "Slide" is displayed on the button object 3401, but there are also cases in which such a guide display is not displayed.

[0388] (1B) is a y-z plane view of the state of (1A) viewed from the side. The user 230 is touching a location 3403 where a button object 3401 is located in the x-y plane with a finger 3402, particularly one finger. At this time, the finger 3402 is positioned, for example, at position (x1, y1, z1). For example, the figure shows a case where the same user operation detection mechanisms 1701 and 1702 as those in FIG. 17 are present.

[0389] (2A) shows a state when the user 230 swipes the button object 3401, for example, to the right (x+ direction) from the state of (1A). (2B) is a y-z plan view of the state of (2A) viewed from the side. The user 230 is moving the finger 3402 to the right (x+ direction). The finger 3402b is located at a position (x2, y1, z1) corresponding to the location 3404, for example. Note that the finger may move to some extent in the z direction during the swipe operation. In response to detection of the planar operation, which is the swipe operation, using the user detection mechanism 1701, the image processing unit moves the button object 3401 to the right and displays it as button object 3401b.

[0390] When the video processing unit receives a swipe operation on the button object 3401 as described above, since the swipe operation is a 2D operation, the video processing unit switches to the 2D operation mode and displays the button object 3401 in a 2D display mode as shown in the figure. This makes it possible to prompt the user 230 to perform a 2D operation on the button object 3401. Furthermore, in the 2D operation mode, high-speed detection is possible using the user detection mechanism 1701, as described above.

[0391] [Examples of 3D operation and 3D display] Fig. 35 shows an example of a push operation as an example of a 3D operation on an object of the floating in space image 3, and an example of the corresponding 3D display. (1A) is a front view of the xy plane of the floating in space image 3, with a button object 3501 displayed at the center position (x coordinate is x1, y coordinate is y1). This button object 3501 is an object that accepts a push operation. For ease of explanation, a case is shown in which the text image "Push" is displayed on the button object 3501, but there are also cases in which such a guide display is not displayed.

[0392] 1B is a y-z plane view of the state of 1A viewed from the side. The user 230 is touching a location 3503 where a button object 3501 is located in the x-y plane with a finger 3502, particularly one finger. At this time, the finger 3502 is positioned, for example, at position (x1, y1, z1). For example, the figure illustrates a case where the same user operation detection mechanisms 1701 and 1702 as those in FIG. 17 are present.

[0393] (2A) shows a state when, from the state of (1A), the user 230 performs an operation to push the button object 3501 backward in the z direction. The user 230 moves the finger 3502, particularly one finger, backward in the z+ direction. As a result, the finger 3502b after the movement is positioned at, for example, a position (x1, y1, z2) corresponding to the position 3503. Note that the finger may move to some extent in the x-y direction during the pushing operation.

[0394] The video processing unit detects 3D operations including the depth operation, which is the above-mentioned pressing operation, using the user detection mechanisms 1701 and 1702. In response to the detection, the video processing unit displays the button object 3501 in a state where it appears as if it has moved further back as seen by the user 230, as shown by button object 3501b. For example, the button object 3501b has a reduced size and a changed color / brightness compared to the button object 3501 in (1A). This creates a visual effect where the button object 3501 appears as if it has moved further back as seen by the user 230.

[0395] When the video processing unit receives a push operation on the button object 3501 as described above, the push operation is a 3D operation, so the video processing unit switches to the 3D operation mode and displays the button object 3501 in a 3D display mode. In this example, the button object 3501 is a rectangular image shaded in the lower right direction, and the text image "Push" is also shaded. This allows the user 230 to be prompted to perform a 3D operation on the button object 3501.

[0396] As in the above example, by applying a suitable 2D display / 3D display depending on the 2D operation / 3D operation, the operability for the user 230 can be improved.

[0397] [Examples of Feedback Display] Fig. 36 shows examples of feedback displays regarding the aforementioned invalid operations. In example (1), similar to Fig. 34 , feedback display indicating invalid is displayed when the user 230 performs a depth operation on object 3601 that accepts, for example, a swipe operation as a 2D operation. In this invalid display, an X mark is displayed on object 3601, and a text image saying "not accepted" is displayed. In example (2), similar to Fig. 35 , feedback display indicating invalid is displayed when the user 230 performs a planar operation on object 3602 that accepts, for example, a press operation as a 3D operation. In this invalid display, an X mark is displayed on object 3602, and a text image saying "not accepted."

[0398] [3-7. Mode Information Output Function] Furthermore, in the third embodiment, the following function may be additionally provided: This function outputs to the user 230 information indicating the current mode status and its switching regarding the operation and display of the floating-in-space image 3.

[0399] One problem is that the effective operations by the user differ between a state where only 2D operations are accepted and a state where 3D operations are also accepted. This state and its state changes are difficult for the user to see / understand, so there is a possibility that this state and the actual user operations will not match, and the user may feel that the operability is poor.

[0400] As a means for solving this problem, the third embodiment has a mode information output function as follows: The mode information output function includes the following functions, and it is sufficient to apply at least one of them.

[0401] 1. When switching between 2D operation and 3D operation, the floating-in-space image display device 1000 outputs a notification sound to inform the user of the current mode or the new mode from speakers or the like installed inside or outside the housing 1190. This sound may be, for example, a short sound, a long sound, a beep, or a natural language voice.

[0402] 2. When switching between 2D operation and 3D operation, the space-floating image display device 1000 displays an image such as an indicator notation indicating the current mode or the mode after switching within the display range of the space-floating image 3, separate from the original display content of the space-floating image 3.

[0403] 3. When switching between 2D operation and 3D operation, the space floating image display device 1000 displays an image indicating that the operation mode will be switched for a predetermined number of seconds or until an operation is performed by the user 230.

[0404] Further, detailed examples of the function 2 above include the following. 2-1. An indicator display unit showing the current mode is provided at a predetermined position within the display range of the floating image 3, and an image showing the current mode, etc. is displayed on the indicator display unit as appropriate. 2-2. The indicator display unit is provided at, for example, the top, bottom, left, or right end within the display range, but is not limited to this and may be displayed anywhere within the display range. Furthermore, the position of the indicator display unit may be changed as appropriate, for example, depending on the GUI or content to be displayed.

[0405] [Mode Information Output Example] FIG. 37 shows an example of an indicator display on the space-floating image 3 as a configuration example related to the mode information output function. The upper part of FIG. 37 shows a perspective view of the vicinity of the opening 3700 on the top surface of the housing 1190 of the space-floating image display device 1000 corresponding to the configuration example shown in FIG. 24 and the like, and the space-floating image 3. The lower part of FIG. 37 shows an example of the display of the space-floating image 3 on the x-y plane. The opening 3700 is provided with the transparent member 100 described above. The user 230 can easily view the space-floating image 3 from the direction indicated by arrow A, which corresponds to the Y and Z directions. The space-floating image 3 has a predetermined display range 3R, which in this example is a roughly square area. In this example, the space-floating image 3 also displays a frame image 3709 representing the outline corresponding to the display range 3R. The display range 3R includes a main content display section 3701. In this example, a character image 3702 is displayed in the main content display section 3701. Character video 3702 is an example of a 3D display.

[0406] An indicator display unit 3703 is provided in a predetermined portion of the display range 3R, for example, in the upper right position. In this example, an image 3704 indicating that the device is in 3D mode, which supports 3D operation and 3D display, is displayed on the indicator display unit 3703. This image 3704 is a three-dimensional character image with shading added to the character "3D."

[0407] The image displayed on the indicator display unit 3703 may be any image that makes it easy for the user 230 to recognize whether the current mode is 2D operation mode, 3D operation mode, etc., and may be, for example, any of letters, figures, symbols, icons, still images, videos, character representations, etc.

[0408] When the image 3704 indicating the current mode is displayed on the indicator display unit 3703, it may be displayed constantly or temporarily. An example of a temporary display may be a display for a fixed period of time when a certain mode is started, or a display for a fixed period of time when the mode is switched. In another example, it may be displayed for a fixed period of time when the user 230 performs some operation, for example, when it is detected that the plane of the floating image 3 in space has been touched.

[0409] [Example of indicator display image] Fig. 38 shows, in a table, a display example of image 3704 as an indicator in indicator display section 3703 of display range 3R of the above-mentioned floating-in-space image 3. Each row indicates whether the indicator display content is text or a graphic. Two columns indicate whether the mode is 2D operation mode or 3D operation mode. For example, in the first row, in 2D operation mode, the text "2D" is displayed as image 3801. In 3D operation mode, the text "3D" is displayed as image 3802. In the second row, in 2D operation mode, a triangle is displayed as image 3803 as a planar graphic. In 3D operation mode, a cube is displayed as image 3804 as a three-dimensional graphic.

[0410] In another example, as shown in the third line, in the 3D operation mode, the letters "3D" may be displayed as image 3806. Image 3806 is a letter image that is based on image 3802 and has depth and shading added to it through image processing such as depth processing or shading addition processing. Also, as shown in the fourth line, in the 3D operation mode, a triangle may be displayed as image 3808 as a planar figure. Image 3808 is based on image 3803 and has depth and shading added to it through image processing such as depth processing or shading addition processing.

[0411] When displaying an image with a sense of depth such as image 3806 or image 3808, in other words, a 3D image / pseudo-3D image, such an image may be prepared and set in advance, or may be generated from a base 2D image by image processing or the like. Such processing may include, for example, a stereoscopic effect by changing the background color, or the addition of shading. Similarly, a 2D image / pseudo-2D image may be generated from a base 3D image by image processing or the like.

[0412] As in the above example, when the current mode is the 3D operation mode, it is preferable to make the indicator display image an image with a 3D / pseudo-3D representation that gives a sense of depth so that the fact that 3D operations are accepted is easily conveyed to the user 230. Conversely, when the current mode is the 2D operation mode, it is preferable to make the indicator display image an image with a 2D / pseudo-2D representation that gives a sense of depth so that the fact that only 2D operations are accepted is easily conveyed to the user 230.

[0413] [Audio Output Example] Figure 39 shows an example of one mode information output function in which, when switching between 2D operation and 3D operation, a notification sound is output from a speaker 3900 installed inside or outside the housing 1190 to inform the user of the current mode or the mode after the switch. The speaker 3900 corresponds to the audio output unit 1140 in Figure 3. The space-floating image display device 1000 performs audio output 3901, for example, when starting 2D operation mode or when switching to 2D operation mode. The audio output 3901 is, for example, audio such as "You are in 2D operation mode." The space-floating image display device 1000 performs audio output 3902, for example, when starting 3D operation mode or when switching to 3D operation mode. The audio output 3902 is, for example, audio such as "You are in 3D operation mode."

[0414] [Another Example of Indicator Display: Superimposed Display] Fig. 40 is an explanatory diagram showing superimposed display as another example of indicator display in the Space Floating Image 3. At the top of Fig. 40, in the xy plane of the display range 3R of the Space Floating Image 3, a first primary content 4001 and a second primary content 4002 are displayed in the primary content display section 3701, and an indicator display section 3703 is located at the top right. This example shows a state in which no indicator image is displayed in the indicator display section 3703. In this state, a part of the character image, which is the first primary content 4001, is superimposed within the indicator display section 3703. The superimposed section 4003 is the portion where the first primary content 4001 and the indicator display section 3703 are superimposed.

[0415] 40 shows three display examples (A), (B), and (C) relating to the image display of the indicator display portion 3703. When the superimposed portion 4003 occurs, any of these display examples can be applied.

[0416] The first example of (A) is an example in which a content portion 4004 corresponding to the superimposed portion 4003 is displayed as a lower layer, and an indicator image 4005 indicating the current mode is superimposed on top of it. In the first example, part of the content portion 4004 is displayed hidden by the indicator image 4005. This example is suitable when priority is given to conveying the current mode.

[0417] The second example of (B) is an example in which a content portion 4004 corresponding to the superimposition portion 4003 is displayed as an upper layer, and an indicator image 4005 is superimposed on the lower layer. In the second example, part of the indicator image 4005 is displayed hidden by part of the content portion 4004. This example is suitable when priority is given to displaying the main content video.

[0418] In the third example (C), indicator image 4005 is displayed as image 4005C, which has been subjected to semi-transparency processing, and is superimposed on content portion 4004 corresponding to superimposition portion 4003. In the third example, image 4005C is displayed so that part of content portion 4004 can be seen through to the back of image 4005C. When the indicator display image and the main content video overlap in this way, one or both of the indicator display image and the main content video, or part of either of them, may be displayed after being subjected to image processing so as to appear semi-transparent.

[0419] By applying the above display example, even if the indicator display overlaps the main image such as the content of the space floating image 3 or the GUI, it is possible to display it appropriately.

[0420] [Another example of indicator display: mode name] Fig. 41A shows another example of indicator display in the floating-in-space image 3 related to mode information display, where a character image of the mode name, an icon, etc. is displayed. In the example (1) of Fig. 41, an indicator display unit 4103 is provided in the upper side region of the display range 3R. In the floating-in-space image 3, for example, a button object 4101 of 2D display is displayed. In the indicator display unit 4103, a character image of "2D operation mode" is displayed. In addition, in the indicator display unit 4103, an arrow image 4102 representing the 2D operation mode is displayed. This arrow image 4102 is an icon image that expresses two dimensions in the x-y directions.

[0421] In the example of (2), the floating-in-space image 3 displays, for example, a 3D button object 4104. A text image of "3D operation mode" is displayed in the indicator display portion 4103. Also, an arrow image 4105 representing the 3D operation mode is displayed in the indicator display portion 4103. This arrow image 4105 is an icon image that represents three dimensions in the x, y, and z directions.

[0422] [Modification: Mode Selection Display] FIG. 41B shows a display example of a space-floating image 3 in a modification. While a main image such as a GUI or content is displayed in the display range 3R of the space-floating image 3, a GUI may be displayed to allow the user 230 to select and switch operation modes. In the example (1) of FIG. 41B, initially, in 3D operation mode, a 3D display object 4111 is displayed, and a "3D" image indicating the current 3D operation mode is displayed in the indicator display portion 3703 in the upper right. Furthermore, a 2D operation button 4112 that allows the user 230 to select and switch operation modes is displayed at the bottom of the display range 3R. If the user 230 wants to perform 2D operation, in other words, if they want to change to 2D operation mode, they touch the 2D operation button 4112. In response to this operation, the space-floating image display device 1000 switches to 2D operation mode. In response to the switch, the display of the space-floating image 3 is also updated to the 2D display mode.

[0423] (2) shows another example. When the user 230 wants to perform 2D operation, in other words, when he wants to change to 2D operation mode, he touches the "3D" image on the indicator display unit 3703. In response to this operation, the space-floating image display device 1000 switches to 2D operation mode. In response to the switch, the display of the space-floating image 3 is also updated to a 2D display mode. In this example, the image on the indicator display unit 3703 also functions to switch modes.

[0424] (3) shows another example. In the 3D operation mode, for example, a 3D displayed object 4111 is displayed on the floating-in-space image 3. The 3D operation mode accepts both 2D and 3D operations. In this example, a 2D operation button 4112 and a 3D operation button 4113 are displayed at the bottom of the display range 3R. If the user 230 wants to perform a 2D operation, he or she touches the 2D operation button 4112. In this case, the floating-in-space image display device 1000 applies an algorithm for 2D operation in response to the operation and waits for a 2D operation (planar operation). The user 230 performs a 2D operation, and the floating-in-space image display device 1000 detects the 2D operation. On the other hand, if the user 230 wants to perform a 3D operation, particularly a depth operation, he or she touches the 3D operation button 4113. In this case, in response to the operation, the space-floating image display device 1000 applies an algorithm for 3D operation and waits for 3D operation, especially depth operation. The user 230 performs a 3D operation, and the space-floating image display device 1000 detects the 3D operation. The 2D operation button 4112 may be an image representing a planar operation. The 3D operation button 4113 may be an image representing a depth operation.

[0425] <Fourth embodiment> As a fourth embodiment of the present invention, an example of the space floating image display device 1000 will be described. As the basic configuration of the space floating image display device 1000 of the fourth embodiment, the configuration of the third embodiment (for example, Fig. 17 or Fig. 24) can be similarly applied. In the fourth embodiment, differences from the third embodiment will be mainly described, and repeated explanations of similar configurations will be omitted.

[0426] In Example 4, the floating-in-space image display device 1000 has a function of estimating and determining whether the user 230 is attempting to perform a 2D operation or a 3D operation based on the detection status of the mid-air operation by the user 230 on the floating-in-space image 3, and automatically switching between the 2D operation mode and the 3D operation mode based on the determination. Example 4 can determine a suitable operation mode and a corresponding display mode based on the tendency and habits of the user 230's finger movements, and change to that mode. For example, for a user who is unfamiliar with mid-air operation, the 2D mode can be used to provide simple operation and display. For example, for a user who is accustomed to mid-air operation, the 3D mode can be used to provide more advanced operation and display.

[0427] 42A and 42B show configuration examples of modes prepared as options in Example 4. The table in Fig. 42A shows the first half, and the table in Fig. 42B shows the second half following Fig. 42A. Example 4 has four types of modes: 2D operation mode A (in other words, the first 2D operation mode), 2D operation mode B (in other words, the second 2D operation mode), 3D operation mode A (in other words, the first 3D operation mode), and 3D operation mode B (in other words, the second 3D operation mode).

[0428] (1) In 2D operation mode A, detection of 2D operations (for example, the aforementioned user operation detection mechanism 2201) is in the ON state, detection of 3D operations (for example, the aforementioned user operation detection mechanism 2301) is in the OFF state, and detectable user operations are 2D operations. Images of GUI, content, etc. displayed in 2D operation mode A are 2D display images. In other words, in 2D operation mode A, control is performed so that 2D display is achieved in accordance with 2D operations. User operations actually accepted and detected in 2D operation mode A, in other words, aerial operations that are the subject of display control processing in accordance with objects, etc., are 2D operations.

[0429] It is possible to change from 2D operation mode A to 3D operation mode A. A trigger for changing from 2D operation mode A to 3D operation mode A is, for example, mode selection by the user. For example, as described above, options for mode selection and switching are displayed on the screen of the floating-in-space image 3, and the user may select a mode, or may operate a physical button. The image displayed in the changed 3D operation mode A is a 3D display image. The image display method in the changed 3D operation mode A is, for example, a method of erasing the 2D image and displaying the 3D image.

[0430] (3) In 3D operation mode A, 2D operation detection is ON and 3D operation detection is ON, and detectable user operations are 2D operations and 3D operations. Image...

Claims

1. A floating-in-the-air image display device comprising: an image processing unit; a display unit that displays an image that has been image-processed by the image processing unit; an optical system that generates a floating-in-the-air image based on the image displayed by the display unit; and a detection mechanism that detects aerial operations by a user on the floating-in-the-air image, wherein the device has a 2D operation mode that accepts planar operations on the plane of the floating-in-the-air image as the aerial operations, and in the 2D operation mode, displays an image on the floating-in-the-air image that prompts the planar operations.

2. A floating-in-the-air image display device comprising: an image processing unit; a display unit that displays an image that has been image-processed by the image processing unit; an optical system that generates a floating-in-the-air image based on the image displayed by the display unit; and a detection mechanism that detects aerial operations by a user on the floating-in-the-air image, wherein the device has a 3D operation mode that accepts depth operations on the plane of the floating-in-the-air image as the aerial operations, and in the 3D operation mode, displays an image on the floating-in-the-air image that prompts the depth operations.

3. A floating-in-the-air image display device comprising: an image processing unit; a display unit that displays an image that has been image-processed by the image processing unit; an optical system that generates a floating-in-the-air image based on the image displayed by the display unit; and a detection mechanism that detects aerial operations by a user on the floating-in-the-air image, the device having a 2D operation mode that accepts planar operations on the plane of the floating-in-the-air image as the aerial operations, and a 3D operation mode that accepts depth operations on the plane of the floating-in-the-air image as the aerial operations, the device switching between the 2D operation mode and the 3D operation mode, and displaying an image on the floating-in-the-air image that prompts the planar operations in the 2D operation mode, and displaying an image on the floating-in-the-air image that prompts the depth operations in the 3D operation mode.

4. In the air-floating image display device according to claim 3, the selection and switching between the 2D operation mode and the 3D operation mode is performed based on any of the following triggers: A. setting of the air-floating image display device, B. setting of an application provided in the air-floating image display device, C. mode designation by the user, D. attribute or mode designation for each image to be displayed, Air-floating image display device.

5. A floating-in-the-air image display device as claimed in claim 3, wherein when the planar operation is accepted as the aerial operation, the floating-in-the-air image is controlled to be displayed as a 2D image, and when the depth operation is accepted as the aerial operation, the floating-in-the-air image is controlled to be displayed as a 3D image.

6. A floating-in-the-air image display device as claimed in claim 3, wherein when a 2D image is displayed on the floating-in-the-air image, the device is controlled to accept the planar operation as the aerial operation, and when a 3D image is displayed on the floating-in-the-air image, the device is controlled to accept the depth operation as the aerial operation.

7. A floating-in-the-air image display device as claimed in claim 3, wherein said floating-in-the-air image display device or an external device connected to said floating-in-the-air image display device describes 2D or 3D display attributes, 2D or 3D operation attributes or mode designation for the image to be displayed as the floating-in-the-air image, and selects between said 2D operation mode and said 3D operation mode based on said description.

8. A floating-in-the-air image display device according to claim 3, wherein the image that prompts planar operation is an image with a planar figure, no shading, or an image that shows movement in a planar direction, and the image that prompts depth operation is an image with a three-dimensional figure, a shading, or an image that shows movement in a depth direction.

9. A floating-in-the-air image display device according to claim 3, further comprising an indicator display provided in a portion of the display range of the floating-in-the-air image, the indicator display indicating whether the current mode is the 2D operation mode or the 3D operation mode.

10. A floating-in-the-air image display device according to claim 9, wherein the indicator display is a 2D image when in the 2D operation mode, and is a 3D image when in the 3D operation mode.

11. A floating-in-the-air image display device according to claim 9, wherein, when a part of the main image displayed in the display range of the floating-in-the-air image overlaps with an image of the indicator display, the indicator display is displayed in any one of the following ways: A. The image of the indicator display is displayed superimposed on top of the part of the main image, B. The image of the indicator display is displayed superimposed on top of the image of the indicator display, C. The image of the indicator display is displayed semi-transparently on top of the part of the main image.

12. A floating-in-the-air image display device according to claim 3, further comprising: an image for designating the 2D operation mode or the 3D operation mode displayed on the floating-in-the-air image; and when the 2D operation mode or the 3D operation mode is designated based on the image, the device switches to the designated 2D operation mode or the 3D operation mode.

13. A floating-in-the-air image display device comprising: an image processing unit; a display unit that displays an image that has been image-processed by the image processing unit; an optical system that generates a floating-in-the-air image based on the image displayed by the display unit; and a detection mechanism that detects a user's aerial operation on the floating-in-the-air image; and comprising: a 2D operation mode that accepts a planar operation on the plane of the floating-in-the-air image as the aerial operation; and a 3D operation mode that accepts a depth operation on the plane of the floating-in-the-air image as the aerial operation, and switches between the 2D operation mode and the 3D operation mode, the 2D operation mode being a mode in which the planar operation and the depth operation can be detected during processing, and the 3D operation mode being a mode in which the planar operation and the depth operation can be detected during processing, and while the floating-in-the-air image is being displayed, a determination is made based on a status of detection of the user's aerial operation whether to switch between the 2D operation mode and the 3D operation mode, and based on the determination, switches between the 2D operation mode and the 3D operation mode.

14. A floating-in-the-air image display device according to claim 13, wherein in the 2D operation mode, an image prompting the planar operation is displayed on the floating-in-the-air image, and in the 3D operation mode, an image prompting the depth operation is displayed on the floating-in-the-air image.

15. A floating-in-the-air image display device as claimed in claim 2, wherein the 3D operation mode is a mode in which both planar operation on the plane of the floating-in-the-air image and the depth operation are accepted as the aerial operation, and in the 3D operation mode, when the planar operation is prompted, an image prompting the planar operation is displayed on the floating-in-the-air image, and when the depth operation is prompted, an image prompting the depth operation is displayed on the floating-in-the-air image.

Images