Levitation image display device

Abstract

To provide a more desirable levitation image display device than before, so as to contribute to "3. Good Health and Well-Being ", "9. Industry, Innovation, and Infrastructure ", and "11. Sustainable Cities and Communities" of the sustainable development goals (SDGs).SOLUTION: A levitation image display device includes an image processing part, a display part, and an optical system. The image processing part generates a shadow image on the assumption that light from a real light source is applied to an object image in a levitation image in an environment in which the levitation image display device is arranged, so as to make the levitation image in which the shadow image is given to the object image displayed.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present invention relates to a floating-in-the-air image display device. [Background technology]

[0002] The floating information display technology is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-128722 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the disclosure of Patent Document 1 does not sufficiently consider a configuration for obtaining practical brightness and quality for the floating image, or a configuration for allowing the user to view the floating image more enjoyably.

[0005] An object of the present invention is to provide a more suitable floating-in-the-air image display device. [Means for solving the problem]

[0006] In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problems, but one example is as follows: A floating-in-the-air image display device, comprising: an image processing unit that processes images; a display unit that displays images that have been processed by the image processing unit; and an optical system that generates a floating-in-the-air image based on the image displayed by the display unit, wherein the image processing unit generates a shadow image that would appear if light from a real light source were irradiated onto an object image in the floating-in-the-air image in an environment in which the floating-in-the-air image display device is installed, and displays the floating-in-the-air image with the shadow image added to the object image. [Effects of the Invention]

[0007] According to the present invention, a more suitable floating-in-the-air image display device can be realized. Other problems, configurations, and effects will become clear in the following description of the embodiments. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of a usage form of a space floating image display device according to an embodiment of the present invention; [Figure 2A] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 2B] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 2C] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 2D] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a floating-in-the-air image display device according to an embodiment of the present invention; [Figure 2E] 1 is a projection view of a retroreflector constituting a floating-in-the-air image display device according to an embodiment of the present invention; [Figure 2F] 1 is a top view of a retroreflector constituting a floating-in-the-air image display device according to an embodiment of the present invention; FIG. [Figure 2G] FIG. 1 is a perspective view showing a corner reflector that constitutes a retroreflector that constitutes a floating-in-the-air image display device according to an embodiment of the present invention. [Figure 2H] FIG. 1 is a top view showing a corner reflector that constitutes a retroreflector that constitutes a floating-in-the-air image display device according to an embodiment of the present invention. [Figure 2I] FIG. 1 is a side view showing a corner reflector that constitutes a retroreflector that constitutes a floating-in-the-air image display device according to an embodiment of the present invention. [Figure 3] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4A] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4B] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4C] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4D] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4E] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4F] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4G] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4H] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4I] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4J] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4K] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4L] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4M] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4N] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 4O] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 5] 1 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. [Figure 6]1 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. [Figure 7] 1 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. [Figure 8] 1 is a layout diagram showing a main part of a space floating image display device according to an embodiment of the present invention; [Figure 9] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 10] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 11] 1 is an explanatory diagram for explaining the light source diffusion characteristics of an image display device according to an embodiment of the present invention. [Figure 12] 1 is an explanatory diagram for explaining the diffusion characteristics of a video display device according to an embodiment of the present invention; [Figure 13A] 1 is a diagram illustrating an example of a problem to be solved by image processing according to an embodiment of the present invention; [Figure 13B] FIG. 10 is an explanatory diagram of an example of image processing according to an embodiment of the present invention. [Figure 13C] FIG. 10 is an explanatory diagram of an example of a video display process according to an embodiment of the present invention. [Figure 13D] FIG. 10 is an explanatory diagram of an example of a video display process according to an embodiment of the present invention. [Figure 14] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 15] 1 is a diagram showing the configuration of a display system including a space floating image display device according to an embodiment of the present invention; [Figure 16] 1 is a diagram illustrating an example of the configuration of a space floating image display device and an external device according to an embodiment of the present invention; [Figure 17] FIG. 10 is a diagram showing a calculated virtual three-dimensional space and a screen display of a space floating image in a user environment according to a comparative example. [Figure 18] FIG. 10 is a diagram showing a mismatch between a shadow in a calculated virtual three-dimensional space and a shadow in a device installation environment in the real world, according to a comparative example. [Figure 19]FIG. 10 is a diagram showing the superimposition of a shadow in a calculated virtual three-dimensional space and a real light source in the device installation environment, according to a comparative example. [Figure 20] FIG. 10 is a diagram illustrating the correspondence between shadows in a calculated virtual three-dimensional space and shadows in a real-world device installation environment, according to one embodiment of the present invention. [Figure 21] FIG. 10 is a diagram illustrating the superimposition of a shadow in a calculated virtual three-dimensional space with a real light source in the device installation environment, according to one embodiment of the present invention. [Figure 22] FIG. 10 is a diagram illustrating an example of a configuration in a case where there is no virtual light source information according to an embodiment of the present invention. [Figure 23] 1A and 1B are diagrams illustrating an object of a 3D model and an image of the object to which a shadow has been generated and added, according to one embodiment of the present invention. [Figure 24] FIG. 10 is a table summarizing control patterns according to one embodiment of the present invention. [Figure 25] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 26] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 27] 1 is a diagram showing an example of the configuration of a space floating image display device according to an embodiment of the present invention; [Figure 28] FIG. 1 is a diagram illustrating an example of the configuration of a sensor according to an embodiment of the present invention. [Figure 29] FIG. 1 is a diagram illustrating an example of the configuration of a sensor according to an embodiment of the present invention. [Figure 30] FIG. 1 is a diagram illustrating an example of the configuration of a sensor according to an embodiment of the present invention. [Figure 31] FIG. 10 is a diagram showing an example of an image captured by a sensor according to one embodiment of the present invention. [Figure 32] FIG. 10 is a diagram showing an example of an image captured by a sensor according to one embodiment of the present invention. [Figure 33] FIG. 10 is a diagram showing a processing flow of a space floating image display device according to an embodiment of the present invention. [Figure 34] FIG. 10 is a diagram showing an example of the configuration of a space floating image display device when a preset shadow image is used, according to an embodiment of the present invention. [Figure 35] FIG. 10 is a diagram illustrating an example of processing when a preset shadow image is used according to an embodiment of the present invention. [Figure 36] 10A and 10B are diagrams illustrating an example of generating a shadow according to the positional relationship of a real light source with respect to an object image in a space floating image on a stage, according to one embodiment of the present invention. [Figure 37] FIG. 10 is a diagram showing an example of the arrangement of floating images on a stage according to one embodiment of the present invention. [Figure 38] FIG. 10 is a diagram showing an example of the arrangement of floating images on a stage according to one embodiment of the present invention. [Figure 39] 10A to 10C are diagrams illustrating an example of generation and display of a shadow according to the positional relationship of a real light source with respect to a space floating image, according to one embodiment of the present invention. [Figure 40] 10A to 10C are diagrams illustrating an example of generation and display of a shadow according to the positional relationship of a real light source with respect to a space floating image, according to one embodiment of the present invention. [Figure 41] 10A and 10B are diagrams illustrating an example of adjusting an object image and a shadow on a screen of a floating image in space according to one embodiment of the present invention. [Figure 42] 10A to 10C are diagrams illustrating an example of generation and display of a shadow according to the positional relationship of a real light source with respect to a space floating image, according to one embodiment of the present invention. [Figure 43] 10A and 10B are diagrams illustrating an example of a relationship between a floating image in space and a background according to one embodiment of the present invention. [Figure 44] FIG. 1 is a diagram illustrating an example of a configuration for cooperation with home appliances in a user environment according to an embodiment of the present invention. [Figure 45] FIG. 10 is a diagram illustrating an example of user settings on a screen of a floating image in space according to one embodiment of the present invention. [Figure 46] FIG. 10 illustrates the generation of shading on an object surface in one control pattern according to one embodiment of the present invention. [Figure 47] FIG. 10 is a diagram illustrating the generation of shadows on a floor surface in one control pattern according to one embodiment of the present invention. [Figure 48] FIG. 10 is a diagram illustrating the generation of shadows on a floor surface in one control pattern according to one embodiment of the present invention. [Figure 49] 10A and 10B are diagrams showing examples of generation and display of shadows of main content and sub-content according to one embodiment of the present invention. [Figure 50] 10A and 10B are diagrams showing examples of generation and display of shadows of main content and sub-content according to one embodiment of the present invention. [Figure 51] 1 is a diagram showing the configuration of a display system including a space floating image display device according to an embodiment of the present invention; [Figure 52] 10A and 10B are diagrams illustrating an example of displaying an image with wind expression / action according to actual wind conditions within a floating image in space according to one embodiment of the present invention. [Figure 53] 10A and 10B are diagrams illustrating an example of displaying an image with wind expression / action according to actual wind conditions within a floating image in space according to one embodiment of the present invention. [Figure 54A] FIG. 2 is a diagram illustrating an example of the configuration of a sensor for detecting wind according to an embodiment of the present invention. [Figure 54B] FIG. 2 is a diagram illustrating an example of the configuration of a sensor for detecting wind according to an embodiment of the present invention. [Figure 55] FIG. 10 is a diagram illustrating an example of user settings on a screen of a floating image in space according to one embodiment of the present invention. [Figure 56] FIG. 1 is a diagram illustrating an example of a configuration for cooperation with home appliances in a user environment according to an embodiment of the present invention. [Figure 57] 10A and 10B are diagrams showing an example of a display of wind in a floating image in space according to one embodiment of the present invention; [Figure 58] 10A and 10B are diagrams showing an example of displaying an object image and a shadow in a space floating image according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, components having the same functions are given the same reference numerals, and repeated explanations thereof may be omitted.

[0010] The following examples relate to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member. In the following explanation of the examples, the image floating in space is expressed using the term "floating image in space." Instead of this term, it is also acceptable to express it as "aerial image," "spatial image," "floating image in space," "floating optical image of displayed image," "floating optical image of displayed image," etc. The term "floating image in space," which is mainly used in the explanation of the examples, is used as a representative example of these terms.

[0011] According to the following embodiments, an image display device suitable for, for example, bank ATMs, train station ticket machines, digital signage, and the like can be realized. For example, currently, bank ATMs, train station ticket machines, and the like typically use touch panels. However, by using a transparent glass surface or a light-transmitting plate, high-resolution image information can be displayed in a floating state on the glass surface or light-transmitting plate. In this case, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflector. This improves light utilization efficiency and suppresses the ghost images that occur in addition to the main floating image, which is a problem with conventional retroreflection systems, thereby achieving a clear floating image. Furthermore, by using a device including the light source of this embodiment, a novel and highly usable floating image display device (floating image display system) can be provided that can significantly reduce power consumption. Furthermore, a floating image display device for a vehicle can be provided that can display a so-called unidirectional floating image that can be viewed inside and / or outside the vehicle. Example 1

[0012] <Example of how to use the space floating image display device> FIG. 1 is a diagram showing an example of a usage form of a space-floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using FIG. 2 and other figures. Light with a narrow-angle directional characteristic and specific polarization is emitted from image display device 1 as an image light beam, reflected by the optical system within the space-floating image display device, and then incident on retroreflector 2. It is then retroreflected and transmitted through transparent member 100 (glass, etc.), forming a real aerial image (space-floating image 3) on the outside of the glass surface. In the following embodiments, the retroreflector 2 (retroreflector) is used as an example of a retroreflector. However, the retroreflector 2 of the present invention is not limited to a planar plate, and is used as an example of a concept including a sheet-like retroreflector attached to a planar or non-planar member, or an entire assembly in which a sheet-like retroreflector is attached to a planar or non-planar member. Furthermore, since the light rays reflected by the retroreflector 2 have the optical property of forming an image, the retroreflector 2 may also be expressed as an imaging optical member or an imaging optical plate.

[0013] In addition, in a store or the like, a space is partitioned by a show window (also called "window glass") 105, which is a translucent member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit such a transparent member and display a floating image in one direction to the outside and / or inside of the store (space).

[0014] 1, the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing a means for reflecting specific polarized waves on the window glass 105, it is possible to reflect the waves and form an aerial image at a desired position inside the store.

[0015] <Configuration example of optical system for space floating image display device> 2A is a diagram showing an example of the configuration of an optical system of a space-floating image display device according to one embodiment of the present invention. The configuration of the space-floating image display device will be described in more detail using FIG. 2A. As shown in FIG. 2A(1), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light with narrow-angle diffusion characteristics.

[0016] Image light of a specific polarization from the display device 1 is reflected by a polarization separator 101 (in the figure, the polarization separator 101 is formed into a sheet and adhered to the transparent member 100) that has a film that selectively reflects image light of a specific polarization and is provided on a transparent member 100, and then enters the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector 2. The image light passes through the λ / 4 plate 21 twice, once when it enters the retroreflector 2 and once when it exits, thereby undergoing polarization conversion from the specific polarization to the other polarization. Here, the polarization separator 101 that selectively reflects image light of a specific polarization has the property of transmitting the polarized light of the other polarization that has been polarization-converted, so the image light of the specific polarization after polarization conversion passes through the polarization separator 101. The image light that has passed through the polarization separator 101 forms a space-floating image 3, which is a real image, outside the transparent member 100. 2A shows an example in which the chief ray of the image light incident on the retroreflector 2 is incident at an angle of 90° to the retroreflector 2. However, the incident angle of the chief ray of the image light on the retroreflector 2 is not limited to 90°, and an angle of, for example, 90°±15° can also be used.

[0017] Here, a first example of polarization design for the optical system of FIG. 2A will be described. For example, S-polarized image light may be emitted from display device 1 to polarization separator 101, which may have the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the S-polarized image light reaching polarization separator 101 from display device 1 is reflected by polarization separator 101 and travels toward retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, thereby converting the S-polarized image light to P-polarized light. The P-polarized image light then travels back toward polarization separator 101. Here, polarization separator 101 has the property of reflecting S-polarized light and transmitting P-polarized light, so the P-polarized image light passes through polarization separator 101 and then through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0018] Next, a second example of polarization design for the optical system of FIG. 2A will be described. For example, P-polarized image light may be emitted from display device 1 to polarization separator 101, which may have the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the P-polarized image light reaching polarization separator 101 from display device 1 is reflected by polarization separator 101 and travels toward retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, converting the P-polarized light to S-polarized light. The S-polarized image light then travels back toward polarization separator 101. Here, polarization separator 101 has the property of reflecting P-polarized light and transmitting S-polarized light, so the S-polarized image light passes through polarization separator 101 and then through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0019] The light that forms the floating image 3 is a collection of light rays that converge from the retroreflector 2 to the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is a highly directional image, unlike the diffused image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2A, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be perceived as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0020] Depending on the performance of the retroreflector 2, the polarization axis of the reflected image light may become irregular. The reflection angle may also become irregular. Such irregular light may not maintain the polarization state and propagation angle assumed in the design. For example, light with an unintended polarization state and propagation angle may re-enter the image display surface of the liquid crystal display panel 11 directly from the position of the retroreflector 2 without passing through a polarization separation member. Such light with an unintended polarization state and propagation angle may re-enter the image display surface of the liquid crystal display panel 11 after being reflected by components within the space-floating image display device. Such light re-entering the image display surface of the liquid crystal display panel 11 may be re-reflected by the image display surface of the liquid crystal display panel 11 constituting the display device 1, potentially generating ghost images and degrading the image quality of the space-floating image. Therefore, in this embodiment, an absorbing polarizer 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorptive polarizer 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorptive polarizer 12, thereby suppressing the re-reflection. This makes it possible to prevent degradation of image quality due to ghost images of spatially floating images. Specifically, if the display device 1 is configured to emit S-polarized image light to the polarization separation member 101, the absorptive polarizer 12 may be a polarizer that absorbs P-polarized light. Furthermore, if the display device 1 is configured to emit P-polarized image light to the polarization separation member 101, the absorptive polarizer 12 may be a polarizer that absorbs S-polarized light.

[0021] The polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

[0022] 2A(2) shows an example of the surface shape of a typical retroreflector 2. Light rays incident on the interior of the regularly arranged hexagonal prisms are reflected by the walls and bottoms of the hexagonal prisms and emitted as retroreflected light in a direction corresponding to the incident light, and a real floating image is displayed based on the image displayed on the display device 1.

[0023] The resolution of this floating image in space depends not only on the resolution of the liquid crystal display panel 11, but also on the outer diameter D and pitch P of the retroreflective portion of the retroreflector 2 shown in Figure 2A(2). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, one pixel of the floating image in space will be equivalent to 300 μm. As a result, the effective resolution of the floating image in space will be reduced to about one-third.

[0024] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire caused by the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Also, it is advisable to arrange the shape so that none of the sides of the retroreflective portion overlaps with any of the sides of one pixel of the liquid crystal display panel.

[0025] The surface shape of the retroreflector according to this embodiment is not limited to the above example. Various surface shapes that achieve retroreflection may be used. Specifically, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which triangular pyramidal prisms, hexagonal pyramidal prisms, other polygonal prisms, or a combination thereof are periodically arranged. Alternatively, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which these prisms are periodically arranged to form cube corners. These may also be referred to as corner reflector arrays or polyhedral reflector arrays. Alternatively, the surface of the retroreflector according to this embodiment may be provided with capsule lens-type retroreflection elements in which glass beads are periodically arranged. The detailed configuration of these retroreflection elements can be achieved using existing technology, so a detailed description will be omitted. Specifically, the techniques disclosed in Japanese Patent Laid-Open Nos. 2001-33609, 2001-264525, 2005-181555, 2008-70898, and 2009-229942 may be used.

[0026] <Another configuration example 1 of the optical system of the space floating image display device> Another example of the configuration of the optical system of the space floating image display device will be explained using Fig. 2B. In Fig. 2B, components with the same reference numerals as Fig. 2A have the same functions and configurations as Fig. 2A. For the sake of simplicity, repeated explanations of such components will be omitted.

[0027] In the optical system of FIG. 2B, as in FIG. 2A, image light of a specific polarization is output from the display device 1. The image light of a specific polarization output from the display device 1 is input to a polarization separator 101B. The polarization separator 101B is a member that selectively transmits image light of a specific polarization. Unlike the polarization separator 101 of FIG. 2A, the polarization separator 101B is not integrated with the transparent member 100 but has an independent plate-like shape. Therefore, the polarization separator 101B may also be referred to as a polarization separator plate. The polarization separator 101B may be configured as a reflective polarizer configured by attaching a polarization separator sheet to a transparent member. Alternatively, the transparent member may be formed of a metal multilayer film that selectively transmits specific polarization and reflects polarization of other specific polarizations. In FIG. 2B, the polarization separator 101B is configured to transmit image light of a specific polarization output from the display device 1.

[0028] The image light that has passed through the polarization separation member 101B is incident on the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector. The image light is polarized and converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when it enters the retroreflector and once when it leaves. Here, the polarization separation member 101B has the property of reflecting the polarized light of the other polarization that has been polarized and converted by the λ / 4 plate 21, so the image light after polarization conversion is reflected by the polarization separation member 101B. The image light reflected by the polarization separation member 101B passes through the transparent member 100 and forms a space-floating image 3, which is a real image, outside the transparent member 100.

[0029] Here, a first example of polarization design for the optical system of FIG. 2B will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from display device 1 to polarization separator 101B, and polarization separator 101B has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized image light that reaches polarization separator 101B from display device 1 passes through polarization separator 101B and proceeds to retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, and the image light is converted from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again to polarization separator 101B. Here, polarization separator 101B has the property of reflecting S-polarized light and transmitting P-polarized light, so the S-polarized image light is reflected by polarization separator 101 and passes through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a space-floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the space-floating image 3 to be formed optimally.

[0030] Next, a second example of polarization design for the optical system of FIG. 2B will be described. For example, S-polarized image light may be emitted from display device 1 to polarization separator 101B, which may have the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized image light reaching polarization separator 101B from display device 1 passes through polarization separator 101B and proceeds to retroreflector 2. When the image light is reflected by retroreflector 2, it passes twice through λ / 4 plate 21 provided on the incident surface of retroreflector 2, converting the image light from S-polarized light to P-polarized light. The P-polarized image light then proceeds again to polarization separator 101B. Here, polarization separator 101B has the property of reflecting P-polarized light and transmitting S-polarized light, so the P-polarized image light is reflected by polarization separator 101 and passes through transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a space-floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the space-floating image 3 to be formed optimally.

[0031] In FIG. 2B , the image display surface of the display device 1 and the surface of the retroreflector 2 are arranged parallel to each other. The polarization separator 101B is arranged tilted at an angle α (e.g., 30°) relative to the image display surface of the display device 1 and the surface of the retroreflector 2. When the polarization separator 101B reflects the image light, the direction of the image light reflected by the polarization separator 101B (the direction of the chief ray of the image light) differs by an angle β (e.g., 60°) from the direction of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light). With this configuration, the optical system of FIG. 2B outputs the image light toward the outside of the transparent member 100 at a predetermined angle shown in the figure, forming the space-floating image 3, which is a real image. In the configuration of FIG. 2B , when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be perceived as an image at all. This characteristic is extremely suitable for use in a system that displays images that require high security or highly confidential images that should be concealed from people directly facing the user.

[0032] As described above, the optical system of FIG. 2B has a different configuration from the optical system of FIG. 2A, but can form a suitable floating image in space, similar to the optical system of FIG. 2A.

[0033] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized waves of the image light from the polarization separation member 101B and absorb the polarized waves that are 90° out of phase with the polarized waves of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2B due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0034] <Another configuration example 2 of the optical system of the space floating image display device> Another example of the configuration of the optical system of the space floating image display device will be explained using Fig. 2C. In Fig. 2C, components with the same reference numerals as Fig. 2B have the same functions and configurations as Fig. 2B. For the sake of simplicity, such components will not be described repeatedly.

[0035] The only difference between the optical system in Figure 2B and the optical system in Figure 2C is the angle at which the polarization separation member 101B is disposed relative to the image display surface of the display device 1 and the surface of the retroreflector 2. All other configurations are the same as those of the optical system in Figure 2B, so repeated explanations will be omitted. The polarization design of the optical system in Figure 2C is also the same as that of the optical system in Figure 2B, so repeated explanations will be omitted.

[0036] In the optical system of FIG. 2C , the polarization separator 101B is tilted at an angle α with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. In FIG. 2C , the angle α is 45°. With this configuration, when the polarization separator 101B reflects, the angle β between the direction of propagation of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light) and the direction of propagation of the image light reflected by the polarization separator 101B (the direction of the chief ray of the image light) is 90°. With this configuration, the image display surface of the display device 1 and the surface of the retroreflector 2 are perpendicular to the direction of propagation of the image light reflected by the polarization separator 101B, simplifying the angular relationships of the surfaces that make up the optical system. By arranging the surface of the transparent member 100 so that it is perpendicular to the direction of propagation of the image light reflected by the polarization separator 101B, the angular relationships of the surfaces that make up the optical system can be further simplified. In the configuration of Figure 2C, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0037] As described above, the optical system of Fig. 2C has a different configuration from the optical systems of Fig. 2A and Fig. 2B, but can form a suitable floating image in space similar to the optical systems of Fig. 2A and Fig. 2B. In addition, the angles of the surfaces constituting the optical system can be made simpler.

[0038] An absorptive polarizer may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizer may transmit the polarized waves of the image light from the polarization separation member 101B and absorb the polarized waves that are 90° out of phase with the polarized waves of the image light from the polarization separation member 101B. This allows the image light for forming the space-floating image 3 to be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This allows the stray light in the optical system of FIG. 2C due to the external light incident on the space-floating image 3 side of the transparent member 100 to be reduced.

[0039] <Another configuration example 3 of the optical system of the space floating image display device> Another example of the configuration of the optical system of the space floating image display device will be described with reference to FIG. 2D. The optical system of FIG. 2D is an optical system that uses a retroreflector 5 that is different from the retroreflector 2 used in FIGS. 2A to 2C. Hereinafter, another example of the configuration 3 of the optical system will be described in more detail with reference to FIGS. 2D to 2I. In FIG. 2D, components that are assigned the same reference numerals as those in FIGS. 2A to 2C have the same functions and configurations as those in FIGS. 2A to 2C. For the sake of simplicity, such components will not be described repeatedly.

[0040] 2D is a diagram showing an example of the main components and retroreflection components of a space-floating image display device according to an embodiment of the present invention. A display device 10 that emits image light is provided obliquely on a transparent member 100 such as glass. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light.

[0041] A chief ray 9020 representing the light beam emitted from the display device 10 travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α may be, for example, 45°. However, the incident angle α is not limited to 45°, and may also be, for example, 45°±15°.

[0042] The retroreflector 5 is an optical element having the optical property of retroreflecting light rays in at least some directions. Furthermore, since the reflected light rays have the optical property of forming an image, the retroreflector 5 may also be referred to as an imaging optical element or an imaging optical plate.

[0043] 2E, 2F, etc., the principal ray 9020 travels in the z direction while being retroreflected in the x and y directions by the retroreflector 5. As a result, the reflected ray 9021 travels in a direction away from the retroreflector 5 along an optical path that is mirror-symmetrical with respect to the principal ray 9020 with the retroreflector 5 as the reference, passes through the transparent member 100, and forms the spatial floating image 3 as a real image on the imaging plane.

[0044] The light beam that forms the floating image 3 is a collection of light rays that converge from the retroreflector 5 to the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is an image with high directionality, unlike a diffuse image formed on a screen by a general projector or the like. Therefore, in the configuration of Figure 2, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be perceived as an image at all. This characteristic is suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0045] An example of the configuration of the retroreflector 5 will be described using Figures 2E and 2F. The retroreflector 5 has a configuration in which multiple corner reflectors 9040 are arranged in an array on the surface of a transparent member 50. This may also be called a corner reflector array or a polyhedral reflector array. The specific configuration of the corner reflector 9040 will be described in detail using Figures 2G, 2H, and 2I. Light rays 9111, 9112, 9113, and 9114 emitted from a light source 9110 are reflected twice by two mirror surfaces 9041 and 9042 of the corner reflector 9040, becoming reflected light rays 9121, 9122, 9123, and 9124. This double reflection is retroreflection in the x and y directions, where the light returns to the same direction as the incident direction (traveling in a direction rotated 180 degrees), and specular reflection in the z direction, where the angle of incidence and the angle of reflection match due to total reflection.

[0046] That is, light rays 9111 to 9114 generate reflected light rays 9121 to 9124 on straight lines symmetrical in the z direction with respect to corner reflector 9040, and form aerial real image 9120. Note that light rays 9111 to 9114 emitted from light source 9110 are four light rays that represent the diffused light from light source 9110, and although the light rays incident on retroreflector 5 are not limited to these four light rays depending on the diffusion characteristics of light source 9110, all incident light rays cause similar reflections and form aerial real image 9120. Note that for ease of viewing the drawing, the position of light source 9110 and the position of aerial real image 9120 are shown shifted in the x direction, but in reality, the position of light source 9110 and the position of aerial real image 9120 in the x direction are the same and are overlapping when viewed from the z direction.

[0047] 2G, 2H, and 2I, the configuration and effects of the corner reflector 9040 that constitutes the retroreflector 5 will be described. The corner reflector 9040 is a rectangular parallelepiped with only two specific faces being mirror surfaces 9041 and 9042, and the other four faces being made of transparent materials. The retroreflector 5 has a configuration in which these corner reflectors 9040 are arrayed so that corresponding mirror surfaces face in the same direction.

[0048] When viewed from the top (+z direction), a light ray 9111 emitted from a light source 9110 enters the mirror surface 9041 (or the mirror surface 9042) at a specific angle of incidence, is totally reflected at a reflection point 9130, and then is totally reflected again at a reflection point 9132 on the mirror surface 9042 (or the mirror surface 9041).

[0049] If the angle of incidence of light ray 9111 with respect to mirror surface 9041 (or mirror surface 9042) is θ, then the angle of incidence of first reflected light ray 9131 reflected by mirror surface 9041 (or mirror surface 9042) with respect to mirror surface 9042 (or mirror surface 9041) can be expressed as 90°-θ. Therefore, with respect to light ray 9111, second reflected light ray 9121 is rotated by 2θ after the first reflection and by 2×(90°-θ) after the second reflection, resulting in a total reversal optical path of 180°. On the other hand, when viewed from the side (the direction halfway between -x and -y), total reflection in the z direction occurs only once. Therefore, if the angle of incidence with respect to mirror surface 9041 or mirror surface 9042 is φ, then reflected light ray 9121 is rotated by 2×φ after one reflection with respect to light ray 9111.

[0050] As described above, light rays incident on the corner reflector 9040 undergo retroreflection, which creates an inverted optical path in the x and y directions, and specular reflection due to total reflection in the z direction. Considering the retroreflector 5, similar reflections occur in each optical path, so that an image is formed at a point symmetrical with respect to the z axis direction by an inverted optical path that is convergent in the x and y directions.

[0051] 2A to 2C, the retroreflector 2 has retroreflection properties in three axes. As a result, when a diffusive incident light beam is incident on the retroreflector 2, a convergent reflected light beam travels toward the side of the retroreflector 2 where the light source of the incident light beam is located. The convergent reflected light beam forms an image in the air, forming a space-floating image 3. The traveling direction of the chief ray of the convergent reflected light beam reflected from the retroreflector 2 is opposite to the traveling direction of the chief ray of the diffusive incident light beam incident on the retroreflector 2.

[0052] In contrast, in the optical system of Fig. 2D, the retroreflector 5 has retroreflection properties in two axial directions and specular reflection in the other axial direction. As a result, when a diffusive incident light beam is incident on the retroreflector 5, the convergent reflected light beam is reflected by the corner reflector array and travels in the direction opposite to the side of the retroreflector 5 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air and forms the space floating image 3.

[0053] The traveling direction of the chief ray of the convergent reflected light beam reflected by the corner reflector array of the retroreflector 5 is not the opposite direction to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5. The normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5 and the normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray after being reflected by the retroreflector 5 and becoming a convergent reflected light beam continue to travel in a straight line, unchanged before and after reflection by the corner reflector array.

[0054] That is, the diffusive incident light beam is converted into a convergent reflected light beam by reflection on the retroreflector 5, but in the normal direction to the plate-shaped surface of the retroreflector 5, the light beam travels as if passing through the retroreflector 5. Here, the diffusive incident light beam incident on the retroreflector 5 and the convergent reflected light beam emerging from the retroreflector 5 are in a geometrically symmetrical relationship with respect to the plate-shaped surface of the retroreflector 5.

[0055] The resolution of the space-floating image formed by the light beams from the video output unit 10 depends not only on the resolution of the liquid crystal display panel 11 but also on the diameter D and pitch P (not shown) of the retroreflector 5 shown in Figures 2E and 2F. For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflector is 240 μm and the pitch P is 300 μm, one pixel of the space-floating image will be equivalent to 300 μm. As a result, the effective resolution of the space-floating image will be reduced to about one-third.

[0056] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 10, it is desirable to make the diameter D and pitch P of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire due to the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Also, it is advisable to arrange the shape of the retroreflective portion so that none of its sides overlaps any of the sides of one pixel of the liquid crystal display panel.

[0057] The shape of the retroreflector (imaging optical plate) according to this embodiment is not limited to the above example. It may have various shapes that achieve retroreflection. Specifically, it may be a variety of cubic corner bodies, a corner reflector array, a slit mirror array, a dihedral corner reflector array, a polyhedral reflector array, or a shape in which a combination of these reflective surfaces is periodically arranged. Alternatively, a capsule lens-type retroreflector element with periodically arranged glass beads may be provided on the surface of the retroreflector according to this embodiment. The detailed configuration of these retroreflectors can be achieved using existing technology, so a detailed description will be omitted. Specifically, the technology disclosed in JP 2017-33005 A, JP 2019-133110 A, JP 2017-67933 A, WO 2009 / 131128 A, etc. may be used.

[0058] 2D, the image light emitted from the display device 10 may be in either polarization state, either S-polarized or P-polarized.

[0059] As explained above, the optical system of Figure 2D is an optical system that uses a retroreflector different from the optical systems of Figures 2A to 2C, but it can form a more suitable floating image in space, similar to the optical systems of Figures 2A to 2C.

[0060] According to the optical systems of FIGS. 2A, 2B, 2C, and 2D described above, it is possible to provide brighter, higher quality floating images in space.

[0061] <<Block diagram of the internal configuration of the space floating image display device>>

[0062] Next, a description will be given of a block diagram of the internal configuration of the space-floating image display device 1000. Fig. 3 is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

[0063] The space-floating image display device 1000 includes a retroreflection unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power supply input interface 1111, an operation input unit 1107, a nonvolatile memory 1108, a memory 1109, a control unit 1110, a video signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, a microphone 1139, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc. In addition, the space-floating image display device 1000 may also include a removable media interface 1134, an attitude sensor 1113, a transmissive self-luminous image display device 1650, a second display device 1680, or a secondary battery 1112.

[0064] Each component of the space floating image display device 1000 is disposed in a housing 1190. The imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0065] The retroreflecting portion 1101 in Fig. 3 corresponds to the retroreflector 2 in Fig. 2A, Fig. 2B, and Fig. 2C. The retroreflecting portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected from the retroreflecting portion 1101, the light output to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0066] 3 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. The light source 1105 in FIG. 3 corresponds to the light source device 13 in FIGS. 2A, 2B, and 2C. The image display unit 1102, the light guide 1104, and the light source 1105 in FIG. 3 correspond to the display device 1 in FIGS. 2A, 2B, and 2C.

[0067] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 (described later). The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0068] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power source 1106 converts AC current input from the outside via the external power input interface 1111 into DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each unit within the space-floating image display device 1000. The secondary battery 1112 stores the power supplied from the power source 1106. The secondary battery 1112 also supplies power to the light source 1105 and other components that require power via the external power input interface 1111 when power is not supplied from the outside. In other words, when the space-floating image display device 1000 is equipped with the secondary battery 1112, the user can use the space-floating image display device 1000 even when power is not supplied from the outside.

[0069] The light guide 1104 guides light generated by the light source 1105 and irradiates it onto the video display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called a backlight for the video display unit 1102. The light guide 1104 may be configured mainly using glass. The light guide 1104 may be configured mainly using plastic. The light guide 1104 may be configured using a mirror. Various methods are possible for combining the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0070] The aerial operation detection sensor 1351 is a sensor that detects an operation on the floating in space image 3 by the finger of the user 230. The aerial operation detection sensor 1351 senses, for example, an area that overlaps with the entire display area of ​​the floating in space image 3. Note that the aerial operation detection sensor 1351 may only sense an area that overlaps with at least a portion of the display area of ​​the floating in space image 3.

[0071] Specific examples of the aerial operation detection sensor 1351 include a distance sensor that uses invisible light such as infrared light, an invisible laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may also be configured to detect coordinates on a two-dimensional plane by combining multiple sensors. The aerial operation detection sensor 1351 may also be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) or an image sensor.

[0072] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect touch operations, etc., made by the user with their finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0073] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and based on the sensing signal, determines whether or not the finger of the user 230 has made contact with an object in the floating in space image 3, and calculates the position (contact position) where the finger of the user 230 has made contact with the object. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array), for example. Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0074] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally as a separate entity from the space-floating image display device 1000. When provided as a separate entity from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to be able to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0075] Also, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the space-floating image display device 1000 without the aerial operation detection function is used as the main body, and only the aerial operation detection function can be added as an option. Also, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the space-floating image display device 1000 may be used. In cases such as when it is desired to more freely position the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0076] The imaging unit 1180 is a camera with an image sensor, and captures images of the space near the floating-in-space image 3 and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180, or by using an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in detecting the touch operation of the floating-in-space image 3 by the user 230. The imaging unit 1180 may be provided separately from the floating-in-space image display device 1000. When the imaging unit 1180 is provided separately from the floating-in-space image display device 1000, it is sufficient to configure it so that an imaging signal can be transmitted to the floating-in-space image display device 1000 via a wired or wireless communication connection path or the like.

[0077] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that targets a plane (intrusion detection plane) including the display surface of the spatial floating image 3 and detects whether or not an object has intruded into this intrusion detection plane, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object (e.g., a user's finger) that has not intruded into the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0078] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the captured images of the multiple imaging units 1180 and object depth information from the depth sensor. These pieces of information and various pieces of information such as the distance between the object and the intrusion detection plane are used for various display controls for the floating in space image 3.

[0079] Furthermore, without using the mid-air operation detection sensor 1351, the mid-air operation detection unit 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging unit 1180.

[0080] Furthermore, the imaging unit 1180 may capture an image of the face of the user 230 operating the space-floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not there is another person standing around or behind the user 230 operating the space-floating image 3 and peeking at the operation of the user 230 on the space-floating image 3, the imaging unit 1180 may capture an image of a range including the user 230 operating the space-floating image 3 and the surrounding area of ​​the user 230.

[0081] The operation input unit 1107 is, for example, an operation button, a signal receiving unit such as a remote controller, or an infrared light receiving unit, and inputs a signal for an operation different from the air operation (touch operation) by the user 230. Apart from the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may also be used by, for example, an administrator to operate the space floating image display device 1000.

[0082] The video signal input unit 1131 is connected to an external video output device and inputs video data. Various digital video input interfaces are possible for the video signal input unit 1131. For example, it may be configured with a video input interface conforming to the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, a video input interface conforming to the DVI (Digital Visual Interface) standard, or a video input interface conforming to the DisplayPort standard.

[0083] Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio signal input unit 1133 may be configured as an HDMI-standard audio input interface, an optical digital terminal interface, a coaxial digital terminal interface, or the like. In the case of an HDMI-standard interface, the video signal input unit 1131 and the audio signal input unit 1133 may be configured as an interface in which a terminal and a cable are integrated. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may be configured as a speaker.

[0084] The audio output unit 1140 may also output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 1140 may be configured to output a digital signal to an external device, like the Audio Return Channel function defined in the HDMI standard. The microphone 1139 is a microphone that collects sounds around the space-floating image display device 1000, converts them into signals, and generates audio signals. The microphone may record a person's voice, such as a user's voice, and the control unit 1110, which will be described later, may perform voice recognition processing on the generated audio signal to obtain text information from the audio signal.

[0085] The nonvolatile memory 1108 stores various data used by the space floating image display device 1000. The data stored in the nonvolatile memory 1108 includes, for example, data for various operations to be displayed on the space floating image 3, display icons, data and layout information for objects to be operated by user operations, etc. The memory 1109 stores image data to be displayed as the space floating image 3, data for controlling the device, etc.

[0086] The control unit 1110 controls the operation of each connected unit. In addition, the control unit 1110 may cooperate with a program stored in the memory 1109 to perform calculations based on information acquired from each unit in the space floating image display device 1000.

[0087] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. If the communication unit 1132 has a wired communication interface, the wired communication interface may be configured, for example, as an Ethernet LAN interface. If the communication unit 1132 has a wireless communication interface, the interface may be configured, for example, as a Wi-Fi communication interface, a Bluetooth communication interface, or a mobile communication interface such as 4G or 5G. Various types of data, such as video data, image data, and audio data, are transmitted and received through communication via the communication unit 1132.

[0088] Furthermore, the removable media interface 1134 is an interface for connecting a removable recording medium (removable media). The removable recording medium (removable media) may be composed of a semiconductor device memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disk. The removable media interface 1134 can read various information such as video data, image data, and audio data recorded on the removable recording medium. The video data, image data, etc. recorded on the removable recording medium are output as the floating image 3 via the video display unit 1102 and the retroreflection unit 1101.

[0089] The storage unit 1170 is a storage device that records various types of information such as video data, image data, audio data, etc. The storage unit 1170 may be configured with a magnetic recording medium recording device such as a hard disk drive (HDD), or a semiconductor element memory such as a solid state drive (SSD). For example, various types of information such as video data, image data, audio data, etc. may be recorded in advance in the storage unit 1170 at the time of product shipment. Furthermore, the storage unit 1170 may record various types of information such as video data, image data, audio data, etc. acquired from an external device, an external server, etc. via the communication unit 1132.

[0090] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflection unit 1101. The video data, image data, etc. of the display icons and objects for the user to operate, etc., displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0091] Layout information of display icons, objects, etc. displayed as the space floating image 3, and various metadata information related to the objects, etc. are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0092] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. The video control unit 1160 may be called a video processing circuit, and may be configured with hardware such as an ASIC, FPGA, or video processor. The video control unit 1160 may also be called a video processing unit or an image processing unit. The video control unit 1160 controls video switching, such as which video signal to input to the video display unit 1102, between the video signal to be stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131, for example.

[0093] In addition, the video control unit 1160 may generate a superimposed video signal by superimposing the video signal to be stored in the memory 1109 and the video signal input from the video signal input unit 1131, and input the superimposed video signal to the video display unit 1102, thereby performing control to form the composite video as the floating-in-space video 3.

[0094] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing to enlarge, reduce, deform, etc. the image, brightness adjustment processing to change the brightness, contrast adjustment processing to change the contrast curve of the image, and Retinex processing to decompose the image into light components and change the weighting of each component.

[0095] Furthermore, the video control unit 1160 may perform special effect video processing or the like to assist the aerial operation (touch operation) of the user 230 on the video signal input to the video display unit 1102. The special effect video processing is performed based on, for example, the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 and the image of the user 230 captured by the imaging unit 1180.

[0096] The attitude sensor 1113 is a sensor configured by a gravity sensor or an acceleration sensor, or a combination of these, and can detect the attitude in which the space-floating image display device 1000 is installed. Based on the attitude detection result of the attitude sensor 1113, the control unit 1110 may control the operation of each connected unit. For example, when an undesirable attitude in the user's usage state is detected, the control unit 1110 may perform control such that the image being displayed on the image display unit 1102 is stopped and an error message is displayed to the user. Alternatively, when the attitude sensor 1113 detects a change in the installation attitude of the space-floating image display device 1000, the control unit 1110 may perform control such that the display direction of the image being displayed on the image display unit 1102 is rotated.

[0097] As explained above, various functions are installed in the space-floating image display device 1000. However, the space-floating image display device 1000 does not need to have all of these functions, and any configuration is acceptable as long as it has the function of forming the space-floating image 3.

[0098] <Configuration example of a space floating image display device> Next, a configuration example of the space-floating image display device will be explained. The layout of the components of the space-floating image display device according to this embodiment can be various depending on the usage form. Below, the layouts of each of Figs. 4A to 4M will be explained. In addition, in each example of Figs. 4A to 4M, the thick line surrounding the space-floating image display device 1000 indicates an example of the housing structure of the space-floating image display device 1000.

[0099] FIG. 4A is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4A is equipped with an optical system corresponding to the optical system of FIG. 2A. The space-floating image display device 1000 shown in FIG. 4A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4A, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230. Note that the x direction is the left-right direction as seen from the user, the y direction is the front-back direction (depth direction) as seen from the user, and the z direction is the up-down direction (vertical direction). Hereinafter, the definitions of the x direction, y direction, and z direction are the same in each drawing of FIG. 4, so repeated explanations will be omitted.

[0100] FIG. 4B is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4B is equipped with an optical system corresponding to the optical system of FIG. 2A. The space-floating image display device 1000 shown in FIG. 4B is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4B, the space-floating image display device is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230. 4B, the mid-air operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so this configuration can improve the accuracy of touch detection.

[0101] FIG. 4C is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4C is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4C is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4C, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the user's 230 finger.

[0102] FIG. 4D is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4D is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4D is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4D, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger. 4D, the mid-air operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so this configuration can improve the accuracy of touch detection.

[0103] FIG. 4E is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4E is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4E is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4E, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels directly upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0104] FIG. 4F is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4F is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4F is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4F, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels toward the user. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger.

[0105] FIG. 4G is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4G is equipped with an optical system corresponding to the optical system shown in FIG. 2C. In the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the central optical path of the image light emitted from the display device 1 was on the yz plane. That is, in the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the image light traveled in the front-to-back and up-to-down directions as seen from the user. In contrast, in the optical system of the space-floating image display device shown in FIG. 4G, the central optical path of the image light emitted from the display device 1 is on the xy plane. That is, in the optical system of the space-floating image display device shown in FIG. 4G, the image light travels in the left-to-right and front-to-back directions as seen from the user. The space-floating image display device 1000 shown in FIG. 4G is installed so that the surface on which the space-floating image 3 is formed faces the front of the device (toward the user 230). That is, in Fig. 4G, the space-floating image display device 1000 has the transparent member 100 installed on the front side of the device (toward the user 230). The space-floating image 3 is formed on the user side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels towards the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0106] FIG. 4H is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4H differs from the space-floating image display device of FIG. 4G in that it has a window with a transparent plate 100B made of glass or plastic on the back of the device (opposite the position where the user 230 views the space-floating image 3, i.e., opposite the traveling direction of the image light of the space-floating image 3 toward the user 230). The rest of the configuration is the same as that of the space-floating image display device of FIG. 4G, so repeated explanations will be omitted. The space-floating image display device 1000 of FIG. 4H has a window with a transparent plate 100B on the opposite side of the traveling direction of the image light of the space-floating image 3 from the space-floating image 3. Therefore, when the user 230 views the space-floating image 3, they can recognize the scenery behind the space-floating image display device 1000 as the background of the space-floating image 3. Therefore, the user 230 can perceive the space floating image 3 as floating in the air in front of the scenery behind the space floating image display device 1000. This can further emphasize the feeling of the space floating image 3 floating in the air.

[0107] Depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separator 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separator 101B and head toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, in order to prevent this stray light, the transparent plate 100B may not be provided in the window on the back of the space-floating image display device 1000.

[0108] Fig. 4I is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4I is different from the space-floating image display device of Fig. 4H in that a light-blocking door 1410 is provided in the window of the transparent plate 100B located on the back of the device (opposite the position where the user 230 views the space-floating image 3). The other configurations are the same as those of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted.

[0109] The opening and closing door 1410 of the space-floating image display device 1000 in FIG. 4I has, for example, a light blocking plate, and is equipped with a mechanism for moving (sliding), rotating, or attaching / detaching the light blocking plate, thereby switching between an open state and a light blocking state for the window (rear window) of the transparent plate 100B located at the back of the space-floating image display device 1000. The movement (sliding) and rotation of the light blocking plate by the opening and closing door 1410 may be electrically driven by a motor (not shown). The motor may be controlled by the control unit 1110 in FIG. 3. Note that the example in FIG. 4I discloses an example in which the opening and closing door 1410 has two light blocking plates. In contrast, the opening and closing door 1410 may have only one light blocking plate.

[0110] For example, when the view seen through the window of the transparent plate 100B of the space-floating image display device 1000 is outdoors, the brightness of sunlight varies depending on the weather. When the outdoor sunlight is strong, the background of the space-floating image 3 may become too bright, reducing the user 230's visibility of the space-floating image 3. In such a case, by moving (sliding), rotating, or attaching the light blocking plate of the opening / closing door 1410 to block the light from the rear window, the background of the space-floating image 3 becomes dark, thereby relatively increasing the visibility of the space-floating image 3. Such a blocking operation by the light blocking plate of the opening / closing door 1410 may be performed directly by the force of the user 230's hand. In response to an operation input via the operation input unit 1107 of FIG. 3, the control unit 1110 may control a motor (not shown) to perform the blocking operation by the light blocking plate of the opening / closing door 1410.

[0111] An illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 may control a motor (not shown) to perform the opening and closing operation of the light blocking plate of the opening and closing door 1410 according to the detection result of the illuminance sensor. By controlling the opening and closing operation of the light blocking plate of the opening and closing door 1410 in this way, it becomes possible to more suitably maintain the visibility of the space-floating image 3, even if the user 230 does not manually open and close the light blocking plate of the opening and closing door 1410.

[0112] Furthermore, the light blocking plate by the opening and closing door 1410 may be manually detachable. Depending on the intended use and installation environment of the space floating image display device 1000, the user can select whether to leave the rear window open or in a light blocking state. If it is planned to use the rear window in a light blocking state for a long period of time, the detachable light blocking plate can be fixed in the light blocking state. Also, if it is planned to use the rear window in an open state for a long period of time, it can be used with the detachable light blocking plate removed. The light blocking plate may be attached and detached using screws, a hook structure, or a fitting structure.

[0113] Even in the example of the space-floating image display device 1000 shown in FIG. 4I, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separator 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separator 101B and directed toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be perceived by the user as stray light. Therefore, to prevent this stray light, the window on the back of the space-floating image display device 1000 may be configured without the transparent plate 100B. The above-described opening / closing door 1410 may be provided in a window that does not have the transparent plate 100B. To prevent this stray light, it is desirable that the inner surface of the housing of the light-shielding plate of the above-described opening / closing door 1410 have a coating or material with low light reflectance.

[0114] Fig. 4J is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4J differs from the space-floating image display device of Fig. 4H in that instead of placing a transparent plate 100B made of glass or plastic on the rear side window, an electronically controlled transmittance variable device 1620 is placed. The other configurations are the same as those of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted. An example of the electronically controlled transmittance variable device 1620 is a liquid crystal shutter.

[0115] In other words, the liquid crystal shutter can control the amount of light transmitted by controlling the voltage of the liquid crystal element sandwiched between two polarizing plates. Therefore, if the liquid crystal shutter is controlled to increase the transmittance, the scenery through the rear window can be seen through the background of the floating image 3. Also, if the liquid crystal shutter is controlled to increase the transmittance, the scenery through the rear window can be hidden as the background of the floating image 3. Furthermore, since the liquid crystal shutter can control the intermediate length, it can also be set to a state of transmittance of 50% or the like. For example, the control unit 1110 can control the transmittance of the electronically controlled transmittance variable device 1620 in response to an operation input via the operation input unit 1107 in Fig. 3. With this configuration, in cases where a viewer wants to see the scenery through the rear window as the background of the Space Floating Image 3, but the scenery through the rear window as the background is too bright and reduces the visibility of the Space Floating Image 3, it is possible to adjust the transmittance of the electronically controlled transmittance variable device 1620 to adjust the visibility of the Space Floating Image 3.

[0116] In addition, an illuminance sensor may be provided on the back side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 in Fig. 3 controls the transmittance of the electronically controlled transmittance variable device 1620 according to the detection result of the illuminance sensor. In this way, even if the user 230 does not perform an operation input via the operation input unit 1107 in Fig. 3, the transmittance of the electronically controlled transmittance variable device 1620 can be adjusted according to the brightness of the space beyond the rear window, making it possible to more suitably maintain the visibility of the space-floating image 3.

[0117] In the above example, a liquid crystal shutter has been described as an example of the electronically controlled variable transmittance device 1620. However, electronic paper may be used as another example of the electronically controlled variable transmittance device 1620. The same effects as those described above can be obtained when electronic paper is used. Furthermore, electronic paper consumes very little power to maintain a halftone state. Therefore, a space floating image display device with lower power consumption can be realized compared to when a liquid crystal shutter is used.

[0118] Fig. 4K is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4K differs from the space-floating image display device of Fig. 4G in that it has a transmissive self-luminous image display device 1650 instead of the transparent member 100. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0119] In the space-floating image display device 1000 of FIG. 4K, an image luminous flux passes through the display surface of the transmissive self-luminous image display device 1650, and then a space-floating image 3 is formed outside the space-floating image display device 1000. That is, when an image is displayed on the transmissive self-luminous image display device 1650, which is a two-dimensional flat display, the space-floating image 3 can be displayed as a pop-up image further in front of the image displayed on the transmissive self-luminous image display device 1650. In this case, the user 230 can simultaneously view two images at different depth positions. The transmissive self-luminous image display device 1650 may be configured using existing technology such as a transmissive organic EL panel disclosed in, for example, Japanese Patent Application Laid-Open No. 2014-216761. Although not shown in FIG. 3, the transmissive self-luminous image display device 1650 may be configured as a component of the space-floating image display device 1000 of FIG. 3 and connected to other processing units such as the control unit 1110.

[0120] Here, if the transmissive self-luminous video display device 1650 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating video 3, it is possible to provide the user 230 with a more effective video experience with a surprise effect.

[0121] Furthermore, if the inside of the space-floating image display device 1000 is kept in a light-blocking state, the background of the transmissive self-luminous image display device 1650 becomes sufficiently dark. Therefore, when no image is displayed on the display device 1 or the light source of the display device 1 is turned off and an image is displayed only on the transmissive self-luminous image display device 1650, the transmissive self-luminous image display device 1650 appears to the user 230 as a normal two-dimensional flat display rather than a transmissive display (since the space-floating image 3 in the embodiment of the present invention is displayed as a real optical image in a space without a screen, if the light source of the display device 1 is turned off, the intended display position of the space-floating image 3 becomes empty space). Therefore, when the transmissive self-luminous image display device 1650 is used to display an image as if it were a normal two-dimensional flat display, characters, objects, etc. can be suddenly displayed in the air as the space-floating image 3, thereby providing the user 230 with a more effective surprise video experience.

[0122] Note that the darker the interior of the space-floating image display device 1000, the more the transmissive self-luminous image display device 1650 appears like a two-dimensional flat display. Therefore, an absorptive polarizer (not shown) that transmits the polarized waves of the image light reflected by the polarization separation member 101B and absorbs polarized waves that are 90° out of phase with the polarized waves may be provided on the surface of the transmissive self-luminous image display device 1650 facing the interior of the space-floating image display device 1000 (the surface where the image light reflected by the polarization separation member 101B enters the transmissive self-luminous image display device 1650, i.e., the surface of the transmissive self-luminous image display device 1650 opposite the space-floating image 3). This does not have a significant effect on the image light that forms the space-floating image 3, but it can significantly reduce the light that enters the interior of the space-floating image display device 1000 from the outside through the transmissive self-luminous image display device 1650, making the interior of the space-floating image display device 1000 darker, which is preferable.

[0123] 4L is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4L is a modified example of the space-floating image display device of FIG. 4K. The orientation of the components in the space-floating image display device 1000 is different from that of the space-floating image display device of FIG. 4K, and is closer to the arrangement of the space-floating image display device of FIG. 4F. The functions and operations of each component are the same as those of the space-floating image display device of FIG. 4K, so repeated explanations will be omitted.

[0124] In the space-floating image display device of FIG. 4L, after the luminous flux of image light passes through the transmissive self-luminous image display device 1650, a space-floating image 3 is formed on the user 230 side of the transmissive self-luminous image display device 1650.

[0125] In both the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, the space-floating image 3 is displayed superimposed on the image of the transmissive self-luminous image display device 1650 as seen by the user 230. Here, the position of the space-floating image 3 and the position of the image of the transmissive self-luminous image display device 1650 are configured to have a difference in the depth direction. Therefore, when the user moves their head (the position of the viewpoint), they can recognize the depth of the two images due to parallax. Therefore, by displaying two images at different depth positions, it is possible to provide the user with a more suitable three-dimensional image experience with the naked eye without the need for stereoscopic glasses or the like.

[0126] Fig. 4M is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4M is provided with a second display device 1680 on the rear side as seen from the user relative to the polarization separation member 101B of the space-floating image display device of Fig. 4G. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0127] In the configuration example shown in FIG. 4M, the second display device 1680 is provided behind the display position of the space-floating image 3, and its image display surface faces the space-floating image 3. With this configuration, from the user 230's perspective, the image of the second display device 1680 and the space-floating image 3, which are displayed at two different depth positions, can be viewed superimposed on each other. In other words, the second display device 1680 is positioned so as to display an image in the direction of the user 230 viewing the space-floating image 3. Although the second display device 1680 is not shown in FIG. 3, it may be configured to be connected to other processing units such as the control unit 1110 as one component of the space-floating image display device 1000 of FIG. 3.

[0128] Note that the image light of the second display device 1680 of the space-floating image display device 1000 of FIG. 4M is viewed by the user 230 after passing through the polarization separator 101B. Therefore, in order for the image light of the second display device 1680 to more suitably pass through the polarization separator 101B, it is desirable that the image light output from the second display device 1680 be polarized in a vibration direction that the polarization separator 101B more suitably transmits. That is, it is desirable that the image light be polarized in the same vibration direction as the polarization of the image light output from the display device 1. For example, if the image light output from the display device 1 is S-polarized, it is desirable that the image light output from the second display device 1680 is also S-polarized. Furthermore, if the image light output from the display device 1 is P-polarized, it is desirable that the image light output from the second display device 1680 is also P-polarized.

[0129] The example of the space-floating image display device of FIG. 4M also has the same effect as the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L in that a second image is displayed behind the space-floating image 3. However, unlike the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, in the example of the space-floating image display device of FIG. 4M, the luminous flux of image light for forming the space-floating image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-luminous image display device, but may be a liquid crystal display, which is a two-dimensional flat display. The second display device 1680 may also be an organic EL display. Therefore, the example of the space-floating image display device of FIG. 4M can realize the space-floating image display device 1000 at a lower cost than the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L.

[0130] Here, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and travel toward the second display device 1680. This light (a portion of the image light) may be reflected again by the surface of the second display device 1680 and may be visually recognized by the user as stray light.

[0131] Therefore, to prevent this stray light, an absorptive polarizer may be provided on the surface of the second display device 1680. In this case, the absorptive polarizer may be an absorptive polarizer that transmits the polarized waves of the image light output from the second display device 1680 and absorbs polarized waves that are 90° out of phase with the polarized waves of the image light output from the second display device 1680. If the second display device 1680 is a liquid crystal display, an absorptive polarizer is also provided on the image output side of the liquid crystal display. However, if there is a cover glass (cover glass on the image display surface side) on the output surface of the absorptive polarizer on the image output side of the liquid crystal display, it is not possible to prevent stray light caused by reflection of the cover glass by light from outside the liquid crystal display. Therefore, it is necessary to separately provide the above-mentioned absorptive polarizer on the surface of the cover glass.

[0132] When an image is displayed on the second display device 1680, which is a two-dimensional flat display, the floating-in-space image 3 can be displayed as an image further in front of the image on the second display device 1680. In this case, the user 230 can simultaneously view two images at different depth positions. By displaying a character on the floating-in-space image 3 and a background on the second display device 1680, it is possible to provide the effect that the user 230 is viewing the space in which the character exists in three dimensions.

[0133] Furthermore, if the second display device 1680 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise visual experience.

[0134] Next, Fig. 4N is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4N is a space-floating image display device that employs the optical system of Fig. 2D. As with the example of the space-floating image display device that employs the optical system of Figs. 2A to 2C, an image is formed in the air as a space-floating image 3 by image light that has passed through a transparent member 100. Furthermore, the operation of the space-floating image 3 by the user's finger 9004 can be detected using sensing light from an aerial operation detection sensor 1351 that is arranged on the back side of the transparent member 100 as seen from the user.

[0135] 2A to 2C, and in the example of the space-floating image display device employing the optical system of Fig. 2D, the space-floating image 3 is formed in front of the transparent member 100, and the operation of the space-floating image 3 by the user's finger can be detected using the sensing light of the mid-air operation detection sensor 1351 arranged on the back side of the transparent member 100 as seen from the user. Therefore, the space-floating image display device employing the optical system of Fig. 2D has a different optical system from the space-floating image display device in which the optical system of Fig. 2A to 2C is arranged on the back side of the transparent member 100 as seen from the user.

[0136] However, from the user's perspective, the usability of the space-floating image display device employing the optical system of FIG. 2D is almost the same as that of the space-floating image display device employing the optical system of FIGS. 2A to 2C.

[0137] Next, Fig. 4O is a diagram showing an example of the configuration of a space-floating image display device. Fig. 4O is a diagram showing the configuration of the internal optical system of the space-floating image display device 1000 of Fig. 4N. The space-floating image display device 1000 shown in Fig. 4O is equipped with an optical system corresponding to the optical system of Fig. 2D. The space-floating image display device 1000 shown in Fig. 4O is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward.

[0138] 4O, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0139] Here, the configuration of Fig. 4O will be compared with the configuration of Fig. 4A to confirm the differences. In Fig. 4A, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the polarization separation member 101. In contrast, in Fig. 4O, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the retroreflector 5. Furthermore, the configuration of Fig. 4A includes the retroreflector 2 and the λ / 4 plate 21, but these do not exist in Fig. 4O. Furthermore, while the presence of an absorbing polarizer 12 is more preferable in Fig. 4A, the absorbing polarizer 12 is not particularly necessary in Fig. 4O.

[0140] That is, to replace the optical system of FIG. 2A in the configuration of FIG. 4A with the optical system of FIG. 2D and to replace it with the configuration of FIG. 4O, the following can be done. That is, the polarization separation member 101 in the configuration of FIG. 4A is replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 plate 21 are removed from the configuration of FIG. 4A. The absorptive polarizer 12 is optional. By performing a replacement based on this idea, the optical system of FIGS. 2A to 2C mounted in the configuration of the space-floating image display device of FIGS. 4A to 4G can be replaced with the optical system of FIG. 2D, and the space-floating image display device can be replaced with the optical system of FIG. 2D. In this case, the polarization separation member 101 in FIGS. 4A and 4B is replaced with the retroreflector 5, and the polarization separation member 101B in FIGS. 4C to 4G is replaced with the retroreflector 5.

[0141] In this way, it is possible to realize a space-floating image display device in which the optical system in the configuration of the space-floating image display device of Figures 4A to 4G is replaced with the optical system of Figure 2D. Even in these space-floating image display devices in which the optical system of Figure 2D is replaced, it is possible to realize a space-floating image display device that is almost as easy to use as the space-floating image display device of Figures 4A to 4G.

[0142] <Display device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes its light source. In Fig. 5, the light source device 13 is shown together with the liquid crystal display panel as an exploded perspective view.

[0143] As shown by arrow 30 in Fig. 5, this liquid crystal display panel (image display element 11) receives an illumination light beam from light source device 13, which is a backlight device, that has narrow-angle diffusion characteristics, i.e., has strong directivity (straightness) and characteristics similar to laser light with a polarization plane aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam in accordance with an input video signal. The modulated image light is reflected by retroreflector 2 and passes through transparent member 100 to form a real image, a floating image in space (see Fig. 1).

[0144] 5 also shows a configuration including a liquid crystal display panel 11 constituting the display device 1, a light redirection panel 54 that controls the directional characteristics of the light beam emitted from the light source device 13, and a narrow-angle diffuser (not shown) as needed. Specifically, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarization is emitted with its intensity modulated by a video signal (see arrow 30 in FIG. 5). This allows a desired image to be projected as highly directional (linearly propagating) light of a specific polarization via the light redirection panel 54 toward the retroreflector 2. After being reflected by the retroreflector 2, the light is transmitted toward the eyes of an observer outside the store (space), forming a floating image 3. A protective cover 50 (see FIGS. 6 and 7) may be provided on the surface of the light redirection panel 54.

[0145] <Display device example 1> FIG. 6 shows an example of a specific configuration of the display device 1. In FIG. 6, a liquid crystal display panel 11 and a light direction conversion panel 54 are disposed on the light source device 13 shown in FIG. 5. The light source device 13 is configured on a case shown in FIG. 5, which is formed of, for example, plastic and contains LED elements 201 and a light guide 203. As shown in FIG. 5 and other figures, the end surface of the light guide 203 is provided with a lens shape whose cross-sectional area gradually increases toward the light receiving section in order to convert the divergent light from each LED element 201 into a substantially parallel beam. The lens shape has an effect of gradually reducing the divergence angle by multiple total reflections during propagation within the light guide 203. The liquid crystal display panel 11 constituting the display device 1 is attached to the top surface of the display device 1. In addition, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202 on which their control circuits are mounted are attached to one side surface (the left end surface in this example) of the case of the light source device 13. A heat sink, which is a member for cooling the heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED substrate 202.

[0146] The liquid crystal display panel frame (not shown) is attached to the top surface of the case of the light source device 13. The liquid crystal display panel 11 is attached to the frame, and an FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel 11 is also attached to the frame. That is, the liquid crystal display panel 11, which is the image display element, generates a display image by modulating the intensity of transmitted light in conjunction with the LED elements 201, which are solid-state light sources, based on a control signal from a control circuit (image control unit 1160 in FIG. 3) constituting the electronic device. The generated image light has a narrow diffusion angle and contains only specific polarization components, resulting in a novel image display device similar to a surface-emitting laser image source driven by a video signal. Currently, it is technically and safety-wise impossible to obtain a laser beam of the same size as the image obtained by the display device 1 described above using a laser device. Therefore, in this embodiment, light similar to the surface-emitting laser image light described above is obtained from a beam of light from a general light source, for example, an LED element.

[0147] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 7 as well as FIG.

[0148] 6 and 7 are cross-sectional views, and only one of the multiple LED elements 201 constituting the light source is shown, and this light is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion of the light guide end surface and the LED element are attached while maintaining a predetermined positional relationship.

[0149] Each light guide 203 is formed of a translucent resin such as acrylic. The LED light receiving surface at the end of light guide 203 has a cone-shaped outer periphery obtained by rotating a parabolic cross section, and the top of the light guide 203 has a concave portion with a convex portion (i.e., a convex lens surface) formed in the center, and the center of the flat portion has a convex lens surface (alternatively, a concave lens surface) that protrudes outward (not shown). The outer shape of the light receiving portion of the light guide to which LED element 201 is attached is a parabolic shape that forms a cone-shaped outer periphery, and is set within an angle range that allows total reflection within the light that is emitted from the LED element toward the periphery, or a reflective surface is formed.

[0150] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed to the LED collimator (light-receiving end surface 203a) so that the LED elements 201 on the surface are positioned in the center of the recessed portion described above.

[0151] According to this configuration, the shape of the light-receiving end surface 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as approximately parallel light, thereby improving the efficiency of use of the generated light.

[0152] As described above, the light source device 13 is configured by attaching a light source unit in which a plurality of LED elements 201 serving as light sources are arranged to the light-receiving end surface 203a, which is a light-receiving section provided on the end surface of the light guide 203, and the divergent light beams from the LED elements 201 are converted into approximately parallel light by the lens shape of the light-receiving end surface 203a of the light guide end surface, which is then guided inside the light guide 203 (in a direction parallel to the drawing) as shown by the arrow, and emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11, which is disposed approximately parallel to the light guide 203 (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of the light beam direction conversion means 204 depending on the shape inside or on the surface of the light guide.

[0153] The light beam direction conversion means 204 described above emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 (in a direction perpendicular to the front of the drawing) which is disposed substantially parallel to the light guide 203, by changing the shape of the surface of the light guide or by providing a portion with a different refractive index inside the light guide. In this case, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is positioned at the same position as the diagonal dimension of the screen, if the relative brightness ratio between the center of the screen and the periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

[0154] 6 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED element 201. In Fig. 6, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface of light source device 13 is attached liquid crystal display panel 11 that has polarizing plates on the light source light entrance surface and the image light exit surface.

[0155] In addition, a film- or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarization (e.g., P-wave) 212 of the natural light beam 210 emitted from the LED element 201. The reflected light is reflected again by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and directed toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49. The reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting the reflected light beam from P-polarized to S-polarized, thereby improving the utilization efficiency of the light source light as image light. The image light beam, the light intensity of which is modulated by a video signal in the liquid crystal display panel 11 (arrow 213 in Figure 6), enters the retroreflector 2. After reflection by the retroreflector 2, a real, floating image can be obtained.

[0156] 7 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED elements 201, similar to Fig. 6. Light source device 13 is similarly composed of light guide 203 formed of, for example, plastic and having light beam direction conversion means 204 on its surface or inside, LED elements 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc. On the top surface of light source device 13, a liquid crystal display panel 11 is attached as an image display element, which has polarizing plates on the light source light entrance surface and the image light exit surface.

[0157] A film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., S wave) 211 of the natural light beam 210 emitted from the LED element 201. In other words, the selective reflection characteristics of the reflective polarizing plate 49 in the example of FIG. 7 differ from those in FIG. 7. The reflected light is reflected by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and returns to the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting it from S-polarized light to P-polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam intensity-modulated by the image signal in the liquid crystal display panel 11 (arrow 214 in FIG. 7) enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0158] In the light source devices shown in Figures 6 and 7, in addition to the function of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, the reflective polarizer reflects the polarized light component on one side, so the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizer multiplied by the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In fact, experiments have confirmed that the contrast performance of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those of self-luminous organic EL displays are obtained.

[0159] <Display device example 2> 8 shows another example of the specific configuration of the display device 1. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, etc. in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. Also, an LED board on which LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and their control circuits are mounted are attached to one side of the case of light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer surface of the LED board.

[0160] The liquid crystal display panel frame attached to the top surface of the case is configured to have attached thereto a liquid crystal display panel 11 attached to the frame, and further to have attached thereto an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements 14a and 14b, which are solid-state light sources, based on a control signal from a control circuit (not shown here) that constitutes the electronic device.

[0161] <Display device example 3> Next, another example of the specific configuration of the display device 1 (Example 3 of the display device) will be described with reference to Fig. 9. The light source device of this display device 1 converts a divergent beam of light (a mixture of P-polarized and S-polarized light) from an LED into a substantially parallel beam by a collimator 18, and reflects the parallel beam toward the liquid crystal display panel 11 by the reflecting surface of a reflective light guide 304. The reflected light is incident on a reflective polarizer 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizer 49 transmits light of a specific polarization (e.g., P-polarized light) and causes the transmitted polarized light to be incident on the liquid crystal display panel 11. Here, light of polarization other than the specific polarization (e.g., S-polarized light) is reflected by the reflective polarizer 49 and directed toward the reflective light guide 304 again.

[0162] The reflective polarizing plate 49 is installed at an angle with respect to the liquid crystal display panel 11 so that the reflective polarizing plate 49 is not perpendicular to the chief ray of light from the reflective surface of the reflective light guide 304. The chief ray of the light reflected by the reflective polarizing plate 49 is incident on the transmission surface of the reflective light guide 304. The light that has entered the transmission surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 which is a retardation plate, and is reflected by the reflector 271. The light reflected by the reflector 271 passes through the λ / 4 plate 270 again, and passes through the transmission surface of the reflective light guide 304. The light that has passed through the transmission surface of the reflective light guide 304 is incident on the reflective polarizing plate 49 again.

[0163] At this time, the light that re-enters the reflective polarizer 49 has passed through the λ / 4 plate 270 twice, and therefore its polarization has been converted to a polarization (for example, P-polarized light) that is transmitted through the reflective polarizer 49. Therefore, the light whose polarization has been converted passes through the reflective polarizer 49 and enters the liquid crystal display panel 11. Note that with regard to the polarization design related to the polarization conversion, the polarization may be configured in reverse from the above explanation (S-polarized light and P-polarized light may be reversed).

[0164] As a result, the light from the LED is aligned to a specific polarization (for example, P polarization), enters the liquid crystal display panel 11, and is brightness-modulated in accordance with the video signal to display an image on the panel surface. As in the above example, multiple LEDs that make up the light source are shown (however, since this is a vertical cross section, only one is shown in Figure 9), and these are attached at predetermined positions relative to the collimator 18.

[0165] Each of the collimators 18 is formed of a translucent resin such as acrylic or glass. The collimator 18 may have a cone-shaped outer peripheral surface obtained by rotating a parabolic cross section. The collimator 18 may have a concave portion with a convex portion (i.e., a convex lens surface) formed in the center of the apex (the side facing the LED substrate 102) of the collimator 18. The collimator 18 may have a convex lens surface protruding outward (or a concave lens surface recessed inward) in the center of the flat portion (the side opposite the apex). The parabolic surface forming the cone-shaped outer peripheral surface of the collimator 18 is set within an angle range that allows total reflection of the light emitted from the LED in the peripheral direction within the parabolic surface, or a reflective surface is formed therein.

[0166] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 102. The LED substrate 102 is arranged and fixed to the collimator 18 so that the LEDs on the surface are positioned at the center of the apex of the convex cone shape (or in the recess if there is a recess at the apex).

[0167] With this configuration, the collimator 18 focuses the light emitted from the LED, particularly the light emitted from the central portion, into parallel light by the convex lens surface that forms the outer shape of the collimator 18. Light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the conical shape of the collimator 18, and is similarly focused into parallel light. In other words, the collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, makes it possible to extract almost all of the light generated by the LED as parallel light, thereby improving the utilization efficiency of the generated light.

[0168] Furthermore, the light converted into approximately parallel light by the collimator 18 shown in FIG. 9 is reflected by the reflective light guide 304. Due to the action of the reflective polarizer 49, light of a specific polarization of the light passes through the reflective polarizer 49, while light of the other polarization reflected by the reflective polarizer 49 passes through the light guide 304 again. The light is reflected by the reflector 271, which is located opposite the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarized and converted twice by passing through the λ / 4 plate 270, which is a retardation plate. The light reflected by the reflector 271 passes through the light guide 304 again and enters the reflective polarizer 49 provided on the opposite surface. Since the incident light has been polarization-converted, it passes through the reflective polarizer 49, aligns its polarization direction, and enters the liquid crystal display panel 11. As a result, all of the light from the light source can be utilized, thereby doubling the geometrical optical utilization efficiency of light. Furthermore, since the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, the use of the light source device of this embodiment significantly improves the contrast ratio of the entire display device. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 allows adjustment of the angle of light reflection and diffusion at each reflective surface. The surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 can be adjusted for each design to optimize the uniformity of the light incident on the liquid crystal display panel 11.

[0169] It should be noted that the λ / 4 plate 270, which is the retardation plate in Fig. 9, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of Fig. 9, any retardation plate may be used as long as the phase changes by 90° (λ / 2) when polarized light passes through it twice. The thickness of the retardation plate may be adjusted according to the incident angle distribution of the polarized light.

[0170] <Display device example 4> Furthermore, another example (Example 4 of Display Device) of the configuration of an optical system such as a light source device of a display device will be described with reference to Fig. 10. This is an example of a configuration in which a diffusion sheet is used instead of reflective light guide 304 in the light source device of Example 3 of the display device. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) that convert the diffusion characteristics in the vertical and horizontal directions of the drawing (front and rear directions in the drawing, not shown) are used on the light emission side of collimator 18, and light from collimator 18 is made to enter between the two optical sheets (diffusion sheets).

[0171] The optical sheet may be a single sheet instead of a two-sheet configuration. In the case of a single sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes on the front and back surfaces of the single optical sheet. Alternatively, multiple diffusion sheets may be used to share the functions. In the example of FIG. 10, the reflection and diffusion characteristics due to the front and back shapes of optical sheets 207A and 207B may be optimally designed using the number of LEDs, the divergence angle from LED substrate (optical element) 102, and the optical specifications of collimator 18 as design parameters so that the surface density of the light beam emitted from liquid crystal display panel 11 is uniform. In other words, the diffusion characteristics are adjusted by the surface shapes of multiple diffusion sheets instead of light guides.

[0172] In the example of FIG. 10, polarization conversion is performed in the same manner as in the display device example 3 described above. That is, in the example of FIG. 10, reflective polarizing plate 49 may be configured to have the property of reflecting S-polarized light (transmitting P-polarized light). In this case, the reflective polarizing plate 49 transmits P-polarized light out of the light emitted from the LED light source, and the transmitted light enters liquid crystal display panel 11. The reflective polarizing plate 49 reflects S-polarized light out of the light emitted from the LED light source, and the reflected light passes through retardation plate 270 shown in FIG. 10. The light that passes through retardation plate 270 is reflected by reflector 271. The light reflected by reflector 271 passes through retardation plate 270 again and is converted to P-polarized light. The polarization-converted light passes through reflective polarizing plate 49 and enters liquid crystal display panel 11.

[0173] It should be noted that the λ / 4 plate 270, which is the retarder in FIG. 10, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of FIG. 10, any retarder that changes the phase by 90° (λ / 2) when polarized light passes through it twice will suffice. The thickness of the retarder may be adjusted according to the distribution of incident angles of the polarized light. It should be noted that in FIG. 10 as well, the polarization design for polarization conversion may be configured in reverse (reversing the S-polarized light and P-polarized light) from the above explanation.

[0174] In a typical TV device, the light emitted from the LCD panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (shown on the X-axis in FIG. 12(a)) and the vertical direction of the screen (shown on the Y-axis in FIG. 12(b)). In contrast, the diffusion characteristics of the light beam emitted from the LCD panel of this embodiment are 1 / 5 of the 62-degree viewing angle of a typical TV device, as shown in Example 1 of FIG. 12, when the viewing angle at which the luminance is 50% of that at a front view (angle of 0 degrees) is set to 13 degrees. Similarly, the vertical viewing angle is asymmetric between the top and bottom, and the reflection angle and the area of ​​the reflective surface of the reflective light guide are optimized to keep the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness more than 50 times higher.

[0175] Furthermore, assuming the viewing angle characteristics shown in Example 2 in Figure 12, if the viewing angle at which brightness is 50% of that when viewed from the front (angle of 0 degrees) is set to 5 degrees, this will be 1 / 12 of the 62 degrees of devices used for general TV applications. Similarly, the vertical viewing angle is optimized by optimizing the reflection angle and the area of ​​the reflective surface of the reflective light guide so that the viewing angle is approximately 1 / 12 of that of devices used for general TV applications, with equal viewing angle both above and below. As a result, the amount of image light directed in the monitoring direction is significantly improved compared to conventional LCD TVs, and brightness is more than 100 times greater.

[0176] As described above, by setting the viewing angle to a narrow angle, the amount of luminous flux directed in the monitoring direction can be concentrated, significantly improving the efficiency of light utilization. As a result, even when using a liquid crystal display panel for general TV applications, by controlling the light diffusion characteristics of the light source device, it is possible to achieve a significant improvement in brightness with similar power consumption, making it possible to create a video display device that is compatible with information display systems facing bright outdoor environments.

[0177] When using a large LCD display panel, the overall brightness of the screen can be improved by directing the light from the periphery of the screen inward so that it is directed toward the observer when the observer is facing the center of the screen. Figure 11 shows the convergence angle between the long and short sides of the panel when the observer's distance from the panel, L, and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in portrait orientation, the convergence angle can be set to match the short side. For example, when using a 22-inch panel in portrait orientation and the monitoring distance is 0.8 m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the observer.

[0178] Similarly, when monitoring with a 15-inch panel in portrait orientation, if the monitoring distance is 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the monitor. As mentioned above, depending on the size of the LCD panel and whether it is used portrait or landscape, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the monitor who is in the optimum position to monitor the centre of the screen.

[0179] As shown in Figure 9, the basic configuration involves a light source device directing a light beam with a narrow angle of directionality to a liquid crystal display panel 11, which is then luminance-modulated according to a video signal. The video information displayed on the screen of the liquid crystal display panel 11 is then reflected by a retroreflector, and the resulting floating image is displayed indoors or outdoors via a transparent member 100.

[0180] By using the display device and light source device according to the embodiment of the present invention described above, it is possible to realize a space floating image display device with higher light utilization efficiency.

[0181] <Example of image display processing in a space floating image display device> Next, an example of a problem solved by the image processing of this embodiment will be described with reference to Fig. 13A. In the space-floating image display device 1000, when the far side of the space-floating image 3 from the user's perspective is inside the housing of the space-floating image display device 1000 and it is sufficiently dark, the user will visually recognize that the background of the space-floating image 3 is black.

[0182] Here, an example of displaying a character "panda" 1525 in the space floating image 3 will be described using Fig. 13A. First, the image control unit 1160 in Fig. 3 distinguishes and recognizes the pixel area where the image of the character "panda" 1525 is drawn from the transparent information area 1520 which is the background image, for an image including the pixel area where the image of the character "panda" 1525 is drawn and the transparent information area 1520 which is the background image, as shown in Fig. 13A(1).

[0183] A method for distinguishing and recognizing the character image from the background image is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the character image layer in front of the background image layer can be processed as separate layers, and the character image and background image can be distinguished and recognized based on the superimposition relationship when these layers are combined.

[0184] Here, the image control unit 1160 recognizes black pixels that depict objects such as character images and transparent information pixels as different information. However, it is assumed that both the black pixels that depict objects and the transparent information pixels have a luminance of 0. In this case, when the space floating image 3 is displayed, there is no difference in luminance between the pixels that depict black in the image of the character "panda" 1525 and the pixels of the transparent information region 1520, which is the background image. Therefore, in the space floating image 3, as shown in FIG. 13A(2), neither the pixels that depict black in the image of the character "panda" 1525 nor the pixels of the transparent information region 1520 have luminance, and they are visually perceived by the user as the same optically black space. In other words, the black parts of the image of the character "panda" 1525, which is an object, blend into the background, and only the non-black parts of the character "panda" 1525 are perceived as floating in the display region of the space floating image 3.

[0185] An example of image processing according to this embodiment will be described with reference to FIG. 13B. FIG. 13B is a diagram illustrating an example of image processing that more suitably resolves the issue of the black image region of the object blending into the background, as described in FIG. 13A. In FIGS. 13B(1) and 13B(2), the upper side shows the display state of the floating image 3 in space, and the lower side shows the input / output characteristics of the image processing of the image of the object. Note that the image of the object (character "panda" 1525) and the corresponding data may be read from the storage unit 1170 or memory 1109 in FIG. 3, or may be input from the video signal input unit 1131, or may be acquired via the communication unit 1132.

[0186] In the state shown in Figure 13B(1), the input / output characteristics of the image processing of the object image are in a linear state with no particular adjustment. In this case, the display state is the same as that shown in Figure 13A(2), and the black image area of ​​the object blends into the background. In contrast, in Figure 13B(2), the video control unit 1160 of this embodiment adjusts the input / output characteristics of the image processing of the image of the object (character "panda" 1525) to the input / output characteristics shown in the lower part.

[0187] That is, the video control unit 1160 performs image processing with input / output characteristics on the image of the object (character "panda" 1525), which has a characteristic of converting pixels in low-brightness areas of the input image into output pixels with increased brightness values. After the image of the object (character "panda" 1525) has been subjected to image processing with the input / output characteristics, a video including the image of the object (character "panda" 1525) is input to the display device 1 and displayed. Then, as shown in the upper part of FIG. 13B(2), the display state of the floating in space image 3 is such that the brightness of pixel areas depicting black in the image of character "panda" 1525 increases. This allows the user to distinguish the areas depicting black among the areas depicting the image of character "panda" 1525 without them blending into the black background, making it possible to display the object more appropriately.

[0188] 13B(2), the area displaying the image of the character "panda" 1525, which is an object, can be distinguished from the black background inside the housing of the space-floating image display device 1000 through the window, improving the visibility of the object. Therefore, for example, even if the object includes pixels with a brightness value of 0 before the image processing (i.e., when the image of the object or the corresponding data is read from the storage unit 1170 or memory 1109 in FIG. 3, or when the image of the object is input from the video signal input unit 1131, or when the data of the object is obtained via the communication unit 1132, etc.), the image processing of the input / output characteristics by the video control unit 1160 converts the object into an object with a brightness value of 0 increased for the pixels in the low-brightness area, and then the object is displayed on the display device 1 and converted into a space-floating image 3 by the optical system of the space-floating image display device 1000.

[0189] That is, the pixels that make up the object after image processing of the input / output characteristics are converted to a state in which they do not include pixels with a brightness value of 0, and then they are displayed on the display device 1 and converted into a floating image 3 in space by the optical system of the floating image display device 1000.

[0190] In the image processing of Figure 13B(2), a method for applying image processing with the input / output characteristics of Figure 13B(2) only to the image area of ​​the object (character "panda" 1525) is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the layer of the character image in front of the background image layer can be processed as separate layers, and the image processing with the input / output characteristics of Figure 13B(2) is applied to the character image layer, while not applying this image processing to the background image layer.

[0191] Then, by combining these layers, image processing with a characteristic of raising the low-brightness areas of the input image is performed only on the character image, as shown in Fig. 13B(2). Alternatively, after the character image layer and background image layer are combined, image processing with the input / output characteristics of Fig. 13B(2) may be performed only on the character image area.

[0192] Furthermore, the input / output image characteristics used in the image processing for boosting low-luminance regions of the input / output characteristics for the input image are not limited to the example shown in FIG. 13B(2). Any image processing for boosting low luminance may be used, including so-called brightness adjustment. Alternatively, image processing for improving visibility by controlling the gain that changes the weighting of Retinex processing, as disclosed in International Publication WO 2014 / 162533, may be performed.

[0193] According to the image processing of FIG. 13B(2) described above, it is possible to make the user aware of areas where black is drawn among areas where images of characters, objects, etc. are drawn without blending into the black background, thereby realizing a more suitable display.

[0194] 13A and 13B, the problems and more suitable image processing were explained using the space-floating image display device in which the background appears black (for example, the space-floating image display device 1000 in FIGS. 4A to 4G, or the space-floating image display device 1000 in the state where the rear window is shielded in FIGS. 4I and 4J). However, the image processing is also effective for devices other than these space-floating image display devices.

[0195] Specifically, in the space-floating image display device 1000 of Fig. 4H, or in Fig. 4I and Fig. 4J where the rear window is not shaded, the background of the space-floating image 3 is not black, but the scenery behind the space-floating image display device 1000 through the window. In this case, the same problems as those described in Fig. 13A and Fig. 13B exist.

[0196] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the scenery behind the space-floating image display device 1000 through the window. In this case too, by using the image processing of Fig. 13B(2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognized as being distinct from the scenery behind the space-floating image display device 1000 through the window, improving the visibility of the object.

[0197] That is, by using the image processing of FIG. 13B(2), the area displaying the image of the object character "panda" 1525 can be recognized as distinct from the scenery behind the space floating image display device 1000 through the window, and it becomes possible to more easily recognize that the object character "panda" 1525 is in front of the scenery, improving the visibility of the object.

[0198] 4K, 4L, and 4M, as described above, when another image (such as an image from the transmissive self-luminous image display device 1650 or an image from the second display device 1680) is displayed at a position different in depth from the space-floating image 3, the background of the space-floating image 3 is not black, but the other image. In this case, the problems explained in FIGS. 13A and 13B also exist.

[0199] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the other image that is displayed at a different depth from the floating image in space 3. In this case as well, by using the image processing of Fig. 13B(2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be able to be recognized as distinct from the other image, improving the visibility of the object.

[0200] In other words, by using the image processing of Figure 13B (2), the area displaying the image of the object character "panda" 1525 can be recognized as distinct from the other image, and it can be more easily recognized that the object character "panda" 1525 is in front of the other image, improving the visibility of the object.

[0201] An example of the image display process of this embodiment will be described with reference to Fig. 13C. Fig. 13C shows an example of the image display of this embodiment in which the space floating image 3 and a second image 2050, which is another image, are simultaneously displayed. The second image 2050 may correspond to the image displayed by the transmissive self-luminous image display device 1650 of Fig. 4K or Fig. 4L. The second image 2050 may also correspond to the image displayed by the second display device 1680 of Fig. 4M.

[0202] That is, the example of the image display in Fig. 13C shows a specific example of the image display examples of the space-floating image display device 1000 in Figs. 4K, 4L, and 4M. In the example of this figure, a bear character is displayed in the space-floating image 3. The area other than the bear character in the space-floating image 3 is displayed in black, and becomes transparent as a space-floating image. In addition, the second image 2050 is a background image in which a plain, a mountain, and a sun are drawn.

[0203] 13C, the floating in space image 3 and the second image 2050 are displayed at different depth positions. When the user 230 views the two images, the floating in space image 3 and the second image 2050, in the line of sight of the arrow 2040, the user 230 can view the two images in a superimposed state. Specifically, the bear character of the floating in space image 3 appears to be superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0204] Here, since the space floating image 3 is formed as a real image in the air, when the user 230 moves his / her viewpoint slightly, he / she can recognize the depth of the space floating image 3 and the second image 2050 due to parallax. Therefore, the user 230 can get a stronger sense of floating in space from the space floating image 3 while viewing the two images in an overlapping state.

[0205] An example of the video display process of this embodiment will be described with reference to Fig. 13D. Fig. 13D(1) is a diagram of the floating in space image 3, from the example of the video display of this embodiment in Fig. 13C, as seen from the line of sight of the user 230. Here, a bear character is displayed in the floating in space image 3. The area other than the bear character in the floating in space image 3 is displayed in black, and becomes transparent as a floating in space image.

[0206] 13D(2) is a diagram showing the second image 2050 in the example of the video display of this embodiment in FIG. 13C as viewed from the line of sight of the user 230. In the example of this figure, the second image 2050 is a background image in which a plain, a mountain, and a sun are drawn.

[0207] 13D(3) is a diagram showing the state in which the second image 2050 and the floating in space image 3 appear superimposed in the line of sight of the user 230, among the example of image display of this embodiment in Fig. 13C. Specifically, the bear character of the floating in space image 3 appears superimposed in front of the background of plains, mountains, and the sun drawn in the second image 2050.

[0208] Here, when simultaneously displaying the space-floating image 3 and the second image 2050, it is desirable to pay attention to the balance of brightness between the two images in order to ensure better visibility of the space-floating image 3. If the second image 2050 is too bright compared to the brightness of the space-floating image 3, the displayed image of the space-floating image 3 will be transparent, and the second image 2050, which is the background, will be strongly visible through it.

[0209] Therefore, the output of the light source of the spatially floating image 3 and the display image brightness of the display device 1, and the output of the light source of the display device displaying the second image 2050 and the display image brightness of the display device should be set so that at least the brightness per unit area of ​​the spatially floating image 3 at the display position of the spatially floating image 3 is greater than the brightness per unit area of ​​the image light that reaches the display position of the spatially floating image 3 from the second image 2050.

[0210] Note that this condition only needs to be satisfied when the space-floating image 3 and the second image 2050 are displayed simultaneously, and therefore when switching from the first display mode in which the space-floating image 3 is not displayed and only the second image 2050 is displayed to the second display mode in which the space-floating image 3 and the second image 2050 are displayed simultaneously, control may be performed to reduce the brightness of the second image 2050 by lowering the output of the light source of the display device that displays the second image 2050 and / or the display image brightness of the display device. These controls may be realized by the control unit 1110 in Fig. 3 controlling the display device 1 and the display device that displays the second image 2050 (the transmissive self-luminous image display device 1650 in Fig. 4K or Fig. 4L or the second display device 1680 in Fig. 4M).

[0211] Note that when switching from the above-described first display mode to the above-described second display mode, if control is performed to reduce the brightness of second image 2050, the brightness may be reduced uniformly across the entire screen of second image 2050. Alternatively, instead of reducing the brightness uniformly across the entire screen of second image 2050, the brightness reduction effect may be greatest in the portion where an object is displayed in space-floating image 3, and the brightness reduction effect may be gradually weakened around that portion. In other words, if the brightness of second image 2050 is reduced only in the portion where space-floating image 3 is visually recognized as being superimposed on second image 2050, the visibility of space-floating image 3 is sufficiently ensured.

[0212] Here, since the space floating image 3 and the second image 2050 are displayed at positions with different depths, when the user 230 slightly changes his / her viewpoint, the parallax causes a change in the superimposed position of the space floating image 3 relative to the second image 2050. Therefore, when switching from the above-mentioned first display mode to the above-mentioned second display mode, if the brightness is to be reduced unevenly across the entire screen of the second image 2050, it is not desirable to reduce the brightness sharply based on the outline of the object displayed in the space floating image 3, but rather it is desirable to perform a gradation process of the brightness reduction effect, which changes the brightness reduction effect stepwise depending on the position as described above.

[0213] In addition, in the space floating image display device 1000 where the position of the object displayed in the space floating image 3 is approximately at the center of the space floating image 3, the position where the brightness reduction effect of the gradation processing of the brightness reduction effect is greatest can be the center position of the space floating image 3.

[0214] According to the image display process of this embodiment described above, the user 230 can visually recognize the space floating image 3 and the second image 2050 more favorably.

[0215] Note that when displaying the space-floating image 3, control may be performed so as not to display the second image 2050. Since not displaying the second image 2050 increases the visibility of the space-floating image 3, this is suitable for applications such as the space-floating image display device 1000 where the user must be able to reliably view the space-floating image 3 when the space-floating image 3 is displayed.

[0216] <Example 2> As Example 2 of the present invention, an example of another configuration example of the space-floating image display device will be described. Note that the space-floating image display device according to this example is obtained by changing the optical system stored in the space-floating image display device described in Example 1 to the optical system shown in FIG. 14(1) or FIG. 14(2). In this example, differences from Example 1 will be described, and repeated explanations of the same configuration as Example 1 will be omitted. Note that in the following description of this example, the predetermined polarized light and the other polarized light are polarized waves whose phases differ by 90° from each other.

[0217] Fig. 14(1) shows an example of an optical system and an optical path according to this embodiment. The optical system shown in Fig. 14(1) is configured such that the display device 1 is closer to the polarization separation member 101B in the optical system of Fig. 2C, making the entire optical system more compact. In Fig. 14(1), the components denoted by the same reference numerals as in Fig. 2C will not be described in detail again.

[0218] 14(1), similar to FIG. 2C, image light of a predetermined polarized light (P polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, similar to FIG. 2C, polarization separation member 101B selectively transmits the predetermined polarized light (P polarized light in the figure) emitted from display device 1 and reflects the other polarized light (S polarized light in the figure).

[0219] Therefore, image light of a predetermined polarization (P-polarized light in the figure) traveling vertically from the image display surface of display device 1 passes through polarization separator 101B and reaches retroreflector 2 to which λ / 4 plate 21 is attached. The image light that is retroreflected by retroreflector 2 and travels again toward polarization separator 101B has passed through λ / 4 plate 21 twice, and is converted from the predetermined polarization (P-polarized light in the figure) at the time of emission from display device 1 to the other polarization (S-polarized light in the figure). The image light that travels again toward polarization separator 101B is the other polarization (S-polarized light in the figure), and is therefore reflected by polarization separator 101B toward the position where the user should be. The traveling direction of the image reflected by polarization separator 101B is determined based on the angle at which polarization separator 101B is disposed.

[0220] In the example of Figure 14(1), the image light traveling toward the polarization separation member 101B is reflected at a right angle by the polarization separation member 101B and travels as shown in the figure. The image light reflected by the polarization separation member 101B forms a space-floating image 3A. The space-floating image 3A can be viewed by the user from the direction of arrow A.

[0221] Here, due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 until it reaches the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 until it reaches the formation position of the space floating image 3A. This relationship determines the formation position of the space floating image 3A in the traveling direction of the image light reflected by the polarization separation member 101B.

[0222] In the example of FIG. 14(1), the display device 1, the polarization separating member 101B, and the retroreflector 2 are arranged closer than in the example of FIG. 2C. This allows the entire optical system to be configured more compactly. However, the amount by which the space-floating image 3A protrudes from the optical system of FIG. 14(1) is not very large. For example, as an index of the amount by which the space-floating image 3A protrudes from the optical system, the figure shows the distance from the position where the central light beam of the image light is reflected by the polarization separating member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of FIG. 14(1)).

[0223] 14(1), the characteristics of P polarization and S polarization may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S polarization, and the characteristics of P polarization and S polarization may be interchanged in the reflection characteristics of the polarization separation member 101B. In this case, the P polarization and S polarization shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0224] Next, Fig. 14(2) shows another example of an optical system and an optical path according to this embodiment. The optical system of Fig. 14(2) is a modified version of the optical system of Fig. 14(1) in order to increase the amount of the floating image projecting from the optical system while realizing the same compactness as the optical system of Fig. 14(1). In Fig. 14(2), the components with the same reference numerals as those in Fig. 14(1) will not be described in detail again.

[0225] 14(2), similar to FIG. 14(1), image light of a predetermined polarized light (P polarized light in the figure) emitted from the display device 1 travels in a direction perpendicular to the image display surface of the display device 1. Here, the polarization characteristics of the polarization separation member 101B are arranged 90 degrees differently from those in FIG. 14(1). Image light of a predetermined polarized light (P polarized light in the figure) traveling in a direction perpendicular to the image display surface of the display device 1 passes through the polarization separation member 101B.

[0226] 14(1), the image light passing through the polarization separation member 101B is not faced with the retroreflector 2 having the λ / 4 plate 21 attached thereto, but with the specular reflector 4 having the λ / 4 plate 21B attached thereto. Here, the reflection at the specular reflector 4 is specular reflection (also called regular reflection), not retroreflection.

[0227] Therefore, the image light that has passed through polarization separation member 101B is specularly reflected by specular reflector 4 to which λ / 4 plate 21B is attached. The image light that has been specularly reflected by specular reflector 4 and travels again toward polarization separation member 101B has been converted from the predetermined polarization (P polarization in the figure) at the time of emission from display device 1 to the other polarization (S polarization in the figure) by having passed through λ / 4 plate 21 twice. The image light that has traveled again toward polarization separation member 101B is the other polarization (S polarization in the figure), and is therefore reflected by polarization separation member 101B.

[0228] Here, because the orientation of the polarization separator 101B in Figure 14(2) is different from that in Figure 14(1), the image light reflected by the polarization separator 101B travels in the opposite direction from where the user should be. A retroreflector 2 with a λ / 4 plate 21C attached is disposed at the destination of the image light reflected by the polarization separator 101B. The image light is retroreflected by the retroreflector 2. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separator 101B has been converted from the other polarized light (S-polarized light in the figure) to the specified polarized light (P-polarized light in the figure) by passing through the λ / 4 plate 21C twice.

[0229] The image light that travels back toward the polarization separation member 101B is of a predetermined polarization (P polarization in the figure), so it passes through the polarization separation member 101B and continues toward the location where the user should be. The image light that has passed through the polarization separation member 101B forms a space-floating image 3B. The space-floating image 3B can be easily viewed by the user from the direction of arrow A.

[0230] 14(2), similarly to FIG. 14(1), due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the formation position of the space-floating image 3B. This relationship determines the formation position of the space-floating image 3B in the traveling direction of the image light transmitted through the polarization separation member 101B.

[0231] 14(2), the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is longer than the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 in FIG. 14(1). This is because in the optical system of FIG. 14(2), an optical path that goes back and forth between the polarization separation member 101B and the specular reflector 4, which does not exist in the optical system of FIG. 14(1), is added to the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2.

[0232] As a result, in the optical system of Figure 14(2), the distance from the position where the central ray of the image light passes through the polarization separation member 101B to the position where the image light forms the space-floating image 3B (L2 in the example of Figure 14(2)) is significantly longer than the distance from the position where the central ray of the image light is reflected by the polarization separation member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)) in the optical system of Figure 14(1).

[0233] 14(2), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0234] As described above, according to the optical systems of Fig. 14(1) and Fig. 14(2) in the second embodiment of the present invention, a more compact optical system can be realized. In particular, according to the optical system of Fig. 14(2), it is possible to increase the amount of the floating image projecting from the optical system, despite the more compact optical system.

[0235] When the optical system of Fig. 14(1) or Fig. 14(2) is incorporated into a space-floating image display device, it can be realized by replacing the optical system in the space-floating image display device described in Example 1 with the optical system of Fig. 14(1) or Fig. 14(2). Specifically, the optical system of Fig. 14(1) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4H, Fig. 4I, Fig. 4J, Fig. 4K, Fig. 4L, or Fig. 4M. In this case, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0236] Specifically, the optical system of Fig. 14(2) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4K, or Fig. 4L. In this case, it becomes possible to increase the amount of space-floating images projecting from the optical system. Also, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0237] Example 3 As a third embodiment of the present invention, an example of a space-floating image display device 1000 will be described. As the basic configuration of the space-floating image display device 1000 of this third embodiment, any of the space-floating image display devices 1000 of the respective drawings described in the first and second embodiments can be applied. In this embodiment, differences from the first and second embodiments will be described, and repeated explanations of the same configurations as those embodiments will be omitted.

[0238] [Summary of Example 3] Fig. 15 shows a display system including a space-floating image display device 1000 as an overview of the third embodiment. The display system of Fig. 15 has a server 2000 on a communication network 1509 and a space-floating image display device 1000 installed in a user environment 1500, which can be connected by communication as appropriate. The user environment 1500 is an environment where a device is installed, such as the home of a user 230, but is not limited to this and may also be a store or public facility. The home of the user 230 has, for example, a LAN 1508 that constitutes a home network, and the space-floating image display device 1000 and the like are connected to the LAN 1508 by communication.

[0239] The server 2000 is an example of an external device 2000 (FIG. 16) for the space-floating image display device 1000, and is not limited to this. The server 2000 may have a three-dimensional model 2001 or image data 2002 as original data of the image to be displayed. The image data 2002 is image data generated by rendering based on the three-dimensional model 2001. The space-floating image display device 1000 may obtain the original data from the server 2000 via communication. When the server 2000 delivers the original data to the space-floating image display device 1000, it may be in the form of live delivery or on-demand delivery.

[0240] 3, the user 230 may operate the space-floating image display device 1000, or may operate it using the remote control 4000 or the mobile terminal 4000 (smartphone, etc.). In other words, in response to the operation of the remote control 4000 or the mobile terminal 4000 by the user 230, information such as instructions may be transmitted from the remote control 4000 or the mobile terminal 4000 to the space-floating image display device 1000 via communication, and the space-floating image display device 1000 may perform an operation based on the instructions.

[0241] Although not shown, the space-floating image display device 1000 may operate in cooperation with other devices, such as a television or a PC, connected to the LAN 1508 of the user environment 1500. The space-floating image display device 1000 may refer to and acquire original data of the image to be displayed from devices such as a television, a PC, or a smartphone 4000.

[0242] In the third embodiment, the original data for displaying the object image 3A as the display target image in the space-floating image 3 may be held by the space-floating image display device 1000, as shown in Fig. 16 described later, or may be held by the server 2000 or other external device 2000. If the original data is held by the space-floating image display device 1000, the space-floating image display device 1000 displays the object image 3A in the space-floating image 3 based on the original data. If the original data is held by the external device 2000 such as the server 2000, the space-floating image display device 1000 receives and acquires the original data from the external device 2000 such as the server 2000, and displays the object image 3A in the space-floating image 3 based on the acquired original data.

[0243] If the original data is a three-dimensional model, the server 2000 or the space-floating image display device 1000 performs rendering processing on the three-dimensional model to generate image data to be displayed on the two-dimensional screen of the space-floating image 3. If the original data is already image data of a two-dimensional image, rendering processing is not necessary.

[0244] In this embodiment 3, the space-floating image display device 1000 performs a process of generating and adding a shadow SH when displaying an object image 3A within the space-floating image 3 based on the original data. In other words, the space-floating image display device 1000 generates image data for displaying the object image 3A, to which an expression / image of a shadow SH has been added, within the screen of the space-floating image 3. This image data is data for displaying an image on the screen of the display unit 1502 (FIG. 16). In this embodiment 3, the space-floating image display device 1000 has a shadow generation function 1510 as a function for generating and displaying a shadow SH within the space-floating image 3 that matches the real light source 400 of the user environment 1500.

[0245] The space-floating image display device 1000 of the third embodiment generates and displays a shadow SH on an object image 3A on the screen of the space-floating image 3, which in this example is a character image based on a 3D model, according to the light of a real light source 400 in a user environment 1500, which is the device installation environment. This shadow generation function 1510 is a function that generates and imparts a representation / image of a shadow SH according to the position and light direction DL of the real light source 400 relative to the display position of the object image 3A of the space-floating image 3. This shadow SH is imparted as a shadow SH at an appropriate position, such as the left / right, front / back, or front / back, relative to the object image 3A, for example, the feet of a character's body, taking into account the position and light direction DL of the real light source 400 relative to the position of the space-floating image 3 or the object image 3A. The real light source 400 is sunlight or indoor lighting, etc. The light direction DL is the direction in which light from the real light source 400 is irradiated onto the object / object image.

[0246] This enhances the sense of depth and realism of the object image 3A when the user 230 views the floating image 3 in space, and the user 230 can enjoy viewing the floating image 3 in space with the shadow SH added.

[0247] The shadow SH in this space floating image 3 is different from the shadow (described later) that is set and generated based on a virtual light source for the original data of the object image 3A. Conventionally, the original data does not take into account the real light source 400.

[0248] Furthermore, this shadow SH is generated as a shadow representation / image that matches the shape and size of the object image 3A within a two-dimensional plane that corresponds to the screen (display range 3R) of the space floating image 3. Furthermore, this shadow SH is calculated to determine whether or not it overlaps in front of or behind the object image 3A (for example, the character's body) in consideration of the position and light direction DL of the real light source 400 relative to the position of the object image 3A (described later).

[0249] Furthermore, in the third embodiment, the generated shadow SH can be broadly classified into two types: one that is imparted as a shadow on the surface of the three-dimensional object of the object image 3A (a second shadow, which will be described later), and one that is imparted so as to be projected onto a virtual floor surface or a physical floor surface (for example, the stage 1190S) assumed for the object image 3A (a first shadow, which will be described later). In the third embodiment, both of these types of shadows are targeted, and at least one of these types of shadows is generated and imparted.

[0250] In particular, the shadow SH (the first shadow described below) that appears to be projected onto the floor surface is merely expressed so as to create a pseudo-visual effect within the plane of the Floating in Space Image 3, and is not projected onto the floor surface like a shadow in the real world. This shadow SH (the first shadow described below) is generated as an expression / image that looks as if it is a shadow projected onto the floor surface (for example, stage 1190S) when the shadow SH displayed within the plane of the Floating in Space Image 3 standing in the vertical direction (Z direction) is viewed from the viewpoint (for example, viewpoint 232) and direction (for example, direction A) of the user 230.

[0251] For the sake of explanation, the following coordinate systems and directions will be used. In FIG. 15, the spatial coordinate system is (X, Y, Z), and the coordinate system in the space floating image 3 is (x, y, z). The X direction / X axis is the first horizontal direction, which is the left-right direction as seen from the user 230. The Y direction / Y axis is the second horizontal direction, which is the depth direction and front-to-back direction as seen from the user 230. The Z direction / Z axis is the vertical direction, which is the up-down direction as seen from the user 230. The x direction / x axis is the horizontal direction within the screen of the space floating image 3, which is the left-to-right direction as seen from the user 230. The y direction / y axis is the vertical direction within the screen of the space floating image 3, which is the up-to-down direction as seen from the user 230. The z direction / z axis is the depth direction and front-to-back direction relative to the screen of the space floating image 3.

[0252] In this third embodiment, the space-floating image display device 1000 uses a sensor 500 such as a camera to estimate the position and light direction DL of the real light source 400 in the device installation environment 1500. Based on the estimated information, the space-floating image display device 1000 generates and imparts a shadow SH to the object image 3A in the space-floating image 3 so that it appears as a shadow according to the position of the real light source 400, etc.

[0253] In this embodiment 3, the method and details of generating and adding the image of the shadow SH to the object image 3A in the space floating image 3 are not particularly limited, but a representative method is a method using a rendering process (FIG. 16). Not limited to this, other methods include a method using an image processing process using a preset shadow image (described later).

[0254] [External device and floating image display device] Fig. 16 shows the basic configuration of the space-floating image display device of Example 3 and Comparative Example. An external device 2000, for example, a server 2000, or the space-floating image display device 1000 generates image data of an object image to be displayed by rendering processing based on a 3D model, which is the original data. This object image to be displayed is a 2D image to be displayed on the 2D screen (in other words, a plane) of the space-floating image 3, like the object image 3A in Fig. 15.

[0255] The comparative example here refers to the configuration and technology for generating an object image based on a 3D model in the server 2000 or the space floating image display device 1000. The comparative example here also refers to the configuration and technology for generating and adding a shadow to an object image when generating an object image based on a 3D model. The conventional shadow generating technology generates a shadow according to a general virtual light source, while the shadow generating technology in this embodiment generates a shadow according to a real light source, as described below.

[0256] 16, server 2000 includes a video processing unit 2011, a database (DB) 2012, and the like. Video processing unit 2011 generates video data 2002 by rendering based on a three-dimensional model 2001. DB 2012 holds a data set 2013 for each video to be displayed. Data set 2013 includes, for example, the three-dimensional model 2001, video data 2002, motion information 2003, virtual light source information 2004, virtual camera information 2005, and audio data 2006. Motion information 2003 is information that defines the motion of an object based on, for example, a skeletal model. Virtual light source information 2004 is information that defines a virtual light source, which will be described later. Virtual camera information 2005 is information that defines a virtual camera, which will be described later. Audio data 2006 is audio data used when audio is output in conjunction with the video output of video data 2002.

[0257] The space-floating image display device 1000 includes an image processing unit 1501, a display unit 1502, an optical system 1503, a user operation detection mechanism 1504, a memory 1505, etc. The image processing unit 1501 generates image data 1602 by rendering based on a three-dimensional model 1601. As part of the rendering process of the image processing unit 1501, a shadow generation process 1511 by a shadow generation function 1510 (FIG. 15) is included. In the shadow generation process 1511, a shadow SH is generated and added to an object image. In one embodiment, the space-floating image display device 1000 generates image data 1602 by rendering based on a three-dimensional model 1601 that is created or registered on its own device and held therein. At that time, a shadow SH is generated and added by the shadow generation process 1511.

[0258] The 3D model 1601 may be data created and stored internally by the space-floating image display device 1000, or may use the 3D model 2001 acquired from the server 2000. Furthermore, the video data 1602 may be generated by rendering by the video processing unit 1501, or may use the video data 2002 acquired from the server 2000. The memory 1505 may store a dataset related to the video to be displayed, similar to the DB 2012 in the server 2000. All or part of the information of the dataset 2013 may be distributed from the server 2000 to the space-floating image display device 1000.

[0259] The image processing unit 1501 displays an image on the screen of the display unit 1502 (having, for example, the display panel described above) based on the image data 1602. Image light emitted based on the image displayed on the screen of the display unit 1502 is optically adjusted via the optical system 1503 (such as the retroreflective member described above), and then forms the space-floating image 3, which is a real image, at a predetermined position. The user 230 performs an operation in the air on the screen (display range 3R) of the space-floating image 3 using an operation object such as a finger. If the user operation detection mechanism 1504 is provided, the user operation detection mechanism 1504 detects the operation in the air using a sensor. The image processing unit 1501 can execute predetermined processing based on the detection information of the user operation detection mechanism 1504. For example, the image processing unit 1501 executes predetermined processing associated with the object image in response to a touch operation in the air on the object image.

[0260] The shadow generation process 1511 performed by the shadow generation function 1510 (FIG. 15) may be configured to be switched on (enabled) / off (disabled) according to settings or instructions from the user 230.

[0261] [Generating shadows through rendering in virtual 3D space] FIG. 17 is an explanatory diagram of the concept of generating, adding, and displaying a shadow representation for an object image 3A of a space-floating image 3 based on the configurations of FIGS. 15 and 16 . The upper part of FIG. 17 shows a schematic diagram of a portion of the rendering process performed by the image processing unit of the server 2000, which is an external device 2000, or the space-floating image display device 1000 in a comparative example. In the comparative example, the image processing unit of the external device 2000 or the space-floating image display device 1000 sets a virtual light source 1702 and a virtual camera 1703 for an object 1701 based on a 3D model of the image to be displayed in a calculated virtual 3D space 1710. The object 1701 is illustrated in the drawing as a 2D image of a certain view, such as a character image. In this example, the object 1701 is set to stand on a virtual floor surface 1704 (VS). The virtual light source 1702 is a calculated temporary light source and is set within the virtual 3D space 1710. Light from a virtual light source 1702 is irradiated onto an object 1701 .

[0262] The video processing unit generates and applies a shadow 1705 to the object 1701 based on the virtual light source 1702. In other words, the shadow 1705 is a representation of a shadow, a shadow image. The shadow 1705 includes at least one of a shadow on the surface of the three-dimensional character body of the object 1701 (a second shadow, which will be described later) and a representation of a shadow projected onto a floor surface 1704 (VS) on which the character, that is, the object 1701, stands (a first shadow, which will be described later). In the example of FIG. 17 , when the object 1701 is viewed from the position of the object 1701, that is, from direction A in FIG. 15 , the virtual light source 1702 is set to a position at the upper left, and the light direction is at the lower right. In accordance with the light from the virtual light source 1702, a shadow 1705 is generated so as to extend to the right at the feet of the object 1701.

[0263] The video processing unit generates a two-dimensional image as a view of these objects 1701 etc. according to the position and orientation etc. of the virtual camera 1703. This two-dimensional image corresponds to the video data 1602 (FIG. 16) for displaying the object image 3A on the screen of the floating-in-space image 3.

[0264] The bottom of Fig. 17 shows a concept as a display 1720 of a screen of a space-floating image 3 in the user environment. On the screen of the space-floating image 3, an object image 1721 (3A) corresponding to the object 1701 is displayed. The space-floating image 3 is formed on a physical floor surface 1724 of the user environment 1500, for example, the upper surface of the stage 1190S of the housing 1190 of the space-floating image display device 1000. An image of a shadow 1725 corresponding to the shadow 1705 is added to the object image 1721 and displayed. When the user 230 looks at the space-floating image 3 from their viewpoint 1723, they can see the object image 1721 with this shadow 1725 added.

[0265] [Difference between virtual and real light sources] Fig. 18 is an explanatory diagram showing that a shadow generated based on a real-world light source (sometimes referred to as a real light source) in device installation environment 1500 differs from a shadow 1725 generated based on a virtual light source 1702 as in the comparative example of Fig. 17. The upper part of Fig. 18 shows an object 1701 and a shadow 1705 in a calculated virtual three-dimensional space 1810 corresponding to Fig. 17, and the lower part shows a real light source 1802, an object 1801, and a shadow 1805 in device installation environment 1500 in the real world. Real light source 1802 corresponds to real light source 400 in Fig. 15, and is, for example, sunlight, external light, or indoor lighting.

[0266] Object 1801 in FIG. 18 is shown schematically, but is assumed to be a real body corresponding to object 1701, a character. The position and light direction of real light source 1802 in FIG. 18 are different from the position and light direction of virtual light source 1702. In the example of FIG. 18, when object 1801 is viewed from direction A in FIG. 15, real light source 1802 is located at the upper right and the light direction is at the lower left. For this reason, the way shadow 1705 is cast in the image of object 1701 based on calculations in the virtual three-dimensional space, as shown in the upper part, does not match the way shadow 1805 of object 1801 in the real world, as shown in the lower part. In the example of FIG. 18, real object 1801 is cast with shadow 1805 extending from the feet to the left on real floor surface 1804.

[0267] The comparative example has the following problems. When a user alone views a space-floating image 3 including an object image with a shadow 1705 generated based on light from a virtual light source 1702, the user does not feel uncomfortable. However, when a user compares a space-floating image 3 with a shadow 1705 with an object 1801 with a shadow 1805 generated based on light from a real light source 1802, as shown in FIG. 18, the shadows are cast differently in the two images, which may cause the user to feel uncomfortable. Although there may be few cases in which an actual comparison is made, there is a risk of a user feeling uncomfortable if they alone view an image such as that shown in the upper part of FIG. 18 (the lower part in FIG. 17) in a real device installation environment 1500.

[0268] FIG. 19 is a schematic diagram illustrating an example in which a floating-in-space image 3 of an object 1701, such as that shown in the upper part of FIG. 18 (the lower part of FIG. 17), is displayed at the position of an object 1801 on a floor surface 1804 in a real device installation environment 1500, such as that shown in the lower part of FIG. 18. In this schematic diagram, when the object image 1721 is viewed from the front, light from a real light source 1802 is irradiated onto the object image 1721 from the upper right to the lower left. In contrast, the image of the shadow 1705 extends to the right of the object image 1721, rather than to the left. In other words, the direction of the light from the real light source 1802 and the direction of the shadow 1705 (the shadow direction) do not match. Therefore, some users may feel uncomfortable when viewing such a floating-in-space image 3 due to the way the shadow 1705 is cast. In other words, the user may feel that the shadow is not very natural. Ideally, it is desirable that the shadow 1705 appears to be cast in roughly the same direction as the light from the real light source 1802 .

[0269] [Example: Shadow generation function and effects] Therefore, in this third embodiment, the following shadow generation function 1510 (FIG. 15) and the resulting effect are realized. In a display system (FIG. 15) including the space-floating image display device 1000 of the third embodiment, when a shadow SH is generated and added to an object image 3A of a space-floating image 3, the shadow SH is generated and added based on a real light source 1802 of FIG. 18 (the real light source 400 in FIG. 15) instead of a virtual light source 1702 (FIG. 17) corresponding to the original data. That is, the space-floating image display device 1000 of the third embodiment has a shadow generation function 1510 that generates, adds, and displays a shadow SH to the space-floating image 3 in accordance with the position and light direction of the real light source 400 in the device installation environment 1500. As a result, the way the shadow is added to an object based on the real light source 400 in the device installation environment 1500 and the way the shadow is added in the space-floating image 3 match or are similar to each other. Therefore, it is possible to reduce the sense of incongruity that the user feels when viewing the space floating image 3 with such a shadow, and it is possible to show a more natural and realistic shadow, thereby improving the image expression.

[0270] [This example: Shadow generation] In contrast to the above comparative example, Fig. 20 shows a conceptual image and method for generating a shadow SH (Fig. 15) as an overview of this embodiment 3. The upper part of Fig. 20 shows a calculated virtual three-dimensional space, and the lower part shows the device installation environment 1500 in the real world. When generating a shadow SH, the space-floating image display device 1000 controls so as not to use information such as the position of the virtual light source 1702 set in the original data for the object image 1701 (3A) to be displayed, or information such as the position of the virtual light source set as a default in the display system including the space-floating image display device 1000 (step S21).

[0271] 16 , when using original data held by server 2000, it is assumed that data set 2013 of the original data has virtual light source information 2004, and information such as the position of virtual light source 1702 is set in virtual light source information 2004. Space-floating image display device 1000 generates shadow SH using the position of real light source 400 (1802) and the like, without using virtual light source information 2004 of the original data.

[0272] Alternatively, the following is also possible. Consider a case where the data set 2013 including the 3D model and video data of the original data does not have the virtual light source information 2004. Even in this case, the space-floating image display device 1000 can generate a shadow SH using the position of the real light source 400 (1802) based on the 3D model or video data of such original data (see FIG. 22 described below).

[0273] In FIG. 20, the space floating image display device 1000 uses the sensor 500 (FIG. 15) to detect and estimate the position and light direction of the real light source 400 in the device installation environment 1500 (step S22).

[0274] Space-floating image display device 1000 sets and uses virtual light source 2052 (referred to as a second virtual light source for the purpose of explanation) corresponding to the position of real light source 400, etc., in the calculational virtual three-dimensional space during rendering processing (step S23). Then, space-floating image display device 1000 generates and imparts shadow SH using the position of real light source 400, etc., i.e., the position of second virtual light source 2052, etc., so that the way shadows are cast in the virtual world and reality will match (step S24). Shadow 2055 (SH) is an expression / image of a shadow generated in accordance with the position of real light source 400, etc.

[0275] FIG. 21 shows the display of a shadow SH (2055) in a space-floating image 3 in the device installation environment 1500, corresponding to FIG. 20 . The position of the real light source 400 and other factors are relative to the position of the object image 3A in the space-floating image 3 formed on the stage 1190S of the space-floating image display device 1000. The shadow SH (2055) is added to the object image 3A of the space-floating image 3 to match the position of the real light source 400 and other factors. Since the light direction from the real light source 400 is, for example, toward the bottom left, the shadow SH (2055) is generated and added to extend from the feet of the object image 3A to the left. Therefore, when viewing this space-floating image 3 from the user's viewpoint 1723, the user is unlikely to feel uncomfortable with the shadow SH (2055). From the user's perspective, a visual effect is achieved in which the shadow SH (2055) appears as if it were created by illuminating the object image 3A with light from the real light source 400.

[0276] [Variation: When there is no virtual light source information] 22 shows an example of generating a shadow from original data when there is no virtual light source information 2004. Image processing unit 1501 of space floating image display device 1000 has 3D model 1601 or image data 1602 as original data of the image to be displayed. Assume that this 3D model 1601 or image data 1602 does not have virtual light source information 2004.

[0277] When using three-dimensional model 1601, as shown in the upper part of Figure 22, video processing unit 1501 sets a second virtual light source corresponding to the position of the real light source for this three-dimensional model 1601 and performs rendering processing to generate video data 1602 (similar to step S23 in Figure 20, etc.).

[0278] The following is the case when rendered video data 1602 is used as the original data. An explanatory diagram is shown at the bottom of Fig. 22. Assume that in the original video data 1602, a shadow 2202 has already been added to an object video 2201 (image A). The space-floating image display device 1000 analyzes this original video data 1602 (image A), extracts an area estimated to be the shadow 2202 from within the image, and erases it (area 2203 in image B). The shadow here is the first shadow. The space-floating image display device 1000 generates and adds a shadow SH (2205) to image B after erasure, using the position of the real light source 400, etc.

[0279] [Concept of shadows in object images] 23 is an explanatory diagram of the concept of a shadow SH applied to an object image in Example 3. Here, as shown in State A, it is assumed that a certain three-dimensional model object 2301 is a simple solid object with a sphere on top of a cylinder. It is assumed that the object 2301 in State A is placed at a predetermined position (x1, y1, z1) on a virtual floor VF in a three-dimensional space (for example, a computational virtual three-dimensional space).

[0280] State B shows object image 2301A to which a shadow SH has been generated and added based on object 2301 in state A. A virtual light source 2302 is set for object 2301. As described above, this virtual light source 2302 corresponds to the second virtual light source (FIG. 20) that is set to match the position of real light source 400, etc. Here, the position of virtual light source 2302 in space is (x2, y2, z2), and the light direction is DL. The coordinate system used here is the coordinate system (x, y, z) of space-floating image 3.

[0281] The shadow SH here includes two types of shadows: a shadow 2320 (also referred to as a second shadow, SH2) that is imparted to the surface of the object 2301 itself based on the virtual light source 2302, and a shadow 2310 (also referred to as a first shadow, SH1) that is imparted to the object 2301 so as to be projected onto the virtual floor surface VF. In this example, when the object 2301 is viewed from the front, the virtual light source 2302 is located at the upper right and the light direction DL is directed to the lower left. In this case, a shadow 2310 (first shadow) is imparted to an object image 2301A corresponding to the spherical or cylindrical object 2301 so as to extend to the lower left on the floor surface VF. Furthermore, a shadow 2320 (second shadow) is imparted to the surface of the object image 2301A corresponding to the spherical or cylindrical object 2301 so as to become darker as it approaches the lower left region.

[0282] The virtual floor VF is a floor on which an object / object image is assumed to be placed, and may be a real physical floor, or may be a virtual floor within the floating-in-space image 3.

[0283] The concept of the shadow SH can be similarly applied to the case where the object 2301 is a three-dimensional object with a complex shape, such as the object image 3A. These first and second shadows are generated and displayed within the two-dimensional image 2300 associated with the space floating image 3.

[0284] In addition, for example, with respect to the shadow 2320 in the lower left region on the surface of the object, it is also possible to give the appearance of gloss or shine to the area opposite to the position of this shadow 2320, i.e., the upper right region where light from the virtual light source 2302 is directly irradiated, as a relatively brighter area.

[0285] This embodiment 3 deals with both of these two types of shadows (first shadow and second shadow). Depending on the detailed embodiment, only one of the shadows may be dealt with, or both shadows may be dealt with simultaneously.

[0286] [Component: Location of original data] A component of the third embodiment is the location of the original data of the image to be displayed. As shown in Fig. 16, there are at least two locations of the original data: inside the space floating image display device 1000 itself, and inside an external device 2000 (for example, the server 2000) to the own device. In either case, the shadow generation function 1510 is applicable.

[0287] The place where the rendering process is performed in this display system may be the external device 2000 or the space-floating image display device 1000. In the third embodiment, the characteristic process including the generation of the shadow SH is performed by the space-floating image display device 1000, but in a modified example, it may be performed by the server 2000 or the like.

[0288] [Component: Contents of original data] The components of the third embodiment include the content of the original data of the image to be displayed. As shown in FIG. 16, the content of this original data can be at least two types: an unrendered three-dimensional model, or rendered image data. In either case, the shadow generation function 1510 can be applied.

[0289] [Component: Type of shadow to generate] The type of shadow to be generated is a component of the third embodiment. As the type of shadow, there are at least two types: a shadow on the floor surface outside the object (first shadow) and a shadow on the surface of the object (second shadow), as shown in FIG. 23 above. Either case is applicable to the shadow generation function 1510.

[0290] [Control Pattern] FIG. 24 is an explanatory diagram summarizing the control patterns in this embodiment 3, in other words, the outline of each detailed embodiment in a table. As shown in the figure, in embodiment 3, there are multiple control patterns depending on the combination of components, and any of them can be applied as an embodiment. The space-floating image display device 1000 is implemented with functions corresponding to one or more control patterns. When multiple pattern functions are implemented, the space-floating image display device 1000 may select the control pattern to be executed by automatic judgment, or the control pattern to be executed may be selected by user instructions or settings.

[0291] In the table of Fig. 24, the components of a combination are "location of original data," "contents of original data," and "type of shadow to be generated." By combining these components, detailed examples as shown in each row (#1 to #10) are possible. Although not shown, examples are also possible by combining other components.

[0292] The "Location of Original Data" column lists the location of the original data of the image to be displayed. In other words, this location is the location where the original data is stored / generated. Patterns #1 to #5 are patterns where the original data is located in the own device, the space-floating image display device 1000 (also written as AD in the table). Patterns #6 to #10 are patterns where the original data is located in the server 2000, which is an external device 2000 relative to the own device (AD). In the latter patterns, the space-floating image display device 1000 receives and acquires the original data from the server 2000, which is the external device 2000, and processes the original data to generate space-floating images including object images (FIG. 16).

[0293] The "Original Data Content" column indicates whether the original data has been rendered from a 3D model. For clarity, an image of the original data is shown. In patterns #1 to #3 and patterns #6 to #8, the original data is an unrendered "3D model" (corresponding to the aforementioned 3D model 2001 or 3D model 1601). In this 3D model object state, neither a shadow (second shadow, SH2) on the object surface nor a shadow (first shadow, SH1) on the floor outside the object is added. In patterns #4 to #5 and patterns #9 to #10, the original data is a rendered "2D image" (corresponding to the aforementioned video data 2002 or video data 1602). In this 2D image state, a shadow (second shadow, SH2) on the object surface is added by rendering based on the aforementioned virtual light source (original virtual light source). Furthermore, patterns #4 and #9 are cases where the rendering does not add a shadow on the floor outside the object (first shadow, SH1), and patterns #5 and #10 are cases where the rendering adds a shadow on the floor (first shadow, SH1).

[0294] The "Type of Shadow to be Generated" column lists the type of shadow to be generated and applied when the Space Floating Image Display Device 1000 generates a Space Floating Image including an object image through rendering processing based on the original data. The type of shadow here can be only the aforementioned first shadow (SH1), only the second shadow (SH2), or both. Pattern #1 applies a shadow on the object surface (second shadow), but does not apply a shadow on the floor surface (first shadow). Pattern #2 applies a shadow on the floor surface (first shadow), but does not apply a shadow on the object surface (second shadow). Pattern #3 applies both a shadow on the object surface (second shadow) and a shadow on the floor surface (first shadow). The same applies to patterns #6, #7, and #8.

[0295] In addition, patterns #4 and #9 are for adding a new floor shadow (first shadow) when the original data, which is a rendered two-dimensional image (object image), already has only the object surface shadow (second shadow).In addition, patterns #5 and #10 are for erasing the floor shadow and replacing it with a floor shadow (first shadow) that corresponds to the real light source when the original data, which is a rendered two-dimensional image (object image), already has the object surface shadow (second shadow) and a floor shadow that corresponds to the virtual light source.

[0296] For ease of understanding, the "Floating Image Outline Image" column shows an image of a shadowed object image in a space floating image, relating to the control of shadow generation in the row.

[0297] [Basic configuration example of a space floating image display device] FIG. 25 shows an example of the basic configuration of a space-floating image display device 1000 in Example 3. In FIG. 25, a schematic cross-sectional view of a housing 1190 as a YZ plane view seen from the side is shown below a perspective view similar to that of FIG. 15. In the configuration example of FIG. 25, components of an optical system such as that shown in FIG. 2D are housed in a box-shaped housing 1190. The housing 1190 is made of a light-shielding material or the like. In this example, the image display device 10 (display device 1 in FIG. 2A) is disposed at the bottom of the housing 1190, and a retroreflective member 5 is disposed obliquely. Image light, which is retroreflected light from the retroreflective member 5, is emitted forward in the Y direction and forms a space-floating image 3 at a predetermined position on a stage 1190S. The stage 1190S is a part of the housing 1190, disposed at the bottom, and extends forward from the front surface of the housing 1190.

[0298] 25, a communication device 1132A including an antenna, which is an example of an implementation of the communication unit 1132, is disposed outside the housing 1190 and is connected to the image display device 10, etc. Note that other components such as the control unit 1110 in FIG. 3 may be disposed inside or outside the housing 1190. Also, in the example of the configuration of FIG. 25, an operation input unit 2501, a speaker 2502, a sensor 500, etc. are mounted on the stage 1190S. The operation input unit 2501 has, for example, physical buttons to enable basic operations. The sensor 500 will be described later.

[0299] In this embodiment 3, a shadow SH is added to the object image in the space floating image 3, and a visual effect is realized as if the shadow SH is projected onto a physical floor surface. Therefore, as an implementation example of the space floating image display device 1000 of the embodiment 3, as shown in the figure, it is a configuration example having a stage 1190S which becomes a physical floor surface, and forming the space floating image 3 on the stage 1190S.

[0300] [Configuration example of a space floating image display device (1)] FIG. 26 shows a more detailed configuration example of the space-floating image display device 1000 based on FIG. 25. FIG. 26 shows a side view, a top view, and a front view. In the configuration example of FIG. 26, the housing 1190 does not have an upper top panel / roof. In addition, in the configuration example of FIG. 26, an aerial operation detection sensor 1351 is installed facing upward on the stage 1190S. The emitted light of the aerial operation detection sensor 1351 passes through the display range 3R of the space-floating image 3 as shown by the arrow, and covers the detection of the display range 3R. In addition, a camera 1180A of the imaging unit 1180 is installed on the top of the housing 1190. The optical axis of the camera 1180A is oriented along the Y axis toward the space-floating image 3 and the user 230 in front, and covers the imaging of the display range 3R and the user 230. A plurality of aerial operation detection sensors 1351 and cameras 1180A may be installed. The mid-air operation detection sensor 1351 and the camera 1180A may be disposed above the floating image 3 in space.

[0301] Furthermore, a sensor 500 is installed on the upper surface of the stage 1190S. The sensor 500 is arranged, for example, on the upper surface of the stage 1190S, in front of the position of the space floating image 3, near the center on the X axis, but other arrangements are also possible. When observing the space floating image 3 in direction A (Y axis) from the user's viewpoint 232, that is, when observing from a point on the normal direction to the xy plane of the space floating image 3, the user can more clearly view the space floating image 3. From the user's perspective, the background of the space floating image 3 appears to be a relatively dark color (black), corresponding to the light-shielding member of the housing 1190.

[0302] [Configuration example of a space floating image display device (2)] Fig. 27 shows another example of the configuration of the space-floating image display device 1000. In the example of the configuration in Fig. 27, components of an optical system like those in Fig. 4M are housed in a housing 1190. Image light emitted in the X direction (for example, left) from the image display device 10 is polarized and converted by the action of the polarization separation member 101B and the retroreflector 2 with the λ / 4 plate 21 described above, and then emitted forward in the Y direction from the polarization separation member 101B, forming a space-floating image 3 at a predetermined position on the stage 1190S.

[0303] [Sensor configuration example (1)] Fig. 28 shows a configuration example of the sensor 500. In Fig. 28, the illustration of the housing 1190 is simplified, and mainly shows only the stage 1190S portion at the bottom of the housing. Note that in Fig. 28, the Y axis is positive toward the front side of the device.

[0304] The sensor 500 is not limited as long as it can detect the position and light direction DL of the real light source 400. The position where the sensor 500 is installed may be any position where it can detect the position and light direction DL of the real light source 400. In this configuration example, corresponding to the configuration example of FIG. 26 etc., the sensor 500 is placed near the center of the front side of the space-floating image 3 on the upper surface 1190S1 (XY plane) of the stage 1190S. The position of this sensor 500 can be considered to be roughly the same as the position of the space-floating image 3 or the object image 3A within the space-floating image 3. In other words, the space-floating image display device 1000 detects and estimates the position and light direction DL of the real light source 400 relative to the position of the space-floating image 3 or the object image in three-dimensional space based on the position of this sensor 500. In other words, the space-floating image display device 1000, particularly the image processing unit 1501, detects and estimates the three-dimensional positional relationship between the space-floating image 3 or the object image and the real light source 400.

[0305] Another example of the installation position of the sensor 500 may be the top surface of the housing 1190. As a simpler configuration, the detection target of the sensor 500 may be the position of the real light source 400 relative to a representative position of the space floating image display device 1000.

[0306] Furthermore, in a modified example, the space-floating image display device 1000 may have a simpler configuration, based on the sensor 500, to detect and estimate the position and light direction DL of the real light source 400 relative to the position of the space-floating image 3 or the object image in a two-dimensional space corresponding to the two-dimensional plane (xy plane) of the space-floating image 3. In other words, the positional relationship in the front-to-back direction (z direction) of the space-floating image 3 may be omitted.

[0307] Furthermore, the installation position of the sensor 500 may be a position away from the position of the floating image 3. Even in this case, if the detection information regarding the position of the real light source 400 based on the position of the sensor 500 is corrected using a setting value of the positional relationship between the position of the sensor 500 and the position of the floating image 3, the positional relationship of the real light source 400 with respect to the floating image 3 in space can be found.

[0308] In other words, this sensor 500 is a photodetector or optical sensor. An example of this sensor 500 may be a camera. This sensor 500 is hardware for detecting and measuring the way light from the real light source 400 hits the position of the space floating image 3, the light intensity distribution, etc. For example, a camera that can capture a hemisphere (a so-called 360° camera) may be used as the camera that is this sensor 500.

[0309] FIG. 29 shows an example of detecting the position and direction of the real light source 400 relative to the object image 3A of the space-floating image 3 in the three-dimensional space of the device installation environment 1500, using front and top views. Here, the position of the object image 3A is designated as point P (X1, Y1, Z1), and the position of the real light source 400 is designated as point R (X2, Y2, Z2). Instead of the position of the real light source 400, the light direction DL from the real light source 400 may be detected or estimated. Here, the light direction DL is the direction corresponding to the line from the position R of the real light source 400 to the position P of the object image 3A. In this example, as shown in the figure, in the XZ plane view, the position R of the real light source 400 can be detected to the right of the X axis and above the Z axis with respect to the position P of the object image 3A, and the light direction DL is in the lower left direction. In the XY plane view, the position R of the real light source 400 can be detected to the right of the X axis and behind the Y axis with respect to the position P of the object image 3A, and the light direction DL is in the forward left direction.

[0310] In the aforementioned modified example, when detecting the position and light direction DL of the real light source 400 in a two-dimensional space corresponding to the two-dimensional plane (xy plane) of the space floating image 3, the sensor 500 does not need to perform detection in the Y-axis direction, which is the depth direction. In this case, in the example of Fig. 29, the position R of the real light source 400 in the Y-axis direction is considered to be the same as the position P of the object image 3A, the position coordinate is set to 0, and the light direction DL is simply detected as the lower left direction as shown in the front view.

[0311] [Sensor configuration example (2)] FIG. 30 shows another example of the configuration of the sensor 500. FIG. 30 shows a case where a plurality of photodetectors (cameras) is used as the sensor 500 to detect the position or light direction DL of the real light source 400. In this example, the sensor 500 has three photodetectors / cameras 500a, 500b, and 500c. The names / IDs for identification are Camera 1, Camera 2, and Camera 3. The stage 1190S has a roughly rectangular parallelepiped shape. A first camera 500a is installed in front of the stage 1190S with its optical axis facing forward. A second camera 500b is installed on the left side of the stage 1190S with its optical axis facing left. A third camera 500c is installed on the right side of the stage 1190S with its optical axis facing right.

[0312] Note that the front camera 500a can be used for other purposes, such as taking pictures of the user 230 in front of the screen or detecting human presence, in addition to its use as the sensor 500. The number of cameras is not limited to three, and may be two, four, or more. A rear-facing camera may be added to the back of the housing 1190.

[0313] The main roles of these three cameras are as follows: The front camera 500a mainly detects and acquires real light source information on the XZ plane (X and Z directions) in the +Y direction (front side) of the space floating image display device 1000 (here, the stage 1190S). The left side camera 500b mainly detects and acquires real light source information on the YZ plane (Y and Z directions) in the -X direction (left side). The right side camera 500c mainly detects and acquires real light source information on the YZ plane (Y and Z directions) in the +X direction (right side).

[0314] Fig. 31 is an explanatory diagram of a method for acquiring the position / light direction of the real light source 400 using the sensor 500 (cameras 500a, 500b, 500c) of Fig. 30. The space floating image display device 1000, for example, the image processing unit 1501 of Fig. 16, determines the position or light direction DL of the real light source 400 based on the detection information from the three cameras. Fig. 31 shows a top view. The origin of the spatial coordinate system (X, Y, Z) is O, and here, the center position of the formation of the space floating image 3 is taken as the origin O.

[0315] In this example, three cameras are used to capture the brightness distribution around the space-floating image 3 of the space-floating image display device 1000. The image / area captured by camera 500a corresponds to the XZ plane, and for ease of understanding, the right (R) and left (L) as seen from camera 500a are also shown. Image / captured area 3101 shows the brightness distribution in the image / captured area captured by camera 500a. Image / captured area 3102 shows the brightness distribution in the image / captured area captured by camera 500b. Image / captured area 3103 shows the brightness distribution in the image / captured area captured by camera 500c.

[0316] Next, the image processing unit 1501 of the space floating image display device 1000 estimates the position / light direction DL of the real light source 400 by a predetermined process (such as a real light source position estimation algorithm) based on the brightness distribution information of the images (3101, 3102, 3103) of the three cameras in Fig. 31. In this process, the image processing unit 1501 determines the positional relationship between the display position of the space floating image 3 (particularly the object image) and the position of the real light source 400.

[0317] FIG. 31 also shows an example of the estimated position and light direction DL of real light source 400. As a simple processing example, it can be seen that image / imaging range 3103 of the brightness distribution (light intensity distribution) of camera 500c is relatively brightest, and that the brightness increases toward the upper left within the YZ plane of image / imaging range 3103. From this, image processing unit 1501 can infer that the position of real light source 400 is to the right (+X) on the X axis, behind (-Y) on the Y axis, and above (+Z) on the Z axis relative to the position of floating-in-space image 3 (origin O). In other words, it can be inferred that real light source 400 is located to the right, behind, and above origin O, and that the light direction DL from real light source 400 is to the left, front, and below.

[0318] Next, the image processing unit 1501 performs a process of generating and adding a shadow SH to the object image of the Floating in Space image 3 based on the position / light direction DL of the real light source 400 calculated by the predetermined process described above, as a shadow generation process 1511 (FIG. 16) that is part of the rendering process described above. In other words, the image processing unit 1501 creates image data for displaying the Floating in Space image 3 in which a shadow SH corresponding to the position / light direction DL of the real light source 400 has been added to the object image. For example, the shadow SH is added to the object image 3A in the xy plane of the Floating in Space image 3 in correspondence with the left, front, and bottom directions as the light direction DL from the real light source 400, as shown in FIG. 21, as a shadow image extending in a left-bottom direction from the position of the object image 3A (point P in FIG. 30, origin O in FIG. 31), for example, the feet of the character. Then, the image processing unit 1501 controls the display unit 1502 based on the image data to display the Floating in Space image 3 in which the shadow SH has been added to the object image. The display of this floating image 3 in space will be similar to that shown in FIG. 21, for example.

[0319] [Prescribed processing: Real light source position estimation algorithm] The following algorithm can be applied as an estimation algorithm for the real light source position in the above-mentioned predetermined processing: The image processing unit 1501 performs estimation based on the brightness distribution of each camera image, and performs estimation and verification by combining these multiple estimation results, and finally estimates the position / light direction of the real light source 400 from the verification results.

[0320] For example, three cameras as the sensor 500 in FIGS. 30 and 31 acquire imaging information (images) at each time point. From these camera images / imaging information, it is possible to grasp the brightness distribution on the imaging plane of each of at least three cameras. Although the positions of the cameras are different, the reference position of the sensor 500 can be considered to be the same as the origin O. Alternatively, correction may be performed based on the setting values ​​of the positional relationship of each camera. The image processing unit 1501 analyzes the acquired images / imaging information of the three cameras and estimates the position of the real light source 400. In the example of FIG. 31, from the image 3101 of camera 1, it can be inferred that the light source is closer to the +X direction and the +Z direction from the position of camera 1. From the image 3102 of camera 2, it can be inferred that the light source is closer to the -Y direction and the +Z direction from the position of camera 2. From the image 3103 of camera 3, it can be inferred that the light source is closer to the -Y direction and the +Z direction from the position of camera 3. It should be noted that the tips of the arrows →X(x), →Y(z), and →Z(y) in FIG. 31 point in the positive directions of the X-axis, Y-axis, and Z-axis, respectively.

[0321] Furthermore, image processing unit 1501 performs estimation and verification by combining these pieces of information. For example, image 3103 of camera 3 has a wider brightness distribution range or greater brightness (light intensity) than image 3102 of camera 2, and therefore it is estimated that the light source is closer to camera 3 than to camera 2. This estimation does not contradict the estimation based on image 3101 of camera 1 (that the light source is closer to the +X direction (right side) of camera 1). From these facts, it is estimated that the position of real light source 400 is to the right, behind, and above origin O, as shown in FIG. 31.

[0322] [Sensor configuration example (3)] FIG. 32 shows another example configuration of the sensor 500. In FIG. 32, the sensor 500 includes only one photodetector / camera 500A. This example configuration can be realized at low cost. In this example, the camera 500A is installed in front of the stage 1190S, with its optical axis facing forward. The image processor 1501 uses the brightness distribution of the image / imaging range acquired by the camera 500A to estimate the position / light direction DL of the real light source 400, as in FIG. 31. This estimation process is simpler than in FIG. 31, but it is still possible to estimate the position / light direction DL of the real light source 400 to a certain extent. Furthermore, in the case of FIG. 32, the estimation of the positional relationship with the real light source 400 may be limited to only the range forward of the space-floating image 3 on the Y axis.

[0323] In the example of Figure 32, from the brightness distribution of the image / capture range 3201 of camera 500A, it can be inferred that the relatively bright parts are to the right in the image, and therefore that real light source 400 is in the -X direction (left side) on the X axis relative to camera 500A and space floating image 3.

[0324] If the accuracy of estimation based on one camera or multiple cameras as sensor 500 is insufficient, that is, if it is not possible to estimate the position of real light source 400, estimation may be made using other information, or estimation may be made by combining camera image information with other information. For example, the following method can be applied.

[0325] [Speculation using other information] First Example: The space-floating image display device 1000 acquires location / position information, date and time information of the installation location of the space-floating image display device 1000 itself using a GNSS receiver or communication. The space-floating image display device 1000 estimates the position and light direction of the real light source 400 (e.g. sunlight) at that location / position and time based on the acquired information and publicly known information such as weather information. The space-floating image display device 1000 may also make estimations by combining the information acquired in this way with camera image information.

[0326] Second Example: The space-floating image display device 1000 may use an attitude sensor, an orientation sensor, etc. to detect the installation direction and attitude of the space-floating image display device 1000 itself (particularly the space-floating image 3). The space-floating image display device 1000 estimates the position and light direction of the real light source 400 (for example, sunlight) based on the detected information or based on a combination with camera image information.

[0327] [When the real light source is indoor lighting] When the real light source 400 in the user environment 1500 is an indoor light, the following method can be applied. In order to grasp the positional relationship between the user's own device and the indoor light, the space-floating image display device 1000 acquires information representing that positional relationship or information capable of calculating that positional relationship by any means. For example, in the configuration of FIG. 15 , the space-floating image display device 1000 communicates with an indoor lighting device (e.g., an IoT-compatible lighting device) installed in the user's home 230, or with another system or device that manages the indoor lighting device. The space-floating image display device 1000 acquires, from these devices, information representing, for example, the position / light direction of the indoor light, information representing the state of operation (ON / OFF), the intensity of the lighting, etc. Based on the acquired information, the space-floating image display device 1000 estimates the position / light direction of the indoor light relative to the user's own device (FIG. 44 described below).

[0328] [Processing flow] Fig. 33 shows the overall processing flow relating to the shadow generation function 1510 (Fig. 15) of the space-floating image display device 1000. In step S301, the space-floating image display device 1000, particularly the image processing unit 1501 (Fig. 16), performs a space-floating image determination process to determine the image (image to be displayed) to be displayed as the space-floating image 3. This process corresponds to, for example, specifying the three-dimensional object 1601 or image data 1602 that will be the original data of the image to be displayed based on the operation and instruction of the user 230 in Fig. 16. The image processing unit 1501 may ascertain the display position of the object image within the display range 3R of the space-floating image 3 based on the original data.

[0329] In step S302, the image processing unit 1501 performs a process of acquiring information detected by the sensor 500 (for example, the multiple cameras in FIG. 30) (sensor detection information acquisition process). This sensor detection information is, for example, camera image information such as in FIG. 31. Note that, in step S302, as described above, information other than that of the sensor 500 (for example, location information, etc.) may be acquired.

[0330] In step S303, the image processing unit 1501 performs processing to estimate the position / light direction DL of the real light source 400 (real light source position estimation processing) based on the sensor detection information in step S302.

[0331] Note that if a contradiction occurs in the estimation result when the position etc. of real light source 400 is estimated based on the sensor detection information in step S303, in other words, if the position etc. of real light source 400 cannot be estimated, the process may return to step S302 to acquire the sensor detection information again and repeat the estimation. Furthermore, if the estimation cannot be realized after several repetitions, the process may end with an error, or a virtual shadow (a shadow that does not take into account the position etc. of the real light source) may be added, or control may be exercised to not generate a shadow.

[0332] Furthermore, if it is difficult to make an estimation in step S303, as described above, other information (such as location information) may be acquired and estimation may be attempted using a method that uses the other information. Alternatively, estimation may always be performed using other information from the beginning, or estimation may always be performed by combining the sensor detection information with other information from the beginning.

[0333] In step S304, the image processing unit 1501 performs processing to generate an object image based on the original data. Here, it is assumed that the original data is a three-dimensional object 1601, and rendering processing is performed. This rendering processing includes the shadow generation processing 1511 described above. In this shadow generation processing 1511, the image processing unit 1501 determines information (shadow image generation information) for generating an image of a shadow SH to be added to the object image in the space floating image 3, based on the positional relationship between the position / light direction of the estimated real light source 400 and the display position of the space floating image 3 (particularly the object image). The image processing unit 1501 determines the position, shape, size, etc. of the shadow SH to match the position, shape, size, etc. of the object image.

[0334] This process is not limited to the method of generating a shadow SH as part of the rendering process (FIG. 20, the method using a second virtual light source), but may also apply a method of generating a shadow SH by selecting and processing a similar or appropriate preset shadow image instead of rendering (FIG. 34). When generating a shadow by rendering calculation, a more realistic shadow can be generated. When using a preset shadow image, the calculation load can be reduced.

[0335] The image of the shadow SH generated in step S304 is the position, direction, shape, size, etc. of the shadow that would be generated when light from the estimated position of the real light source 400 is irradiated onto the object image (e.g., character image) represented as the floating image in space 3, assuming that the object image exists at the display position of the floating image in space 3 as a real object.

[0336] In step S305, the image processing unit 1501 controls the display unit 1502 based on the image data in which the shadow image is added to the object image, thereby displaying the space floating image 3.

[0337] Step S306 is a confirmation as to whether to end the display of the floating in space image 3, and if to end (Yes), this flow ends, and if not to end (No), proceed to step S307. Step S307 is a confirmation as to whether to update the display of the floating in space image 3, and if to update (Yes), return to step S301, and if not to update (No), return to step S306. Examples of display updates include screen transitions, movement of object images, execution of processing in response to manipulation of object images in the air, changes in the state of light from the real light source 400, etc.

[0338] In addition, when the original data is rendered video data 1602, the above flow determines the video data 1602 to be displayed in step S301, and in step S304, there is no rendering process and a shadow is generated for the video data 1602, in other words, video data 1602 is created with a shadow added to the object video.

[0339] [Variation: Preset Shadow Image] 34 and 35 show a configuration example in which a shadow SH is generated using a preset shadow image as a modified example of the third embodiment. In Fig. 34, the space-floating image display device 1000 has a three-dimensional model 1601 or image data 1602 as the original data of the image to be displayed. The original data may be obtained from the server 2000 as described above. In the case of the three-dimensional model 1601, the image data 1602 is obtained by rendering processing. In the configuration example of Fig. 34, the space-floating image display device 1000 has a preset shadow image 3401 stored in the memory 1505 in advance.

[0340] In the initial video data 1602, no shadow (the first shadow described above) is added to the object video. The video processing unit 1501 generates and adds a shadow SH to the object video in this video data 1601 using the preset shadow image 3401. The video data obtained as a result of this processing is called video data 1602B.

[0341] FIG. 35 shows an example of shadow generation processing using a preset shadow image 3401. The original object image 3A in the two-dimensional image of the video data 1602 does not have a shadow SH. The video processing unit 1501 calculates the positional relationship of the real light source 400, the light direction DL, and other factors relative to the position of the space-floating image 3 or the position of the original object image 3A. This calculation may be realized by estimating the positional relationship using the sensor 500 in step S303 of FIG. 33. This calculation may be a calculation of the positional relationship in three-dimensional space, but as a simpler method, it may be a calculation of the positional relationship in two-dimensional space. In the example of FIG. 35, the position and light direction of the real light source 400 relative to the position of the object image 3A in the two-dimensional image are considered in two-dimensional space. For example, considering the xy plane corresponding to the space-floating image 3, the real light source 400 is calculated to be located roughly to the upper right of the object image 3A shown, and the light direction DL is calculated to be in the lower left direction. Even such a simple calculation has a considerable effect.

[0342] The image processing unit 1501 selects one of the preset shadow images 3401 to be used based on the positional relationship between the object image 3A and the real light source 400. The preset shadow image 3401 to be selected may be selected based on a user setting or instruction.

[0343] A plurality of preset shadow images 3401 are prepared according to, for example, the shape, color, pattern, and presence or absence of contours of the shadow. In the illustrated example, a preset shadow image 3412 with a cloud shape, gray color, and no contours is selected as one preset shadow image. The image processing unit 1501 appropriately processes the preset shadow image 3412 selected from the preset shadow images 3401, and places it as a shadow SH (3402) on the object image 3A in a two-dimensional image that simulates the screen of the space floating image 3. The image processing unit 1501 adjusts the position, orientation, size, etc. of the shadow SH (3402) to match the position of the real light source 400, etc., with respect to the object image 3A as closely as possible. Image processing includes enlarging / reducing the size, rotating, trimming, etc.

[0344] The placement of this shadow SH (3402) also includes calculations regarding the overlapping relationship, such as whether the shadow SH (3402) should be placed in front of or behind the object image 3A in the z direction. In the example shown, the shadow SH (3402) is placed so as to extend to the left of the feet of the object image 3A. The shadow SH (3402) is also placed so as to overlap behind the feet of the object image 3A.

[0345] Next, we will explain the calculations and techniques used to effectively generate and add a shadow SH image to the object image in the space floating image 3 on the stage 1190S.

[0346] [Position of object image in the device installation environment] 36 is a schematic perspective view of a space-floating image 3 formed on the upper surface 1190S1 of a stage 1190S, seen obliquely from above and in front. In the space-floating image display device 1000, a predetermined position is designed in three-dimensional space to form the xy plane, which is the screen (display range 3R) of the space-floating image 3. In this example, this predetermined position is a predetermined position on the upper surface 1190S1 of the stage 1190S. In the example of FIG. 36, the xy plane of the space-floating image 3 is positioned in the vertical direction (y), approximately halfway in the depth direction (z direction) of the stage 1190S (see also FIG. 37).

[0347] In the calculation for generating the shadow SH, it is assumed that the object OB to be displayed is placed within the plane of the Floating in Space Image 3 at a predetermined position in the three-dimensional space of the device installation environment 1500, as in state A. The object OB corresponds to the object image OBG. In particular, it is assumed that the object OB / object image OBG is placed within the xy plane of the Floating in Space Image 3, which is placed on the top surface 1190S1 of the stage 1190S, which is the physical floor surface. For the calculation, the position of this object OB within the three-dimensional space is set. Here, the position coordinates of the object OB are set to (x1, y1, z1) of point P. Point P is the position where the object OB is assumed to be standing on the virtual floor surface, and is within the xy plane of the Floating in Space Image 3.

[0348] Note that a distinction is made between the stage top surface 1190S1, which is the physical floor surface, and a virtual floor surface that is calculated. The position of the virtual floor surface may coincide with the position of the physical floor surface, or the position of the virtual floor surface may not coincide with the position of the physical floor surface. For example, the virtual floor surface may be set at a predetermined distance above the physical floor surface (see FIG. 37, which will be described later).

[0349] Point R is a hypothetical position of the real light source 400 or the corresponding light source light, and the position coordinates of point R are (x2, y2, z2). In this example, it is assumed that light from the real light source 400 is irradiated from the upper right to the position (point P) of object OB. In other words, the light direction DL is toward the lower left. Image processing unit 1501 generates a shadow SH based on a calculation of the position of object OB / object image OBG in device installation environment 1500, such as state A.

[0350] State B shows the state of the object image OBG formed by generating and applying a shadow SH (3605) corresponding to the light direction DL of the light from the real light source 400 through calculations during rendering to the object OB in state A. Within the two-dimensional plane of the space floating image 3, the object image OBG displays a shadow 3605 as a representation / image of a shadow corresponding to the light direction DL of the real light source 400. An example of this shadow 3605 includes the first shadow (SH1) and second shadow (SH2) in FIG. 23.

[0351] In a modified example, the calculations as described above may be omitted / simplified, and control may be performed based on the relationship between the representative position of the space-floating image 3 in the device installation environment 1500 (for example, the center point of the screen) and the position of the real light source 400, or based on the relationship between the representative positions of the space-floating image display device 1000 or sensor 500 and the position of the real light source 400.

[0352] In this embodiment, the housing 1190 of the space-floating image display device 1000 has a stage 1190S as an example of implementation, but this is not limited to this. Implementation without an object such as the stage 1190S is also possible. In that case, an object such as a desk or table on which the space-floating image display device 1000 is placed in the device installation environment 1500 may serve as the physical floor surface.

[0353] [Relationship between the stage surface and the floating video screen] Fig. 37 is a supplementary explanatory diagram for Fig. 36, showing an XZ plane view as a front view, a YZ plane view as a side view, and an XY plane view as a top view. Fig. 37 also shows a configuration example regarding the design of the arrangement of the space-floating image 3 in the space-floating image display device 1000. Fig. 37 shows the positional relationship between the upper surface 1190S1 of the stage 1190S and the screen (xy plane) of the space-floating image 3.

[0354] 37, the bottom side of the screen (display range 3R) of the floating-in-space image 3 is placed above the stage upper surface 1190S1 in the vertical direction (Z direction), with a gap of a predetermined distance DZ1 provided. This distance DZ1 is not particularly limited.

[0355] The configuration example in Fig. 37 is an example in which this distance DZ1 is designed to be a relatively small value. In this case, a visual effect can be achieved in which the object image (for example, a character) in the space floating image 3 appears to be standing on the stage 1190S. Also, in this case, a visual effect can be achieved in which the shadow SH given to the object image appears as if it were projected onto the stage 1190S.

[0356] 37, a virtual floor VF is set at a position a short distance DZ2 above the bottom edge in the xy plane of the screen. The position of the bottom edge (point P) of the object OB is set at the position of the virtual floor VF. Here, this bottom edge position (point P) is set as the representative position of the object OB. However, the representative position of the object OB may be another position.

[0357] By setting a virtual floor VF (corresponding distance DZ2) within the xy plane of the screen of the Floating in Space Image 3, an area from the bottom edge to the virtual floor VF is provided within the screen. This makes it possible to secure a space within the screen to display a suitable shadow SH. There are no particular limitations on this distance DZ2 and the virtual floor VF.

[0358] Figure 38 shows a modified example of Figure 37. State A in Figure 38 shows a configuration example in which the distance DZ1 is designed to be a relatively large value. Such a configuration is also possible. In this case, the shadow SH applied to the object image OBG will appear as if it were projected onto an invisible plane located above the stage top surface 1190S1.

[0359] State B in Fig. 38 shows a configuration example in which the distance DZ2 of the virtual floor VF within the screen of the Space Floating Image 3 is designed to be a relatively large value. Such a configuration is also possible. In this case, the shadow SH given to the object image OBG will appear as if it is projected onto an invisible plane located above the stage top surface 1190S1.

[0360] 38, the visual effect of the shadow SH being projected onto the stage upper surface 1190S1 is weakened in the generation and display of the shadow SH, but this configuration may be used in cases where it is acceptable to have the shadow SH (first shadow) appear to be floating in the air. Also, this configuration may be used in cases where, for example, a shadow extending outward from the character's feet (first shadow) is not necessary and only a shadow on the object surface (second shadow) is sufficient.

[0361] [Shadow position, etc.] FIG. 39 shows display variation examples regarding the position, shape, size, etc. of the shadow SH when a shadow SH according to a real light source 400 is added and displayed on an object image OBG in an xy plane view, which is a front view of the floating in space image 3. In this example, a virtual floor VF is set at a predetermined position within the xy plane of the floating in space image 3. Also, for a character image, which is an example of the object image OBG (3A), a point P at a representative position is set on the virtual floor VF. In other words, point P is at the feet of the character image. The virtual floor VF may be a straight line extending in the x direction, or may be set to be a slope when viewed from the user (z direction) as shown in the figure.

[0362] In this example, the positional relationship of the real light source 400 with respect to the object image OBG is as shown in the figure. In state A, the position of the real light source 400 with respect to the object image OBG is to the right in the x direction, above in the y direction, and behind in the z direction. In other words, the light direction DL of the light irradiated from the real light source 400 onto the object image OBG is to the left, below, and in front. Here, this positional relationship is considered in three-dimensional space (x, y, z). The object image OBG corresponds to the object OB described above.

[0363] In this example, consider the case where a first shadow is added to the object image OBG. In state A, in accordance with the positional relationship with the real light source 400, the shadow 3901 (SH1) is generated in a position and shape that extends to the left, bottom, and front with respect to the position (point P) of the object image OBG. Furthermore, this shadow 3901 may be positioned so that it overlaps in front of the character body (e.g., feet) of the object image OBG on the z-axis. In this example, there is a space corresponding to the distance DZ2 described above within the plane of the floating in space image 3, so such a shadow 3901 can be suitably positioned. What to do if such a space does not exist will be described later.

[0364] State B is an example of another way of casting a shadow. In the example of state B, the position of the real light source 400 is assumed to be left, upper, and front with respect to the object image OBG, and the light direction DL is assumed to be right, lower, and rear. Correspondingly, the shadow 3902 (SH1) is positioned and shaped to extend to the right, lower, and rear with respect to point P. Furthermore, this shadow 3902 may be positioned so as to overlap the rear side of the character's body (e.g., the feet) of the object image OBG.

[0365] As in the above example, different types of shadow SH can be added and displayed depending on the positional relationship of the real light source 400 with respect to the object image OBG.

[0366] [Shadow color, size, etc.] FIG. 40 shows another example of a display variation. In this example, in state A and state B, the light direction DL of the real light source 400 relative to the object image OBG is in a direction to the lower left. Also, in state A and state B, the distance between the object image OBG and the real light source 400, or the brightness / intensity of the light from the real light source 400, differs. For example, state A is brighter. In response to this difference, in state A, the generated shadow SH (4001) is generated as a darker, more noticeable shadow. For example, in state A, the shadow 4001 is generated in a color (shown in black in the drawing) that has a larger luminance difference from the background (e.g., black) of the Floating in Space image 3, and is larger in size. In state B, the shadow 4002 is generated in a color (shown in gray in the drawing) that has a smaller luminance difference from the background (e.g., black) of the Floating in Space image 3, and is smaller in size. In the drawings, the background is white, so the shadow SH is shown in black or gray, but in the actual floating in space image 3, the background is, for example, black, so it would be preferable to express or display the shadow SH in a bright color such as white, or a color with high luminance. As a variation, for example, if the real light source 400 is the sun, the angle at which the light is irradiated onto the object image OBG and the brightness / intensity of the light change depending on the time of day, season, and weather conditions, so by adjusting the direction, length (size), and color of the generated shadow SH according to these conditions, it is possible to further reduce the sense of incongruity and achieve a more realistic image representation.

[0367] [Adjust the space in which shadows are displayed] Fig. 41 shows a display example in which, within the xy plane that is the screen of the floating in space image 3, an object image OBG (e.g., a character) is placed near the bottom edge of the screen, and there is not much space to place a shadow SH. In state A, the bottom edge of the screen is designed to be as close as possible to the stage top surface 1190S1, and the aforementioned virtual floor surface VF is not provided. Based on the original data, the feet of the character in the object image OBG are placed near the bottom edge of the screen. The light direction DL from the real light source 400 is assumed to be in a direction to the lower left.

[0368] In this case, suppose that it is desired to place a shadow 3901 as shown in Fig. 39 as a shadow SH on the object image OBG. However, there is not enough space on the screen, so it is not possible to place this shadow 3901. If the shadow 3901 is forcibly placed, it will become a shadow 4101 as shown in the figure, and in this case only a part of the shadow 3901 will be displayed.

[0369] Even in this case, a shadow SH can be placed in the area above the character's feet that touch the bottom edge of the screen, so if the position of the object image on the screen is fixed and cannot be moved, the shadow SH can be placed in the area above the bottom edge of the screen.

[0370] Therefore, as a countermeasure, the image processing unit 1501 may generate a shadow SH (4102) as shown in state B. The image processing unit 1501 adjusts the position at which the object image OBG is placed on the screen of the Floating in Space image 3 based on the original data as needed. Specifically, the image processing unit 1501 sets a virtual floor VF (in other words, the height position from the bottom edge of the screen) so as to ensure, for example, a distance DZ2 within the screen, and shifts the placement position of the object image OBG (for example, point P) from its original position to a position above on the y-axis in accordance with the virtual floor VF. Alternatively, the distance DZ2 may be ensured by, for example, reducing the size of the object image OBG. The image processing unit 1501 uses the space thus secured to generate and impart the shadow SH (4102).

[0371] Fig. 42 shows an example in which a shadow SH is placed using an area above the bottom edge of the screen in the case of state A in Fig. 41. In this example, the shadow SH (4201) is generated in an area above the bottom edge of the screen, with a position and shape that extends from the feet of the character in the object image OBG to the left. This shadow SH (4201) is less accurate in terms of its positional relationship with the real light source 400 than in the case of state B in Fig. 41, but because the relationship between the light source and the shadow is consistent at least in the left-right direction, it has a reasonable effect as a simple shadow representation.

[0372] [Relationship between background and shadow] 43 shows an example of generation and display of a shadow SH in relation to the background of the space-floating image 3 when viewing an object image 3A on the screen of the space-floating image 3 in the Y direction (z direction) from the user's viewpoint, similar to the example of FIG. 26 etc. As described above (FIG. 13A etc.), when the inside of the housing 1190 is sufficiently dark (in other words, when it is light-blocked), the user will view the background of the space-floating image 3 as black.

[0373] In state A of FIG. 43, in the front view, the background 4301 of the screen of the floating in space image 3 appears black to the user. The general conceptual image of a shadow in the real world is black, but if the shadow SH is black, it will blend into the black of the background 4301 and become difficult to see. Therefore, in this embodiment, when generating the shadow SH on the screen, an appropriate shadow SH is generated taking into consideration the influence of overlap with the background. Specifically, when generating the shadow SH in the object image 3A, the image processing unit 1501 selects and adjusts the color, brightness, etc. of the shadow SH taking into consideration the state (e.g., black) of the background 4301 of the screen. In this example, the shadow 4302 is selected and adjusted to be, for example, gray so that it can at least be distinguished from the black of the background 4301 by the user.

[0374] State B is another example. In contrast to the similar background 4301, the shadow 4303 is selected to be white so that there is a greater difference in brightness.

[0375] Moreover, state C is a modified example. In state C, the background 4304 is a display screen of the above-mentioned second display device 1680 or the transmissive self-luminous image display device 1650 (FIG. 3, etc.) on the screen of the space floating image 3. On this display screen, for example, the background image is a solid color of a light color such as white. In this case, when generating a shadow SH in the object image 3A, the image processing unit 1501 selects and adjusts the color, brightness, etc. of the shadow SH taking into consideration the state (for example, white) of the background 4304. In this example, the shadow 4305 is selected and adjusted to be, for example, black in contrast to the white of the background 4304.

[0376] Any image can be displayed as background 4304 on the display screen of second display device 1680 or transmissive self-luminous image display device 1650, etc. For example, a background image of a stage on which object image 3A (e.g., a character) is standing, or of art, direction, etc., may be displayed on this display screen. In this case, the representation / image of shadow SH can be selected / adjusted taking into consideration the relationship with the background image of such a display screen.

[0377] [Linkage with indoor lighting devices] A supplementary explanation will be given regarding the cooperation with the indoor lighting device mentioned above. The space-floating image display device 1000 may cooperate with home appliances such as indoor lighting devices and systems that manage and control them as external devices in the device installation environment 1500. Examples of systems / devices that manage and control home appliances include smart remote controls, smartphones, and HEMS (Home Energy Management Systems). In a modified example, the space-floating image display device 1000 may be provided with a function to control and manage home appliances.

[0378] FIG. 44 shows a configuration example in which the space-floating image display device 1000 cooperates with home appliances such as an indoor lighting device and an air conditioner. The home of user 230, which is device installation environment 1500, has home appliances 4400 such as a lighting device 4401, an air conditioner 4402, and an electric fan 4403. The home may also have a management system 4410 that automatically manages and controls the home appliances 4400, or a smart remote control 4420 that can manage and operate the home appliances 4400. The space-floating image display device 1000 cooperates with the home appliances 4400, or the management system 4410 or the smart remote control 4420, via a communication device 1132A of the communication unit 1132, and exchanges information. In FIG. 44, the lighting device 4401 corresponds to the real light source 400. The space floating image display device 1000 acquires information such as the position and state of the lighting device 4401 from, for example, the lighting device 4401, and based on the acquired information, makes inferences and judgments regarding the position and state of the above-mentioned real light source 400. The image processing unit 1501 may perform control so as to enable the function of generating a shadow SH when the operation of the target home appliance 4400 is turned on, based on the operation ON / OFF information acquired from the home appliance 4400.

[0379] [Real light source settings] In the above embodiment, the state and changes of the position, light direction DL, brightness, etc. of the real light source 400 relative to the space-floating image display device 1000 in the device installation environment 1500 can be grasped by the sensor 500 or communication, etc., and a shadow SH can be generated according to the state of the real light source 400. Not limited to this, in a modified example, the positional relationship, etc. of the real light source 400 relative to the space-floating image display device 1000 in the device installation environment 1500 may be set in advance by initial setting or user setting.

[0380] FIG. 45 shows a modified example in which the positional relationship of the real light source 400 can be set as a user setting for the shadow generation function 1510 in the graphical user interface (GUI) of the screen of the space-floating image 3 of the space-floating image display device 1000. In this example, a guide such as "Please set the position of the light source" is displayed on the screen, and a schematic diagram of the space-floating image 3 on the stage 1190S and the real light source 400 relative to it is also displayed. The user can move the icon of the real light source 400 by operating the cursor or by manipulating in the air, and set it to the desired position. Alternatively, the position of the real light source 400 may be set numerically. The setting of the position of the real light source 400 corresponds to the setting of the light direction DL. The light intensity of the real light source 400 may also be set.

[0381] When there is setting information for the real light source 400 as described above (i.e., the default position of the real light source 400, etc.), the space-floating image display device 1000 can generate a shadow SH based on the setting information. In FIG. 33, the image processing unit 1501 can omit the processing in steps S302 and S303 by referring to the setting information, and in step S304, the shadow SH can be generated based on the setting information. Furthermore, when the position of the real light source 400 cannot be estimated based on the results of processing using the sensor 500, the image processing unit 1501 may refer to this setting information and perform processing. Note that there may be a difference between this setting information and the actual operation and state of, for example, the lighting device 4401 in the user environment 1500, but the image processing unit 1501 may prioritize this setting information, or may control the image processing unit 1501 to prioritize the actual operation and state.

[0382] Up to this point, we have mainly explained the control patterns #2 and #3 in Figure 24 (when generating a first shadow). Other control patterns can basically be realized in the same way. Below, we will provide additional information about the other control patterns.

[0383] [Control patterns #1 and #6] The control patterns #1 and #6 in Fig. 24 will be explained in more detail. Fig. 46 is an explanatory diagram of the control patterns #1 and #6. State A is an example of video data as the content of the original data, and is video data including an unrendered three-dimensional model object 4602, which is illustrated here as a view of a two-dimensional image 4601. No shadow (second shadow) is formed on the surface of this object 4602. Furthermore, no shadow (first shadow) extending outward is given to this object 4602.

[0384] In contrast, state B is an example in which a shadow SH is generated and imparted by the shadow generation function 1510. When rendering based on an object 4602, the image processing unit 1501 imparts a new shadow 4604 (second shadow) on the object surface to an object image 4603 corresponding to the object 4602 based on the positional relationship with the real light source 400.

[0385] As a result, when the user looks at the object image 4603 of the space floating image 3, the user can see the shadow 4604 (second shadow) corresponding to the real light source 400, and the sense of realism of the object image 4603 can be enhanced.

[0386] [Control patterns #4 and #9] A supplementary explanation will be given regarding the control patterns #4 and #9 in Fig. 24. Fig. 47 is an explanatory diagram of the control patterns #4 and #9. State A is an example of video data as the content of the original data, and is a two-dimensional image 4701 that has been rendered based on a three-dimensional model, and includes an object image 4702. A shadow 4704 (second shadow) based on a virtual light source 4703 is formed on the surface of this object image 4702. No shadow (first shadow) that extends outward is added to this object image 4702.

[0387] In contrast, state B is an example in which a shadow SH is generated and imparted by the shadow generation function 1510. The image processing unit 1501 imparts a new shadow 4705 (first shadow) that extends outward to the object image 4702 based on the positional relationship with the real light source 400. The image processing unit 1501 also leaves unchanged the shadow 4704 (second shadow) on the object surface that is already formed in the image data of the original data.

[0388] In the case of this control pattern, when comparing the shadow 4704 (SH2) and the shadow 4705 (SH1) of the object image 4702, the light sources are different, so more precisely, the shadows of the two do not match. However, this is thought to not cause much discomfort when the user views the space floating image 3, so it is acceptable to display it this way.

[0389] [Control patterns #5 and #10] A supplementary explanation will be given regarding control patterns #5 and #10 in Fig. 24. Fig. 48 is an explanatory diagram of control patterns #5 and #10. State A is an example of video data as the content of the original data, and is a two-dimensional image 4801 that has been rendered based on a three-dimensional model, and includes an object image 4802. A shadow 4804 (second shadow) based on a virtual light source 4803 is formed on the surface of this object image 4802. A shadow 4805 (first shadow) that extends outward is also applied to this object image 4802 based on the virtual light source 4803.

[0390] In contrast, state B is an example in which a shadow SH is generated and imparted by the shadow generation function 1510. The image processing unit 1501 imparts a shadow 4806 (first shadow) extending outward to the object image 4802 in a replacement manner, based on the positional relationship with the real light source 400. Furthermore, the image processing unit 1501 leaves the shadow 4804 (second shadow) on the object surface already formed in the image data of the original data unchanged.

[0391] An example of the process of replacing this shadow (first shadow) is as follows: Image processing unit 1501 analyzes two-dimensional image 4801, detects the area of ​​shadow 4805 that extends outside the area of ​​object image 4802, and erases it so that it becomes the same as the background. Then, image processing unit 1501 generates shadow 4806 for two-dimensional image 4801B after erasure, based on the positional relationship with real light source 400. Space floating image 3 in state B in FIG. 48 has the same content as Space floating image 3 in state B in FIG. 47.

[0392] Even in the case of this control pattern, when comparing the shadow 4804 (SH2) and the shadow 4806 (SH1) of the object image 4802, the light sources are different, and more specifically, the way the shadows are cast is inconsistent. However, this display is acceptable because it is thought that this will not cause much discomfort when the user views the Space Floating Image 3. Note that if this type of shadow replacement process is performed while the user is viewing the Space Floating Image 3, the shadow SH1 will appear to have moved, causing discomfort. In order to eliminate or reduce this discomfort, the shadow replacement process may be performed, for example, when the display content of the image is changed or when the user is not gazing at the Space Floating Image 3.

[0393] [When generating shadows on GUI images] In the above example, a case where a shadow SH is added to a character image (in other words, the main content) has been explained, but the object to which a shadow SH can be added is not limited to this. An example of generating a shadow SH for other types of object images in the Space Floating Image 3 will be explained. Here, the case where a shadow SH is generated for objects such as buttons, icons, menus, etc. that make up the GUI image displayed on the screen of the Space Floating Image 3 (in other words, GUI objects, sub-contents) will be explained.

[0394] 49 shows a display example of GUI object images and the like in a front view of the screen of the space floating image 3 on the stage 1190S. In state A, a character image and the like are displayed on the screen as an object image 4901 (main content). A shadow 4902 (first shadow) generated by the shadow generation function 1510 is also added to the object image 4901. In this example, the positional relationship of the real light source 400 with respect to the space floating image 3 is such that the light direction DL is in a lower left direction.

[0395] Furthermore, suppose that a menu 4903 is placed in the upper right corner area of ​​the xy plane of the screen, for example, as an example of a GUI object image constituting the GUI. Menu 4903 shown in state B is an example of an image of menu 4903 in GUI data previously stored in space-floating image display device 1000. Menu 4903 includes, for example, start and end buttons. This menu 4903 is a simple, flat image with no shadow. State C shows menu 4904 as another example of an image of menu 4903 in GUI data. This menu 4904 is an image to which a predefined fixed shadow 4905 is added, and the direction of this shadow 4905 is a downward and rightward direction corresponding to a predefined fixed virtual light source. Menu 4903 in state A is an example in which menu 4903 in state B is placed as is on the screen.

[0396] First, a screen display like state A may be used. That is, the space floating image display device 1000 controls the shadow generation function 1510 to add a shadow SH to main content such as character images, but not to add a shadow SH to sub-content such as GUI objects. Adding a shadow SH to character images increases reality. On the other hand, in situations where there is less need to increase the reality and three-dimensionality of GUI objects than character images, it is not necessary to add a shadow SH, as in state A.

[0397] State D is an example in which the menu 4904 of state C is placed on the screen as is. In state D, the menu 4904 has a strong three-dimensional appearance when viewed alone. In situations where it is desired to enhance the reality and three-dimensional appearance of the GUI object (for example, when encouraging GUI operations), it may be displayed with a shadow 4905, as in state D. At that time, the shadow 4902 of the object image 4901, which is the main content, may be temporarily erased.

[0398] However, in the example of state D, the way the shadow 4902 is cast on the object image 4901, which is a character image, and the way the shadow 4905 is cast on the menu 4904 are not consistent (in other words, synchronized). In other words, the assumed light source position and light direction are different between the two. Therefore, when the user looks at the screen of this space-floating image 3, there is a risk of feeling uncomfortable. Therefore, the space-floating image display device 1000 controls the shadow generation function 1510 so that the display does not become like state D.

[0399] Specifically, the following applies: First, when it is anticipated that state D will occur, the video processing unit 1501 may change the display so that the menu 4903 is displayed without a shadow, as in state A.

[0400] FIG. 50 is a continuation of FIG. 49. A display such as state E may be used. In state E, a menu 4906 is displayed on the screen. This menu 4906 is given a shadow 4907 generated by the shadow generation function 1510. The image processing unit 1501 creates a menu 4906 as shown in state F based on the menu 4903 in state B of FIG. 49 or the menu 4904 in state C. When creating the menu, the image processing unit 1501 adds the shadow 4907 so as to correspond to the positional relationship with the real light source 400. Alternatively, a plurality of variations with different shadows, including the menu 4906 in state F, may be prepared in advance in the GUI data, and the menu may be selected and used.

[0401] In the example of the screen in state E, the way shadow 4902 is attached to object image 4901, which is a character image, and the way shadow 4907 is attached to menu 4906 are consistent (in other words, synchronized). That is, the light direction of the assumed light source is roughly the same for both. Specifically, both shadow 4902 and shadow 4907 extend to the left in the x direction. Therefore, when the user looks at the screen of this space floating image 3, it is unlikely to feel strange. Space floating image display device 1000 may control the shadow generation function 1510 to display as in state E.

[0402] Furthermore, if we consider this in more detail, the influence of the shadow SH will differ depending on whether the position of a light source such as real light source 400 is in front or behind the screen of the Floating in Space Image 3 in the depth direction. The shadow 4907 of the menu 4906 in state E is given so as to extend behind the plane of the menu, corresponding to the case where light is irradiated onto the plane of the menu from the front with respect to the position in the depth direction of the Floating in Space Image 3 (for example, FIG. 37). On the other hand, the shadow 4902 of the object image 4901 is generated corresponding to the case where the light direction DL from the real light source 400 is from the back to the front with respect to the position in the depth direction of the Floating in Space Image 3. In this case, the light directions of both are opposite in the front-to-back direction. Therefore, some users may feel uncomfortable with the way the shadows of both are cast.

[0403] In this case, the following control may be performed. First, when the difference between the front and rear directions is taken into consideration with regard to the light direction as described above, the image processing unit 1501 performs control so as not to display the screen as in state E. Specifically, the menu 4903 may be displayed without a shadow, as in state A.

[0404] Furthermore, state G is menu 4908, which is another display example related to menu 4906. This menu 4908 is an example in which a shadow 4909 is added to the front side of the plane of the menu when the light direction from real light source 400 is from back to front. Such a menu 4908 may be displayed on the screen instead of menu 4906 in state E. In this case, the shadow 4909 of menu 4906 and the shadow 4902 of object image 4901 are consistent in the front-to-back direction. However, with such menu 4908, there is a risk that the shadow 4909 may make it difficult to see buttons and the like on the plane of the menu. Therefore, control may be performed so that such a menu 4908 is not adopted.

[0405] State H is an example of another screen. In state H, the position of the real light source 400 is assumed to be in front of the object image 4901 in the depth direction. A shadow 4910 added to the object image 4901 extends from the feet of the character image to the rear. When displaying a menu in this case, the image processing unit 1501 employs the same menu 4906 as in state E. In this case, the shadow 4910 and the shadow 4907 are consistent, including in the front-to-back direction.

[0406] [Effects of Example 3] As described above, according to the third embodiment, by generating and displaying a shadow SH corresponding to the real light source 400 on the object image in the floating-in-space image 3, it is possible to enhance the three-dimensional effect, presence, reality, etc. of the object image.

[0407] FIG. 58 is a supplementary explanatory diagram regarding the effect. State A and State B are comparative examples of front-view images of a floating-in-space image 3 generated based on original data. In State A, the object image 3A, e.g., a character image, is positioned a short distance above the bottom edge of the screen. In State B, the object image 3A, e.g., a character image, is positioned close to the bottom edge of the screen. This character image is not provided with a shadow SH projected onto the floor. There is also no background image displayed. In these cases, from the user's perspective, the character image appears to be floating in the air in three-dimensional space. The character image's feet are unstable, and the user may not be able to tell whether the character image is standing or jumping in the air. As a result, the user may not perceive the character image as very realistic.

[0408] In contrast, state C is a case where a shadow SH (particularly a first shadow) is generated and displayed on the object image 3A in the floating-in-space image 3 on the stage 1190S using the function of the third embodiment. The floating-in-space image 3 in state C differs from state A only in the presence or absence of an image of the shadow SH. In state C, the mere presence of the shadow SH allows the user to imagine a virtual floor, making the character image's footing more stable. From the user's perspective, the character image appears to be standing on the virtual floor. As a result, the user can sense the reality that the character image truly exists there. Furthermore, since this shadow SH is generated in detail in accordance with the real light source 400, the user can sense the reality that the character image exists within the user environment 1500.

[0409] Example 4 A fourth embodiment will be described. The fourth embodiment deals with the expression of wind instead of shadow. In the fourth embodiment, an image based on the expression / action of wind is displayed in the floating image in space so as to match the actual wind situation (for example, direction and strength) in the floating image display device in the user's environment.

[0410] [Wind generation function] FIG. 51 shows a display system including a space-floating image display device 1000 of a fourth embodiment. The space-floating image display device 1000 has a wind generation function 5120 that uses a sensor 700 to detect the state of real wind 600 in a user environment 1500 and generates an image with the expression / action of wind WD corresponding to the state of real wind for the object image 3B of the space-floating image 3. The space-floating image display device 1000 uses the sensor 700 to detect the state of wind 600 relative to the position of the space-floating image display device 1000, the space-floating image 3, or the object image 3B in the user environment 1500, such as the position of the source of wind 600, wind direction (wind direction DW), wind speed, wind volume, etc. Regarding the state of wind 600, when outdoors, it is sufficient to know the wind direction DW, etc., even if the position of the source of generation is not known. When indoors, the position of the source of generation may be identified.

[0411] The source of the wind 600 in the room can be, for example, an air conditioner 4402 or an electric fan 4403 in Fig. 44. The space floating image display device 1000 may detect and grasp the state of the wind 600 in cooperation with these home appliances 4400 or a management system 4410, as in Fig. 44. The sensor 700 is any hardware (for example, a wind direction and speed sensor) that can detect the position of the source of the wind 600, the wind direction DW, or the wind speed and volume, and is not limited to this.

[0412] The image processing unit 1501 generates, adds, and displays an image with the expression / action of wind as wind WD to the object image 3B in the spatial floating image 3, so as to match the state of the detected wind 600. An image with the expression / action of wind WD is an image in which the object image 3B, for example, a character image, is affected as if by the real wind 600, such as image changes such as hair or clothes fluttering or fluttering in the wind.

[0413] [Representation of wind / video using the effects] Figure 52 shows an example of an image with the expression / effect of wind WD on object image 3B of the floating image 3 on stage 1190S. In state A, the character image as object image 3B is standing on stage 1190S with the back of its body exposed. In state A, the wind 600 condition is assumed to be a state where there is no wind (calm), in other words, a state where a specific wind direction DW or a certain amount of wind volume cannot be detected.

[0414] In state B, the state of wind 600 is one in which there is wind, and the wind is blowing from right to left along the X-axis relative to the position of the floating image 3 in space. In other words, the wind direction DW is roughly to the left. The posture of the character image does not change between state A and state B. The difference is that images 5201 and 5202 representing the expression / action of wind WD are generated. In this example, image 5201 representing the expression / action of wind WD shows hair fluttering to the left, and image 5202 representing the expression / action of wind WD shows clothes fluttering to the left. These images representing the expression / action of wind WD can be made more effective by using moving images / animations. Furthermore, not only the wind direction DW but also the magnitude of the wind speed / volume can be reflected in the expression / image of wind WD; for example, the greater the wind volume, the more the hair and clothes will flutter.

[0415] The wind 600 in state A and state B is not a virtual wind but a real wind. In this embodiment 4, even if the object / object image of the original data has a setting for virtual wind, the space floating image display device does not adopt it, but generates an image with the expression / action of the wind WD so as to match the situation of the detected real wind 600.

[0416] Fig. 53 is another example of an image using the expression / effect of wind WD. Not limited to the hair and clothes on a character's body, any object image within the floating in space image 3 may also be affected by the wind and change. In the example of Fig. 53, in the floating in space image 3, there are object images such as a road, trees, and flag 5203 as a background to object image 3A, which is the character. Depending on the wind direction DW of wind 600, object image such as flag 5203 is fluttering to the left.

[0417] It is also possible to combine the third embodiment with the fourth embodiment. That is, it is possible to simultaneously control both the expression of shadows in the third embodiment and the expression of wind in the fourth embodiment.

[0418] In the case of an embodiment that combines the third and fourth embodiments, the following control may be performed. Assume that the detailed shape of the object image (for example, the character's body) in the floating in space image 3 changes due to the fluttering of clothes, etc., depending on the state of the real wind 600. Assume that the light direction, etc., of the real light source 400 remains roughly constant. In this case, the effect of the light irradiated by the real light source 400 on the object image also changes in detail. That is, when the above-mentioned shadow is added to the object image, the detailed shape of the shadow may also change to match the state of the wind. This increases the realism, etc. It is preferable to change not only the character image but also objects in the background, such as trees and flags.

[0419] FIG. 54A shows an example of the configuration of a sensor 700 related to wind 600. The space floating image display device 1000 is equipped with a sensor 700 capable of detecting and measuring wind direction DW, wind speed, etc. When the user environment 1500 is indoors, a wind direction and wind speed sensor (a gentle breeze sensor) capable of detecting a gentle breeze indoors may be applied as the sensor 700. One example of a known sensor 700 is one equipped with multiple sensor elements and capable of detecting and measuring wind direction and speed in two dimensions. The type of sensor 700 may be a wind pressure type wind speed sensor, an ultrasonic type wind speed sensor, or a thermal type wind speed sensor. Another example of the sensor 700 may be a microphone (voice input device). A known technique is to calculate the wind volume based on noise (wind noise) input to a microphone.

[0420] In the example of Fig. 54A, together with the sensor 500 using the camera described above, one wind direction / wind speed sensor 701 is installed as the sensor 700 on the upper surface 1190S1 of the stage 1190S. This sensor 700 detects the wind direction DW on at least two axes corresponding to the XY plane. The position of the sensor 700 is not limited to the position shown in the figure and can be any position.

[0421] FIG. 54B shows another example of the sensor 700. A microphone 702 is installed as the sensor 700, for example, on the front side of the stage 1190S. The microphone 702 may be a directional microphone. The microphone 702 (microphone 1139 in FIG. 3) can also be used to input the user's voice. A plurality of microphones 702 may be installed. An omnidirectional microphone may be used as the microphone 702. For example, similar to the above-mentioned camera, the microphones 702 may be installed at the front, back, left, right, etc. of the stage 1190S.

[0422] The video processing unit 1501 (FIG. 16) estimates the wind direction DW of the wind 600 based on the detection signal of the sensor 700. Even when multiple omnidirectional microphones are used, the wind direction DW can be estimated by calculation. The estimated wind direction DW may be a direction in a three-dimensional space (X, Y, Z directions) in detail, or may simply be a two-dimensional direction (for example, X, Y directions).

[0423] Furthermore, when the image processing unit 1501 (FIG. 16) cannot adequately estimate the wind direction DW using the sensor 700, that is, when it cannot determine a specific wind direction DW, it may refer to other information such as weather information and make a determination in combination with the sensor information. This can be realized in the same manner as in the third embodiment.

[0424] When the cooperation configuration of Fig. 44 described above is used for generating the wind WD, the following control is possible, for example. The space floating image display device 1000 cooperates with the home appliance 4400 or the management system 4410 via communication, and acquires, for example, information on the air conditioner 4402, the electric fan 4403, etc. as information on the source of the wind 600. The information may include, for example, information on whether the appliance is operating or not, and information on the air volume. Based on the acquired information on whether the appliance is operating or not, the image processing unit 1501 may perform control to enable the function of generating the wind WD when the appliance 4400 (for example, the electric fan 4403) turns on.

[0425] Image processing unit 1501 estimates the wind direction DW from the source of wind 600 such as electric fan 4403 relative to the position of floating in space image 3 (particularly object image) based on information acquired from home appliance 4400 and the like and detection information from sensor 700. When estimating wind direction DW, if wind direction DW can be estimated directly from only the detection information from sensor 700, only that detection information may be used. Alternatively, using information on the position of home appliance 4400 as the source of wind 600, the direction from the position of that home appliance 4400 to the position of floating in space image 3 may be considered to be wind direction DW.

[0426] As in the third embodiment, in the fourth embodiment, as a modified example, the positional relationship of the real wind 600 with respect to the space floating image display device 1000 in the device installation environment 1500 may be set in advance by initial setting or user setting.

[0427] FIG. 55 shows a modified example in which the GUI on the screen of the Space Floating Image 3 of the Space Floating Image Display Devic...

Claims

1. A floating-in-the-air image display device, a video processing unit that performs video processing; a display unit that displays the image that has been processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; Equipped with the image processing unit generates a shadow image that would appear if light from a real light source were irradiated onto the object image in the floating-in-the-air image in an environment in which the floating-in-the-air image display device is installed, and displays the floating-in-the-air image in which the shadow image is added to the object image; A floating video display device.

2. 2. The airborne image display device according to claim 1, the image processing unit generates the image of the shadow so as to match the position of the real light source or the direction of light from the real light source. A floating video display device.

3. 2. The airborne image display device according to claim 1, the shadow image is generated as at least one of a shadow on the object surface of the object image and a shadow projected onto a floor surface assumed for the object image. A floating video display device.

4. The airborne image display device according to claim 2, a sensor for estimating at least one of the position of the real light source and the direction of the light; the image processing unit estimates at least one of the position of the real light source and the direction of the light using the detection signal of the sensor, and generates the image of the shadow so as to match the position of the real light source or the direction of the light based on the estimation result. A floating video display device.

5. 2. The airborne image display device according to claim 1, the image processing unit generates the image of the shadow by using information about the real light source without using information about a virtual light source set in the original data of the object image. A floating video display device.

6. 2. The airborne image display device according to claim 1, the image processing unit creates image data in which the image of the shadow is added to the image of the object by rendering processing from a three-dimensional model that is the original data of the image of the object, and displays the floating-in-the-air image based on the image data; A floating video display device.

7. 2. The airborne image display device according to claim 1, the image processing unit creates image data in which the shadow image is added to the object image from image data including a rendered object image, which is original data of the object image, and displays the floating-in-the-air image based on the image data. A floating video display device.

8. 2. The airborne image display device according to claim 1, the image processing unit receives a three-dimensional model, which is the original data of the object image, from an external device connected to the floating-in-the-air image display device, and creates image data from the three-dimensional model in which the shadow image is added to the object image by rendering processing, and displays the floating-in-the-air image based on the image data; A floating video display device.

9. 2. The airborne image display device according to claim 1, the image processing unit receives image data including a rendered object image, which is the original data of the object image, from an external device of the floating-in-the-air image display device, creates image data in which the shadow expression is added to the object image from the image data, and displays the floating-in-the-air image based on the image data; A floating video display device.

10. 2. The airborne image display device according to claim 1, the image processing unit extracts and erases the image of a shadow when the image of a shadow is included in the original data of the object image, and generates the image of the shadow using information about the real light source. A floating video display device.

11. 2. The airborne image display device according to claim 1, the image processing unit, when the original data of the object image is image data including a rendered object image, and the object image has a shadow on the object surface based on a virtual light source and no shadow projected on a floor surface that is assumed for the object image, leaves the shadow on the object surface as it is and newly adds the image of the shadow; A floating video display device.

12. 2. The airborne image display device according to claim 1, When the original data of the object image is image data including a rendered object image, and the object image has a shadow on the object surface based on a virtual light source and a shadow projected onto an assumed floor surface for the object image, the image processing unit leaves the shadow on the object surface as it is, extracts and erases the shadow projected onto the floor surface based on the virtual light source, and assigns it to the shadow image instead. A floating video display device.

13. 2. The airborne image display device according to claim 1, The floating image display device has a stage on its housing, The floating image is formed on the upper surface of the stage. A floating video display device.

14. The airborne image display device according to claim 4, The sensor comprises one or more cameras; The image processing unit estimates the position or direction of the actual light source based on a brightness distribution in the image captured by the camera. A floating video display device.

15. 2. The airborne image display device according to claim 1, the image processing unit generates the shadow image by using a preset shadow image selected from preset shadow images stored in a memory in advance, so as to match the shadow image with the position or direction of the actual light source; A floating video display device.

16. 2. The airborne image display device according to claim 1, the image processing unit superimposes the image of the shadow on the rear side of the object image when the direction of light from the real light source in the front-to-back direction relative to the floating-in-the-air image is a direction in which light is irradiated from the front, and superimposes the image of the shadow on the front side of the object image when the light is irradiated from the rear; A floating video display device.

17. 2. The airborne image display device according to claim 1, the image processing unit adjusts the position of the object image within the screen of the floating image so as to secure a predetermined distance from a bottom edge, and arranges the shadow image using the space defined by the predetermined distance; A floating video display device.

18. 2. The airborne image display device according to claim 1, the image processing unit changes the brightness, color, or size of the display of the shadow image in accordance with the intensity of the light from the real light source. A floating video display device.

19. 2. The airborne image display device according to claim 1, The floating-in-the-air image display device cooperates with a home appliance corresponding to the real light source as an external device, and acquires information about the real light source from the home appliance; The image processing unit generates an image of the shadow using the acquired information. A floating video display device.

20. A floating-in-the-air image display device, a video processing unit that performs video processing; a display unit that displays the image that has been processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; Equipped with the image processing unit generates an image resulting from the action of wind when a real wind is assumed to be blowing on the object image in the floating-in-the-air image in an environment where the floating-in-the-air image display device is installed, and displays the floating-in-the-air image in which the image resulting from the action of the wind is added to the object image; A floating video display device.

Images