Aerial floating video display device, portable terminal, and display method

Abstract

The purpose of the present invention is to provide a more suitable aerial floating image display device. The present invention contributes to the sustainable development goals (SDGs) "3. Good health and well-being," "9. Industries, innovation and infrastructure," and "11. Sustainable cities and communities." This aerial floating image display device performs control such that, for example, a display image displayed on a screen of a portable terminal of a user is displayed as a display image also in an aerial floating image on the basis of a communication connection with the portable terminal, and on the basis of detection and determination of a user operation on the display image displayed on the screen of the portable terminal, predetermined processing associated with the display image on the screen is executed, and the processing is reflected also in the display image in the aerial floating image. Further, in the present system, setting and control as to in which device operation and detection are enabled / disabled are performed.

Description

Floating-air image display device, mobile terminal, and display method

[0001] The present invention relates to a floating-in-the-air image display device.

[0002] The floating information display technology is disclosed in, for example, Patent Document 1.

[0003] Japanese Patent Application Laid-Open No. 2019-128722

[0004] However, the disclosure of Patent Document 1 does not sufficiently consider configurations for obtaining practical brightness and quality for the floating image, or configurations for allowing users to view the floating image more enjoyably.

[0005] An object of the present invention is to provide a more suitable floating-in-the-air image display device.

[0006] In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problems, and one example thereof may be configured as follows. One embodiment is a floating-in-the-air image display device, which includes an image processing unit, a display unit that displays an image that has been image-processed by the image processing unit, an optical system that generates a floating-in-the-air image based on the image displayed by the display unit, a detection unit that detects contact of an object with the display area of ​​the floating-in-the-air image, and a communication unit that communicates with a user's mobile device, and when contact of the mobile device with the display area is detected, a target image specified on the mobile device is displayed as a floating-in-the-air image.

[0007] According to the present invention, a more suitable floating-in-the-air image display device can be realized. Other problems, configurations, and effects will become clear in the following description of the embodiments.

[0008] FIG. 1 is a diagram showing an example of a usage form of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of a configuration of a main part and a retroreflector of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a projection diagram of a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 7 is a top view of a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 8 is a perspective view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 9 is a top view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 10 is a side view of a corner reflector constituting a retroreflector constituting a space-floating image display device according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of a configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 1 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention.1 is a diagram showing an example of the configuration of a space-floating image display device according to an embodiment of the present invention. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 3 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 4 is a cross-sectional view showing an example of a specific configuration of a light source device according to an embodiment of the present invention. FIG. 5 is a layout diagram showing main parts of a space-floating image display device according to an embodiment of the present invention. FIG. 6 is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. FIG. 7 is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. FIG. 8 is an explanatory diagram for explaining light source diffusion characteristics of an image display device according to an embodiment of the present invention. FIG. 9 is an explanatory diagram for explaining diffusion characteristics of an image display device according to an embodiment of the present invention. FIG. 10 is an explanatory diagram for explaining an example of a problem solved by image processing according to an embodiment of the present invention. FIG. 11 is an explanatory diagram for explaining an example of image processing according to an embodiment of the present invention. FIG. 12 is an explanatory diagram for explaining an example of image display processing according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of the configuration of a main part and a retroreflection part of a space-floating image display device according to an embodiment of the present invention. FIG. 14 is a diagram showing the configuration of a space-floating image display system according to an embodiment of the present invention. FIG. 15 is an overview of a space-floating image display system according to an embodiment of the present invention. FIG. 16 is a diagram showing the configuration of a mobile terminal according to an embodiment of the present invention. FIG. 17 is a perspective view showing an operation of inserting and touching a mobile terminal into a space-floating image according to an embodiment of the present invention. 1 is an XY plane view showing an insertion and touch operation of a mobile terminal into a floating-in-space image according to an embodiment of the present invention; a YZ plane view showing an insertion and touch operation of a mobile terminal into a floating-in-space image according to an embodiment of the present invention; an XZ plane view showing an insertion and touch operation of a mobile terminal into a floating-in-space image according to an embodiment of the present invention; a perspective view showing a posture etc. when inserting and touching a mobile terminal into a floating-in-space image according to an embodiment of the present invention; an XY plane view showing an example of a posture etc. when inserting and touching a mobile terminal into a floating-in-space image according to an embodiment of the present invention; a YZ plane view showing an example of a posture etc. when inserting and touching a mobile terminal into a floating-in-space image according to an embodiment of the present invention; an XZ plane view showing an example of a posture etc. when inserting and touching a mobile terminal into a floating-in-space image according to an embodiment of the present invention; a YZ plane view showing an example of a posture etc. when inserting and touching a mobile terminal into a floating-in-space image according to an embodiment of the present invention;1 is a YZ plane view showing an example of a position when inserting or touching a mobile terminal into a space-floating image according to an embodiment of the present invention; 2 is an XY plane view showing an example of a position when inserting or touching a mobile terminal into a space-floating image according to an embodiment of the present invention; 3 is an XY plane view showing an example of a position when inserting or touching a mobile terminal into a space-floating image according to an embodiment of the present invention; 4 is an explanatory diagram showing an example of a display of a guide image on a space-floating image according to an embodiment of the present invention; 5 is an explanatory diagram showing an example of a display of a guide image on a space-floating image according to an embodiment of the present invention; 6 is an XY plane view showing an example of a mobile terminal touching a space-floating image according to an embodiment of the present invention; 7 is an XZ plane view showing an example of an invisible area due to a mobile terminal touching a space-floating image according to an embodiment of the present invention; 8 is a YZ plane view showing an example of shading due to a mobile terminal touching a space-floating image according to an embodiment of the present invention; 9 is an XY plane view showing an example of a display of a guide image on a space-floating image and an example of an invisible area due to a mobile terminal touching a space-floating image according to an embodiment of the present invention; 10 is an XY plane view showing an example of a display of a guide image on a space-floating image according to an embodiment of the present invention and an example of a display area restriction due to a mobile terminal touching a space-floating image according to an embodiment of the present invention; 1 is an xy plane view showing an example of a guide image displayed on a space-floating image according to an embodiment of the present invention. an xy plane view showing an example of a guide image displayed on a space-floating image according to an embodiment of the present invention. an xy plane view showing an example of a guide image displayed on a space-floating image according to an embodiment of the present invention. an explanatory diagram showing an example of a guide image displayed on a space-floating image and an example of a display area restriction according to an embodiment of the present invention. an explanatory diagram showing an example of a guide voice output according to an embodiment of the present invention. a YZ plane cross-sectional view showing an example of an arrangement of sensors in a space-floating image display device according to an embodiment of the present invention. an xy plane view showing an example of a configuration of an aerial operation detection sensor according to an embodiment of the present invention. a YZ plane cross-sectional view showing an example of an arrangement of sensors in a space-floating image display device according to an embodiment of the present invention. an explanatory diagram showing an example of an application screen display on a mobile terminal according to an embodiment of the present invention. an explanatory diagram showing an example of an application screen display on a mobile terminal according to an embodiment of the present invention. an explanatory diagram showing an example of a QR code screen display on a mobile terminal according to an embodiment of the present invention. a YZ plane cross-sectional view showing an example of an image of a QR code on a mobile terminal in a space-floating image display device according to an embodiment of the present invention. a perspective view showing an example of a QR code display on a space-floating image according to an embodiment of the present invention.1 is a YZ-plane cross-sectional view showing an example of capturing an image of a QR code of a space-floating image of a space-floating image display device by a mobile terminal according to an embodiment of the present invention; 2 is a perspective view showing an example of a QR code displayed on the screen of a second display device according to an embodiment of the present invention; 3 is a diagram showing an example of the configuration of a system in which a mobile terminal and a space-floating image display device are communicatively connected according to an embodiment of the present invention; 4 is a diagram showing an example of dual display between a mobile terminal and a space-floating image display device according to an embodiment of the present invention; 5 is a diagram showing a basic control flow of a system according to an embodiment of the present invention; 6 is a diagram showing an example of display and setting before connection of the system according to an embodiment of the present invention; 7 is a diagram showing an example of display and setting (mode) after connection of the system according to an embodiment of the present invention; 8 is a diagram showing a GUI example (1) for mode setting as an example of a screen display of a mobile terminal according to an embodiment of the present invention; 9 is a diagram showing a GUI example (2) for mode setting as an example of a screen display of a mobile terminal according to an embodiment of the present invention; 10 is a diagram showing an example of a functional block configuration of a system according to an embodiment of the present invention; 11 is a diagram showing an example of the configuration of a screen of a space-floating image display device and an example of the configuration of a screen of a mobile terminal according to an embodiment of the present invention; 12 is a diagram showing an example 1 of a problem and solution according to an embodiment of the present invention; FIG. 1 is a diagram showing example 2 of the problem and solution according to an embodiment of the present invention. FIG. 2 is a diagram showing example 3 of the problem and solution according to an embodiment of the present invention. FIG. 3 is a diagram showing example 4 of the problem and solution according to an embodiment of the present invention. FIG. 4 is a diagram showing example (pattern) of image display between a space-floating image display device and a mobile terminal according to an embodiment of the present invention. FIG. 5 is a diagram showing examples of operations to be controlled and sensors used for control according to an embodiment of the present invention. FIG. 6 is a diagram showing example 1 of the attitude of a space-floating image display device according to an embodiment of the present invention. FIG. 7 is a diagram showing example 2 of the attitude of a space-floating image display device according to an embodiment of the present invention. FIG. 8 is a diagram showing example 3 of the attitude of a space-floating image display device according to an embodiment of the present invention. FIG. 9 is a diagram showing example (1) of a sequence in mode 1 according to an embodiment of the present invention. FIG. 10 is a diagram showing example (2) of a sequence in mode 1 according to an embodiment of the present invention. FIG. 11 is a diagram showing example (3) of a sequence in mode 1 according to an embodiment of the present invention. FIG. 12 is a diagram showing example (4) of a sequence in mode 1 according to an embodiment of the present invention. FIG. 13 is a diagram showing example (1) of a sequence in mode 2 according to an embodiment of the present invention. FIG. 14 is a diagram showing example (2) of a sequence in mode 2 according to an embodiment of the present invention. FIG. 15 is a diagram showing example (1) of a sequence in mode 3 according to an embodiment of the present invention.FIG. 1 illustrates an example sequence (2) for mode 3 according to an embodiment of the present invention. FIG. 2 illustrates an example sequence (1) for mode 5 according to an embodiment of the present invention. FIG. 3 illustrates an example sequence (1) for mode 7 according to an embodiment of the present invention. FIG. 4 illustrates an example sequence (2) for mode 7 according to an embodiment of the present invention. FIG. 5 illustrates an example sequence (3) for mode 7 according to an embodiment of the present invention. FIG. 6 illustrates an example sequence (4) for mode 7 according to an embodiment of the present invention. FIG. 7 illustrates an example sequence (1) for mode 8 according to an embodiment of the present invention. FIG. 8 illustrates an example sequence (2) for mode 8 according to an embodiment of the present invention. FIG. 9 illustrates an example sequence (1) for mode 9 according to an embodiment of the present invention. FIG. 9 illustrates an example sequence (2) for mode 9 according to an embodiment of the present invention. FIG. 10 illustrates an example sequence (1) for mode 11 according to an embodiment of the present invention.

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, components having the same functions are given the same reference numerals, and repeated explanations thereof may be omitted.

[0010] The following examples relate to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member. In the following explanation of the examples, the image floating in space is expressed using the term "floating image in space." Instead of this term, it is also acceptable to express it as "aerial image," "spatial image," "floating image in space," "floating optical image of displayed image," "floating optical image of displayed image," etc. The term "floating image in space," which is mainly used in the explanation of the examples, is used as a representative example of these terms.

[0011] According to the following embodiments, a suitable image display device can be realized for, for example, bank ATMs, train station ticket machines, digital signage, and the like. For example, currently, bank ATMs, train station ticket machines, and the like typically use touch panels, but by using a transparent glass surface or a light-transmitting plate, high-resolution image information can be displayed in a floating state on the glass surface or light-transmitting plate. In this case, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflector, resulting in high light utilization efficiency and suppressing the ghost images that occur in addition to the main floating image, which is a problem with conventional retroreflection systems, thereby achieving a clear floating image. Furthermore, a device including the light source of this embodiment can provide a novel, highly usable floating image display device (floating image display system) that can significantly reduce power consumption. Furthermore, a floating image display device for a vehicle can be provided that can display a so-called unidirectional floating image that can be viewed inside and / or outside the vehicle. Example 1

[0012] <Example of Usage of the Space-Floating Image Display Device> Figure 1 is a diagram showing an example of usage of a space-floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using Figure 2 and other figures. Light with a narrow-angle directional characteristic and specific polarization is emitted from the image display device 1 as an image light beam, reflected by the optical system within the space-floating image display device, and then incident on the retroreflector 2, retroreflected and transmitted through a transparent member 100 (glass, etc.), forming a real aerial image (space-floating image 3) on the outside of the glass surface. In the following examples, the retroreflector 2 (retroreflector) will be used as an example of the retroreflector. However, the retroreflector 2 of the present invention is not limited to a planar plate, and is used as an example of a concept including a sheet-like retroreflector attached to a planar or non-planar member, or an entire assembly in which a sheet-like retroreflector is attached to a planar or non-planar member. Furthermore, since the light rays reflected by the retroreflector 2 have the optical property of forming an image, the retroreflector 2 may also be expressed as an imaging optical member or an imaging optical plate.

[0013] In addition, in a store or the like, a space is partitioned by a show window (also called "window glass") 105, which is a transparent member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit such a transparent member and display a floating image in one direction to the outside and / or inside of the store (space).

[0014] 1, the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing a means for reflecting specific polarized waves in the window glass 105, it is possible to form an aerial image at a desired position inside the store by reflecting the specific polarized waves.

[0015] <Configuration Example of Optical System of Space-Floating Image Display Device> Figure 2A is a diagram showing an example of the configuration of an optical system of a space-floating image display device according to one embodiment of the present invention. The configuration of the space-floating image display device will be described in more detail using Figure 2A. As shown in Figure 2A (1), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light with narrow-angle diffusion characteristics.

[0016] Image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 (in the figure, the polarization separation member 101 is formed in a sheet shape and adhered to the transparent member 100) provided on a transparent member 100 and has a film that selectively reflects image light of a specific polarization, and then enters the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector 2. The image light is polarized and converted from the specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when entering the retroreflector 2 and once when exiting. Here, the polarization separation member 101, which selectively reflects image light of a specific polarization, has the property of transmitting the polarized light of the other polarization that has been polarization-converted, so the image light of the specific polarization after polarization conversion passes through the polarization separation member 101. The image light that has passed through the polarization separation member 101 forms a spatially floating image 3, which is a real image, outside the transparent member 100. 2A shows an example in which the chief ray of the image light incident on the retroreflector 2 is incident at an angle of 90° to the retroreflector 2. However, the incident angle of the chief ray of the image light on the retroreflector 2 is not limited to 90°, and an angle of, for example, 90°±15° can also be used.

[0017] Here, a first example of polarization design for the optical system of FIG. 2A will be described. For example, S-polarized image light may be emitted from the display device 1 to the polarization separation member 101, which may have the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the S-polarized image light reaching the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and travels toward the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the S-polarized light to P-polarized light. The P-polarized image light then travels back toward the polarization separation member 101. Here, the polarization separation member 101 has the property of reflecting S-polarized light and transmitting P-polarized light, so the P-polarized image light passes through the polarization separation member 101 and then through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0018] Next, a second example of polarization design for the optical system of FIG. 2A will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from the display device 1 to the polarization separation member 101, and the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the P-polarized image light that reaches the polarization separation member 101 from the display device 1 is reflected by the polarization separation member 101 and proceeds toward the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the image light from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again toward the polarization separation member 101. Here, the polarization separation member 101 has the property of reflecting P-polarized light and transmitting S-polarized light, so the S-polarized image light passes through the polarization separation member 101 and then through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101. This polarization design allows the floating image 3 to be formed optimally.

[0019] The light forming the space-floating image 3 is a collection of light rays that converge from the retroreflector 2 to the optical image of the space-floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the space-floating image 3. Therefore, the space-floating image 3 is an image with high directionality, unlike the diffuse image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2A, when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in systems that display images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0020] Depending on the performance of the retroreflector 2, the polarization axis of the reflected image light may become irregular. The reflection angle may also become irregular. Such irregular light may not maintain the polarization state and propagation angle assumed in the design. For example, light with an unintended polarization state and propagation angle may re-enter the image display surface of the liquid crystal display panel 11 directly from the position of the retroreflector 2 without passing through the polarization separation member. Such light with an unintended polarization state and propagation angle may be reflected by components within the space-floating image display device and then re-enter the image display surface of the liquid crystal display panel 11. Such light re-entering the image display surface of the liquid crystal display panel 11 may be re-reflected by the image display surface of the liquid crystal display panel 11 constituting the display device 1, potentially generating ghost images and degrading the image quality of the space-floating image. Therefore, in this embodiment, an absorbing polarizer 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorbing polarizer 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorbing polarizer 12, thereby suppressing the re-reflection. This makes it possible to prevent degradation of image quality due to ghost images of spatially floating images. Specifically, if the display device 1 is configured to emit S-polarized image light to the polarization separation member 101, the absorbing polarizer 12 may be a polarizer that absorbs P-polarized light. Furthermore, if the display device 1 is configured to emit P-polarized image light to the polarization separation member 101, the absorbing polarizer 12 may be a polarizer that absorbs S-polarized light.

[0021] The polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

[0022] 2A(2) shows an example of the surface shape of a typical retroreflector 2. Light rays incident on the interior of the regularly arranged hexagonal prisms are reflected by the walls and bottoms of the hexagonal prisms and emitted as retroreflected light in a direction corresponding to the incident light, and a real floating image is displayed on the display device 1 based on the image displayed.

[0023] The resolution of this floating image in space depends not only on the resolution of the liquid crystal display panel 11, but also on the outer diameter D and pitch P of the retroreflective portion of the retroreflector 2 shown in Figure 2A (2). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, one pixel of the floating image in space will be equivalent to 300 μm. As a result, the effective resolution of the floating image in space will be reduced to about one-third.

[0024] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Furthermore, it is advisable to arrange the shape so that none of the sides of the retroreflective portion overlaps any of the sides of one pixel of the liquid crystal display panel.

[0025] The surface shape of the retroreflector according to this embodiment is not limited to the above example. Various surface shapes that achieve retroreflection may be used. Specifically, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which triangular pyramidal prisms, hexagonal pyramidal prisms, other polygonal prisms, or a combination thereof are periodically arranged. Alternatively, the surface of the retroreflector according to this embodiment may be provided with retroreflection elements in which these prisms are periodically arranged to form cube corners. These may also be referred to as corner reflector arrays or polyhedral reflector arrays. Alternatively, the surface of the retroreflector according to this embodiment may be provided with capsule lens-type retroreflection elements in which glass beads are periodically arranged. The detailed configuration of these retroreflection elements can be achieved using existing technology, so a detailed description will be omitted. Specifically, the techniques disclosed in Japanese Patent Laid-Open Nos. 2001-33609, 2001-264525, 2005-181555, 2008-70898, and 2009-229942 may be used.

[0026] <Another configuration example 1 of the optical system of the space-floating image display device> Another configuration example of the optical system of the space-floating image display device will be described using Fig. 2B. In Fig. 2B, components with the same reference numerals as Fig. 2A have the same functions and configurations as Fig. 2A. For such components, repeated explanations will be omitted to simplify the explanation.

[0027] In the optical system of FIG. 2B , similar to FIG. 2A , image light of a specific polarization is output from the display device 1. The image light of a specific polarization output from the display device 1 is input to a polarization separation member 101B. The polarization separation member 101B is a member that selectively transmits image light of a specific polarization. Unlike the polarization separation member 101 of FIG. 2A , the polarization separation member 101B is not integrated with the transparent member 100 but has an independent plate-like shape. Therefore, the polarization separation member 101B may also be referred to as a polarization separation plate. The polarization separation member 101B may be configured as a reflective polarizing plate configured by attaching a polarization separation sheet to a transparent member. Alternatively, the transparent member may be formed of a metal multilayer film that selectively transmits specific polarization and reflects polarization of other specific polarizations. In FIG. 2B , the polarization separation member 101B is configured to transmit image light of a specific polarization output from the display device 1.

[0028] The image light that has passed through the polarization separation member 101B is incident on the retroreflector 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflector. The image light is polarized and converted from a specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when it enters the retroreflector and once when it leaves. Here, the polarization separation member 101B has the property of reflecting the polarized light of the other polarization that has been polarized and converted by the λ / 4 plate 21, so the image light after polarization conversion is reflected by the polarization separation member 101B. The image light reflected by the polarization separation member 101B passes through the transparent member 100 and forms a space-floating image 3, which is a real image, outside the transparent member 100.

[0029] Here, a first example of polarization design for the optical system of FIG. 2B will be described. For example, a configuration may be adopted in which P-polarized image light is emitted from the display device 1 to the polarization separation member 101B, and the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light. In this case, the P-polarized image light that reaches the polarization separation member 101B from the display device 1 passes through the polarization separation member 101B and proceeds to the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, and the image light is converted from P-polarized light to S-polarized light. The image light converted to S-polarized light proceeds again to the polarization separation member 101B. Here, the polarization separation member 101B has the property of reflecting S-polarized light and transmitting P-polarized light, so the S-polarized image light is reflected by the polarization separation member 101 and passes through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the floating image 3 to be formed optimally.

[0030] Next, a second example of polarization design for the optical system of FIG. 2B will be described. For example, S-polarized image light may be emitted from the display device 1 to the polarization separator 101B, which may have the property of reflecting P-polarized light and transmitting S-polarized light. In this case, the S-polarized image light reaching the polarization separator 101B from the display device 1 passes through the polarization separator 101B and proceeds to the retroreflector 2. When the image light is reflected by the retroreflector 2, it passes twice through the λ / 4 plate 21 provided on the incident surface of the retroreflector 2, thereby converting the S-polarized image light to P-polarized light. The P-polarized image light then proceeds again to the polarization separator 101B. Here, the polarization separator 101B has the property of reflecting P-polarized light and transmitting S-polarized light, so the P-polarized image light is reflected by the polarization separator 101 and passes through the transparent member 100. The image light transmitted through the transparent member 100 is light generated by the retroreflector 2, and therefore forms a floating image 3, which is an optical image of the image displayed on the display device 1, at a position that is in a mirror relationship with the image displayed on the display device 1 relative to the polarization separation member 101B. This polarization design allows the floating image 3 to be formed optimally.

[0031] In FIG. 2B , the image display surface of the display device 1 and the surface of the retroreflector 2 are arranged parallel to each other. The polarization separation member 101B is arranged at an angle α (e.g., 30°) with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. Then, upon reflection by the polarization separation member 101B, the direction of travel of the image light reflected by the polarization separation member 101B (the direction of the chief ray of the image light) differs by an angle β (e.g., 60°) from the direction of travel of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light). With this configuration, in the optical system of FIG. 2B , the image light is output at a predetermined angle shown toward the outside of the transparent member 100, forming a real image, the floating-in-space image 3. In the configuration of FIG. 2B , when viewed by a user from the direction of arrow A, the floating-in-space image 3 is perceived as a bright image. However, when viewed by another person from the direction of arrow B, the floating-in-space image 3 cannot be perceived as an image at all. This characteristic is extremely suitable for use in a system that displays images that require high security or highly confidential images that should be concealed from people directly facing the user.

[0032] As described above, the optical system of FIG. 2B has a different configuration from the optical system of FIG. 2A, but can form a suitable floating image in space, similar to the optical system of FIG. 2A.

[0033] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized light of the image light from the polarization separation member 101B and absorb the polarized light that is 90° out of phase with the polarized light of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2B due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0034] <Another configuration example 2 of the optical system of the space-floating image display device> Another configuration example of the optical system of the space-floating image display device will be described using Fig. 2C. Note that in Fig. 2C, components with the same reference numerals as Fig. 2B have the same functions and configurations as Fig. 2B. For the sake of simplicity, such components will not be described repeatedly.

[0035] The only difference between the optical system of Fig. 2B and the optical system of Fig. 2C is the arrangement angle of the polarization separation member 101B with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. All other configurations are the same as those of the optical system of Fig. 2B, so repeated explanations will be omitted. The polarization design of the optical system of Fig. 2C is also the same as that of the optical system of Fig. 2B, so repeated explanations will be omitted.

[0036] In the optical system of FIG. 2C , the polarization separation member 101B is tilted at an angle α with respect to the image display surface of the display device 1 and the surface of the retroreflector 2. In FIG. 2C , the angle α is 45°. With this configuration, when the polarization separation member 101B reflects, the angle β between the direction of propagation of the image light incident from the retroreflector 2 (the direction of the chief ray of the image light) and the direction of propagation of the image light reflected by the polarization separation member 101B (the direction of the chief ray of the image light) is 90°. With this configuration, the image display surface of the display device 1 and the surface of the retroreflector 2 are perpendicular to the direction of propagation of the image light reflected by the polarization separation member 101B, thereby simplifying the angular relationship between the surfaces that make up the optical system. If the surface of the transparent member 100 is positioned perpendicular to the direction of propagation of the image light reflected by the polarization separation member 101B, the angular relationship between the surfaces that make up the optical system can be further simplified. In the configuration of Figure 2C, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0037] As described above, the optical system of Fig. 2C has a different configuration from the optical systems of Fig. 2A and Fig. 2B, but can form a suitable floating image in space similar to the optical systems of Fig. 2A and Fig. 2B. In addition, the angles of the surfaces constituting the optical system can be made simpler.

[0038] An absorptive polarizing plate may be provided on the surface of the transparent member 100 facing the polarization separation member 101B. This absorptive polarizing plate may transmit the polarized light of the image light from the polarization separation member 101B and absorb the polarized light that is 90° out of phase with the polarized light of the image light from the polarization separation member 101B. In this way, the image light for forming the space-floating image 3 can be sufficiently transmitted while reducing the external light incident on the space-floating image 3 side of the transparent member 100 by approximately 50%. This makes it possible to reduce stray light in the optical system of FIG. 2C due to the external light incident on the space-floating image 3 side of the transparent member 100.

[0039] <Another Configuration Example 3 of the Optical System of the Space-Floating Image Display Device> Another configuration example of the optical system of the space-floating image display device will be described using FIG. 2D. The optical system of FIG. 2D is an optical system that uses a retroreflector 5 that is different from the retroreflector 2 used in FIGS. 2A to 2C. Hereinafter, Another Configuration Example 3 of the optical system will be described in more detail using FIGS. 2D to 2I. In FIG. 2D, components that are assigned the same reference numerals as those in FIGS. 2A to 2C have the same functions and configurations as those in FIGS. 2A to 2C. Such components will not be described repeatedly in order to simplify the explanation.

[0040] 2D is a diagram showing an example of the main components and retroreflection components of a space-floating image display device according to an embodiment of the present invention. A display device 10 that emits image light is provided obliquely on a transparent member 100 such as glass. The display device 10 includes a liquid crystal display panel 11 and a light source device 13 that generates light.

[0041] A chief ray 9020 representing the light beam emitted from the display device 10 travels toward the retroreflector 5 and is incident on the retroreflector 5 at an incident angle α. The incident angle α may be, for example, 45°. However, the incident angle α is not limited to 45°, and may also be, for example, 45°±15°.

[0042] The retroreflector 5 is an optical element having the optical property of retroreflecting light rays in at least some directions. Furthermore, since the reflected light rays have the optical property of forming an image, the retroreflector 5 may also be referred to as an imaging optical element or an imaging optical plate.

[0043] 2E, 2F, etc., the principal ray 9020 travels in the z direction and is retroreflected in the x and y directions by the retroreflector 5. As a result, the reflected light ray 9021 travels in a direction away from the retroreflector 5 along an optical path that is mirror-symmetrical with respect to the principal ray 9020 with the retroreflector 5 as the reference, passes through the transparent member 100, and forms the floating image 3 in space as a real image on the imaging plane.

[0044] The light beam forming the space-floating image 3 is a collection of light rays that converge from the retroreflector 5 to the optical image of the space-floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the space-floating image 3. Therefore, the space-floating image 3 is an image with high directionality, unlike a diffuse image formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2, when a user views the space-floating image 3 from the direction of arrow A, the space-floating image 3 is perceived as a bright image. However, when another person views the space-floating image 3 from the direction of arrow B, the space-floating image 3 cannot be perceived as an image at all. This characteristic is suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0045] An example of the configuration of the retroreflector 5 will be described using Figures 2E and 2F. The retroreflector 5 has a configuration in which a plurality of corner reflectors 9040 are arranged in an array on the surface of a transparent member 50. This may also be called a corner reflector array or a polyhedral reflector array. The specific configuration of the corner reflector 9040 will be described in detail using Figures 2G, 2H, and 2I. Light rays 9111, 9112, 9113, and 9114 emitted from a light source 9110 are reflected twice by two mirror surfaces 9041 and 9042 of the corner reflector 9040, becoming reflected light rays 9121, 9122, 9123, and 9124. This double reflection is retroreflection in the x and y directions, where the light is reflected back in the same direction as the incident direction (traveling in a direction rotated 180 degrees), and in the z direction, where the angle of incidence and the angle of reflection match due to total reflection.

[0046] That is, the light rays 9111 to 9114 generate reflected light rays 9121 to 9124 on straight lines symmetrical in the z direction with respect to the corner reflector 9040, forming an aerial real image 9120. Note that the light rays 9111 to 9114 emitted from the light source 9110 are four light rays representing the diffused light from the light source 9110, and although the light rays incident on the retroreflector 5 are not limited to these, depending on the diffusion characteristics of the light source 9110, all incident light rays cause similar reflections and form an aerial real image 9120. Note that to make the drawing easier to see, the position of the light source 9110 and the position of the aerial real image 9120 are shown shifted in the x direction, but in reality, the position of the light source 9110 and the position of the aerial real image 9120 in the x direction are the same, and are overlapping when viewed from the z direction.

[0047] 2G, 2H, and 2I, the configuration and effects of the corner reflector 9040 that constitutes the retroreflector 5 will be described. The corner reflector 9040 is a rectangular parallelepiped with only two specific surfaces being mirror surfaces 9041 and 9042, and the other four surfaces being made of transparent materials. The retroreflector 5 has a configuration in which these corner reflectors 9040 are arrayed so that the corresponding mirror surfaces face in the same direction.

[0048] When viewed from the top (+z direction), a light ray 9111 emitted from the light source 9110 enters the mirror surface 9041 (or the mirror surface 9042) at a specific angle of incidence, is totally reflected at the reflection point 9130, and then is totally reflected again at the reflection point 9132 on the mirror surface 9042 (or the mirror surface 9041).

[0049] If the angle of incidence of light ray 9111 with respect to mirror surface 9041 (or mirror surface 9042) is θ, then the angle of incidence of first reflected light ray 9131 reflected by mirror surface 9041 (or mirror surface 9042) with respect to mirror surface 9042 (or mirror surface 9041) can be expressed as 90°-θ. Therefore, with respect to light ray 9111, second reflected light ray 9121 undergoes a rotation of 2θ after the first reflection and 2×(90°-θ) after the second reflection, resulting in a total reversal optical path of 180°. On the other hand, when viewed from the side (the direction halfway between -x and -y), total reflection in the z direction occurs only once. Therefore, if the angle of incidence with respect to mirror surface 9041 or mirror surface 9042 is φ, then reflected light ray 9121 undergoes a rotation of 2×φ after one reflection with respect to light ray 9111.

[0050] As described above, the light rays incident on the corner reflector 9040 undergo retroreflection, which results in an inverted optical path in the x and y directions, and specular reflection due to total reflection in the z direction. Considering the retroreflector 5, similar reflection occurs in each optical path, so that an image is formed at a point symmetrical with respect to the z axis direction by an inverted optical path that is convergent in the x and y directions.

[0051] 2A to 2C, the retroreflector 2 has retroreflection properties in three axes. As a result, when a diffusive incident light beam is incident on the retroreflector 2, a convergent reflected light beam travels toward the side of the retroreflector 2 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air, forming a floating image 3. The traveling direction of the chief ray of the convergent reflected light beam reflected from the retroreflector 2 is opposite to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 2.

[0052] 2D, the retroreflector 5 has retroreflection properties in two axial directions and specular reflection in the other axial direction. As a result, when a diffusive incident light beam is incident on the retroreflector 5, the convergent reflected light beam is reflected by the corner reflector array and travels in the direction opposite to the side of the retroreflector 5 where the light source of the incident light is located. The convergent reflected light beam forms an image in the air and forms the space floating image 3.

[0053] The traveling direction of the chief ray of the convergent reflected light beam reflected by the corner reflector array of the retroreflector 5 is not the opposite direction to the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5. The normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray of the diffusive incident light beam that is incident on the retroreflector 5 and the normal direction component of the plate-shaped surface of the retroreflector 5 in the traveling direction of the chief ray after being reflected by the retroreflector 5 and becoming a convergent reflected light beam, travel in a straight line, unchanged before and after reflection by the corner reflector array.

[0054] That is, the diffusive incident light beam is converted into a convergent reflected light beam by reflection on the retroreflector 5, but in the normal direction to the plate-shaped surface of the retroreflector 5, the light beam travels as if passing through the retroreflector 5. Here, the diffusive incident light beam incident on the retroreflector 5 and the convergent reflected light beam emerging from the retroreflector 5 are in a geometrically symmetrical relationship with respect to the plate-shaped surface of the retroreflector 5.

[0055] The resolution of the space-floating image formed by the light beams from the video output unit 10 depends not only on the resolution of the liquid crystal display panel 11, but also on the diameter D and pitch P (not shown) of the retroreflective portion of the retroreflector 5 shown in Figures 2E and 2F. For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch P is 300 μm, one pixel of the space-floating image will be equivalent to 300 μm. As a result, the effective resolution of the space-floating image will be reduced to about one-third.

[0056] Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 10, it is desirable to make the diameter D and pitch P of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire due to the retroreflective plate and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Furthermore, it is advisable to arrange the shape of the retroreflective portion so that none of its sides overlaps any of the sides of one pixel of the liquid crystal display panel.

[0057] The shape of the retroreflector (imaging optical plate) according to this embodiment is not limited to the above example. It may have various shapes that achieve retroreflection. Specifically, it may be a variety of cubic corner bodies, corner reflector arrays, slit mirror arrays, dihedral corner reflector arrays, polyhedral reflector arrays, or shapes in which a combination of these reflective surfaces is periodically arranged. Alternatively, capsule lens-type retroreflecting elements in which glass beads are periodically arranged may be provided on the surface of the retroreflector according to this embodiment. Since existing technology can be used for the detailed configuration of these retroreflecting elements, a detailed description will be omitted. Specifically, the technology disclosed in JP 2017-33005 A, JP 2019-133110 A, JP 2017-67933 A, WO 2009 / 131128 A, etc. may be used.

[0058] 2D, the image light emitted from the display device 10 may be in any polarization state, either S-polarized or P-polarized.

[0059] As described above, the optical system of FIG. 2D is an optical system that uses a retroreflector different from the optical systems of FIGS. 2A to 2C, but it can form a more suitable floating image in space, similar to the optical systems of FIGS. 2A to 2C.

[0060] According to the optical systems of FIGS. 2A, 2B, 2C, and 2D described above, it is possible to provide a brighter, higher quality floating image in space.

[0061] <<Block diagram of the internal configuration of the space floating image display device>>

[0062] Next, a description will be given of a block diagram of the internal configuration of the space-floating image display device 1000. Fig. 3 is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

[0063] The space-floating image display device 1000 includes a retroreflection unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an external power supply input interface 1111, an operation input unit 1107, a nonvolatile memory 1108, a memory 1109, a control unit 1110, a video signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, a microphone 1139, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc. In addition, the space-floating image display device 1000 may also include a removable media interface 1134, an attitude sensor 1113, a transmissive self-luminous image display device 1650, a second display device 1680, or a secondary battery 1112.

[0064] Each component of the space floating image display device 1000 is disposed in a housing 1190. Note that the imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0065] The retroreflecting portion 1101 in Fig. 3 corresponds to the retroreflector 2 in Fig. 2A, Fig. 2B, and Fig. 2C. The retroreflecting portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected from the retroreflecting portion 1101, the light output to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0066] The image display unit 1102 in Fig. 3 corresponds to the liquid crystal display panel 11 in Fig. 2A, 2B, and 2C. The light source 1105 in Fig. 3 corresponds to the light source device 13 in Fig. 2A, 2B, and 2C. The image display unit 1102, the light guide 1104, and the light source 1105 in Fig. 3 correspond to the display device 1 in Fig. 2A, 2B, and 2C.

[0067] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 (described later). The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIGS. 2A, 2B, and 2C. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0068] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power source 1106 converts AC current input from the outside via the external power input interface 1111 into DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each component within the space-floating image display device 1000. The secondary battery 1112 stores the power supplied from the power source 1106. The secondary battery 1112 also supplies power to the light source 1105 and other components requiring power via the external power input interface 1111 when power is not supplied from the outside. In other words, when the space-floating image display device 1000 is equipped with the secondary battery 1112, the user can use the space-floating image display device 1000 even when power is not supplied from the outside.

[0069] The light guide 1104 guides light generated by the light source 1105 and irradiates it onto the video display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called a backlight for the video display unit 1102. The light guide 1104 may be configured mainly using glass. The light guide 1104 may be configured mainly using plastic. The light guide 1104 may be configured using a mirror. Various methods are possible for combining the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0070] The aerial operation detection sensor 1351 is a sensor that detects an operation of the floating in space image 3 by the finger of the user 230. The aerial operation detection sensor 1351 senses, for example, an area that overlaps with the entire display area of ​​the floating in space image 3. Note that the aerial operation detection sensor 1351 may only sense an area that overlaps with at least a portion of the display area of ​​the floating in space image 3.

[0071] Specific examples of the aerial operation detection sensor 1351 include a distance sensor that uses invisible light such as infrared light, an invisible light laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may also be configured to detect coordinates on a two-dimensional plane by combining multiple sensors. The aerial operation detection sensor 1351 may also be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) sensor or an image sensor.

[0072] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect touch operations, etc., made by the user with their finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0073] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and determines whether or not the finger of the user 230 has made contact with an object in the floating-in-space image 3 based on the sensing signal, and calculates the position (contact position) where the finger of the user 230 has made contact with the object. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array), for example. Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0074] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally as separate entities from the space-floating image display device 1000. When provided as separate entities from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to be able to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0075] Furthermore, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the space-floating image display device 1000, which does not have an aerial operation detection function, is used as the main body, and only the aerial operation detection function can be added as an option. Furthermore, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the space-floating image display device 1000 may be used. In cases where it is desired to more freely position the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0076] The imaging unit 1180 is a camera with an image sensor, and captures images of the space near the floating-in-space image 3 and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180, or by using an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in detecting the touch operation of the floating-in-space image 3 by the user 230. The imaging unit 1180 may be provided separately from the floating-in-space image display device 1000. When the imaging unit 1180 is provided separately from the floating-in-space image display device 1000, it is sufficient to configure it so that an imaging signal can be transmitted to the floating-in-space image display device 1000 via a wired or wireless communication connection path, etc.

[0077] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that targets a plane (intrusion detection plane) including the display surface of the spatial floating image 3 and detects whether or not an object has intruded into this intrusion detection plane, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object that has not intruded into the intrusion detection plane (for example, a user's finger) is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0078] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the images captured by the multiple image capturing units 1180 and object depth information from the depth sensor. These pieces of information and various other information such as the distance between the object and the intrusion detection plane are then used for various display controls for the floating in space image 3.

[0079] Furthermore, without using the aerial operation detection sensor 1351 , the aerial operation detection unit 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging unit 1180 .

[0080] Furthermore, the imaging unit 1180 may capture an image of the face of the user 230 operating the space-floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not there is another person standing around or behind the user 230 operating the space-floating image 3 and peeking at the operation of the user 230 on the space-floating image 3, the imaging unit 1180 may capture an image of the user 230 operating the space-floating image 3 and a range including the area surrounding the user 230.

[0081] The operation input unit 1107 is, for example, an operation button, a signal receiving unit such as a remote controller, or an infrared light receiving unit, and inputs signals for operations different from the aerial operation (touch operation) by the user 230. Apart from the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may also be used by, for example, an administrator to operate the space floating image display device 1000.

[0082] The video signal input unit 1131 is connected to an external video output device and inputs video data. Various digital video input interfaces are possible for the video signal input unit 1131. For example, the video signal input unit 1131 may be configured as a video input interface conforming to the HDMI (registered trademark) standard (High-Definition Multimedia Interface), a video input interface conforming to the DVI (Digital Visual Interface) standard, or a video input interface conforming to the DisplayPort standard.

[0083] Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio signal input unit 1133 may be configured as an HDMI-standard audio input interface, an optical digital terminal interface, a coaxial digital terminal interface, or the like. In the case of an HDMI-standard interface, the video signal input unit 1131 and the audio signal input unit 1133 may be configured as an interface in which a terminal and a cable are integrated. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may be configured as a speaker.

[0084] The audio output unit 1140 may also output built-in operation sounds or error warning sounds. Alternatively, the audio output unit 1140 may be configured to output a digital signal to an external device, like the Audio Return Channel function defined in the HDMI standard. The microphone 1139 is a microphone that collects sounds around the space-floating image display device 1000, converts them into signals, and generates audio signals. The microphone may record a person's voice, such as a user's voice, and the control unit 1110, described later, may perform voice recognition processing on the generated audio signal to obtain text information from the audio signal.

[0085] The non-volatile memory 1108 stores various data used by the space floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations to be displayed on the space floating image 3, display icons, data and layout information for objects to be operated by user operations, etc. The memory 1109 stores image data to be displayed as the space floating image 3, data for controlling the device, etc.

[0086] The control unit 1110 controls the operation of each connected unit. In addition, the control unit 1110 may cooperate with a program stored in the memory 1109 to perform calculations based on information acquired from each unit in the space floating image display device 1000.

[0087] The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless communication interface. If the communication unit 1132 has a wired communication interface, the wired communication interface may be configured, for example, by an Ethernet-standard LAN interface. If the communication unit 1132 has a wireless communication interface, the interface may be configured, for example, by a Wi-Fi communication interface, a Bluetooth communication interface, or a mobile communication interface such as 4G or 5G. Various types of data, such as video data, image data, and audio data, are transmitted and received through communication via the communication unit 1132.

[0088] The removable media interface 1134 is an interface for connecting a removable recording medium (removable media). The removable recording medium (removable media) may be composed of a semiconductor device memory such as a solid state drive (SSD), a magnetic recording medium recording device such as a hard disk drive (HDD), or an optical recording medium such as an optical disk. The removable media interface 1134 can read various information such as video data, image data, and audio data recorded on the removable recording medium. The video data, image data, etc. recorded on the removable recording medium are output as the floating image 3 via the video display unit 1102 and the retroreflection unit 1101.

[0089] The storage unit 1170 is a storage device that records various types of information, such as video data, image data, and audio data. The storage unit 1170 may be configured with a magnetic recording medium recording device, such as a hard disk drive (HDD), or a semiconductor element memory, such as a solid state drive (SSD). For example, various types of information, such as video data, image data, and audio data, may be recorded in advance in the storage unit 1170 at the time of product shipment. The storage unit 1170 may also record various types of information, such as video data, image data, and audio data, acquired from an external device, an external server, or the like via the communication unit 1132.

[0090] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflection unit 1101. The video data, image data, etc. of display icons, objects for the user to operate, etc., displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0091] Layout information of display icons, objects, etc. displayed as the spatial floating image 3, and various metadata information related to the objects, etc. are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0092] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. The video control unit 1160 may be referred to as a video processing circuit, and may be configured with hardware such as an ASIC, FPGA, or video processor. The video control unit 1160 may also be referred to as a video processing unit or an image processing unit. The video control unit 1160 controls video switching, such as determining which video signal to input to the video display unit 1102, between the video signal to be stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131, for example.

[0093] In addition, the image control unit 1160 may generate a superimposed image signal by superimposing the image signal to be stored in the memory 1109 and the image signal input from the image signal input unit 1131, and input the superimposed image signal to the image display unit 1102, thereby controlling the formation of a composite image as a floating image in space 3.

[0094] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing that enlarges, reduces, or deforms the image, brightness adjustment processing that changes the brightness, contrast adjustment processing that changes the contrast curve of the image, and Retinex processing that decomposes the image into light components and changes the weighting of each component.

[0095] Furthermore, the video control unit 1160 may perform special effect video processing or the like on the video signal input to the video display unit 1102 to assist the aerial operation (touch operation) of the user 230. The special effect video processing is performed based on, for example, the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 or the image of the user 230 captured by the imaging unit 1180.

[0096] The attitude sensor 1113 is a sensor configured with a gravity sensor, an acceleration sensor, or a combination of these, and can detect the attitude in which the space-floating image display device 1000 is installed. Based on the attitude detection result of the attitude sensor 1113, the control unit 1110 may control the operation of each connected unit. For example, if an undesirable attitude is detected as the user's usage state, the control unit 1110 may perform control such that the image displayed on the image display unit 1102 is stopped and an error message is displayed to the user. Alternatively, if the attitude sensor 1113 detects a change in the installation attitude of the space-floating image display device 1000, the control unit 1110 may perform control such that the display orientation of the image displayed on the image display unit 1102 is rotated.

[0097] As explained above, various functions are installed in the space floating image display device 1000. However, the space floating image display device 1000 does not need to have all of these functions, and any configuration is acceptable as long as it has the function of forming the space floating image 3.

[0098] <Configuration Example of Space-Floating Image Display Device> Next, a configuration example of the space-floating image display device will be described. The layout of the components of the space-floating image display device according to this embodiment can be various depending on the usage form. Below, the layouts of each of Figs. 4A to 4M will be described. Note that in each of the examples of Figs. 4A to 4M, the thick line surrounding the space-floating image display device 1000 indicates an example of the housing structure of the space-floating image display device 1000.

[0099] FIG. 4A is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4A is equipped with an optical system corresponding to the optical system shown in FIG. 2A. The space-floating image display device 1000 shown in FIG. 4A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4A, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user's finger. Note that the x direction is the left-right direction as seen from the user, the y direction is the front-back direction (depth direction) as seen from the user, and the z direction is the up-down direction (vertical direction). Hereinafter, the definitions of the x direction, y direction, and z direction are the same in each drawing of FIG. 4, so repeated explanations will be omitted.

[0100] FIG. 4B is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4B is equipped with an optical system corresponding to the optical system of FIG. 2A. The space-floating image display device 1000 shown in FIG. 4B is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4B, the space-floating image display device is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user 230's finger. 4B , the aerial operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so that this configuration can improve the accuracy of touch detection.

[0101] FIG. 4C is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4C is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4C is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4C, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is installed as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user's 230 finger.

[0102] FIG. 4D is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4D is equipped with an optical system corresponding to the optical system of FIG. 2B. The space-floating image display device 1000 shown in FIG. 4D is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4D, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger. 4D , the aerial operation detection sensor 1351 can use the reflection of sensing light by the user's nail for touch detection by sensing the finger of the user 230 from above. Generally, the reflectivity of the nail is higher than that of the finger pad, so that this configuration can improve the accuracy of touch detection.

[0103] FIG. 4E is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4E is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4E is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 4E, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels directly upward. When the mid-air operation detection sensor 1351 is provided as shown in the figure, it is possible to detect the operation of the space-floating image 3 by the user's 230 finger.

[0104] FIG. 4F is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4F is equipped with an optical system corresponding to the optical system of FIG. 2C. The space-floating image display device 1000 shown in FIG. 4F is installed vertically so that the surface on which the space-floating image 3 is formed faces the front of the space-floating image display device 1000 (toward the user 230). That is, in FIG. 4F, the space-floating image display device 1000 is installed with the transparent member 100 facing the front of the device (toward the user 230). The space-floating image 3 is formed on the user 230 side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels toward the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the user 230's finger.

[0105] FIG. 4G is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 shown in FIG. 4G is equipped with an optical system corresponding to the optical system shown in FIG. 2C. In the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the central optical path of the image light emitted from the display device 1 was on the yz plane. That is, in the optical systems of the space-floating image display devices shown in FIGS. 4A to 4F, the image light traveled in the front-to-back and up-to-down directions as seen from the user. In contrast, in the optical system of the space-floating image display device shown in FIG. 4G, the central optical path of the image light emitted from the display device 1 is on the xy plane. That is, in the optical system of the space-floating image display device shown in FIG. 4G, the image light travels in the left-to-right and front-to-back directions as seen from the user. In the space-floating image display device 1000 shown in FIG. 4G, the surface on which the space-floating image 3 is formed is installed so that it faces the front of the device (toward the user 230). 4G, the space-floating image display device 1000 has the transparent member 100 installed on the front side of the device (toward the user 230). The space-floating image 3 is formed on the user side of the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels towards the user. If the mid-air operation detection sensor 1351 is installed as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0106] 4H is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4H differs from the space-floating image display device of FIG. 4G in that it has a window with a transparent plate 100B made of glass or plastic on the back of the device (opposite the position where the user 230 views the space-floating image 3, i.e., opposite the traveling direction of the image light of the space-floating image 3 toward the user 230). The rest of the configuration is the same as the space-floating image display device of FIG. 4G , so repeated explanations will be omitted. The space-floating image display device 1000 of FIG. 4H has a window with a transparent plate 100B on the opposite side of the traveling direction of the image light of the space-floating image 3 from the space-floating image 3. Therefore, when the user 230 views the space-floating image 3, they can recognize the scenery behind the space-floating image display device 1000 as the background of the space-floating image 3. Therefore, the user 230 can perceive the space floating image 3 as floating in the air in front of the scenery behind the space floating image display device 1000. This can further emphasize the floating feeling of the space floating image 3.

[0107] Depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and head toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, in order to prevent this stray light, the transparent plate 100B may not be provided in the window on the back of the space-floating image display device 1000.

[0108] Fig. 4I is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4I differs from the space-floating image display device of Fig. 4H in that a light-blocking door 1410 is provided in the window of the transparent plate 100B located on the back of the device (opposite the position where the user 230 views the space-floating image 3). The rest of the configuration is the same as that of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted.

[0109] The opening and closing door 1410 of the space-floating image display device 1000 in FIG. 4I has, for example, a light blocking plate and is equipped with a mechanism for moving (sliding), rotating, or attaching / detaching the light blocking plate, thereby switching between an open state and a light blocking state for the window (rear window) of the transparent plate 100B located at the back of the space-floating image display device 1000. The movement (sliding) and rotation of the light blocking plate by the opening and closing door 1410 may be electrically driven by a motor (not shown). The motor may be controlled by the control unit 1110 in FIG. 3. Note that the example in FIG. 4I discloses an example in which the opening and closing door 1410 has two light blocking plates. In contrast, the opening and closing door 1410 may have only one light blocking plate.

[0110] For example, if the view seen through the window of the transparent plate 100B of the space-floating image display device 1000 is outdoors, the brightness of sunlight varies depending on the weather. When the outdoor sunlight is strong, the background of the space-floating image 3 may become too bright, reducing the user 230's visibility of the space-floating image 3. In such a case, by moving (sliding), rotating, or attaching the light-shielding plate of the opening / closing door 1410 to block the light from the rear window, the background of the space-floating image 3 becomes dark, thereby relatively increasing the visibility of the space-floating image 3. Such a blocking operation by the light-shielding plate of the opening / closing door 1410 may be performed directly by the force of the user 230's hand. In response to an operation input via the operation input unit 1107 of FIG. 3 , the control unit 1110 may control a motor (not shown) to perform the blocking operation by the light-shielding plate of the opening / closing door 1410.

[0111] An illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 may control a motor (not shown) to perform the opening and closing operation of the light blocking plate of the opening and closing door 1410 according to the detection result of the illuminance sensor. By controlling the opening and closing operation of the light blocking plate of the opening and closing door 1410 in this way, it is possible to more suitably maintain the visibility of the space-floating image 3, even if the user 230 does not manually open or close the light blocking plate of the opening and closing door 1410.

[0112] Furthermore, the light blocking plate provided by the opening and closing door 1410 may be manually detachable. Depending on the intended use and installation environment of the space floating image display device 1000, the user can select whether to leave the rear window open or in a light blocking state. If it is planned to use the rear window in a light blocking state for a long period of time, the detachable light blocking plate can be fixed in the light blocking state. Furthermore, if it is planned to use the rear window in an open state for a long period of time, the detachable light blocking plate can be left detached. The light blocking plate may be attached and detached using screws, a hook structure, or a fitting structure.

[0113] Even in the example of the space-floating image display device 1000 shown in FIG. 4I, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, a portion of the image light output from the display device 1 may be reflected by the polarization separation member 101B and directed toward the transparent plate 100B. Depending on the coating performance of the surface of the transparent plate 100B, this light may be reflected again by the surface of the transparent plate 100B and be visible to the user as stray light. Therefore, to prevent this stray light, the window on the back of the space-floating image display device 1000 may be configured without the transparent plate 100B. The above-described opening / closing door 1410 may be provided in a window that does not have the transparent plate 100B. To prevent this stray light, it is desirable that the inner surface of the housing of the light-shielding plate of the above-described opening / closing door 1410 have a coating or material with low light reflectance.

[0114] Fig. 4J is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4J differs from the space-floating image display device of Fig. 4H in that instead of arranging a transparent plate 100B made of glass or plastic on the rear side window, an electronically controlled transmittance variable device 1620 is arranged. The other configurations are the same as those of the space-floating image display device of Fig. 4H, so repeated explanations will be omitted. An example of the electronically controlled transmittance variable device 1620 is a liquid crystal shutter.

[0115] That is, the liquid crystal shutter can control the amount of light transmitted by controlling the voltage of the liquid crystal element sandwiched between two polarizing plates. Therefore, by controlling the liquid crystal shutter to increase the transmittance, the scenery through the rear window can be seen through the background of the space-floating image 3. Furthermore, by controlling the liquid crystal shutter to increase the transmittance, the scenery through the rear window can be hidden as the background of the space-floating image 3. Furthermore, since the liquid crystal shutter can control the intermediate length, it can also be set to a transmittance of 50% or more. For example, the control unit 1110 can control the transmittance of the electronically controlled transmittance variable device 1620 in response to an operation input via the operation input unit 1107 of FIG. 3 . With this configuration, in cases where a view through the rear window is desired as the background of the space-floating image 3, but the view through the background rear window is too bright and reduces the visibility of the space-floating image 3, the visibility of the space-floating image 3 can be adjusted by adjusting the transmittance of the electronically controlled transmittance variable device 1620.

[0116] In addition, an illuminance sensor may be provided on the rear side (opposite the user 230) of the space-floating image display device 1000, such as near the rear window, to measure the brightness of the space beyond the rear window. In this case, the control unit 1110 of Fig. 3 controls the transmittance of the electronically controlled transmittance variable device 1620 according to the detection result of the illuminance sensor. In this way, even if the user 230 does not perform an operation input via the operation input unit 1107 of Fig. 3, the transmittance of the electronically controlled transmittance variable device 1620 can be adjusted according to the brightness of the space beyond the rear window, making it possible to more suitably maintain the visibility of the space-floating image 3.

[0117] In the above example, a liquid crystal shutter has been described as an example of the electronically controlled transmittance varying device 1620. However, electronic paper may be used as another example of the electronically controlled transmittance varying device 1620. The same effect as described above can be obtained when electronic paper is used. Furthermore, electronic paper consumes very little power to maintain a halftone state. Therefore, a space floating image display device with lower power consumption can be realized compared to when a liquid crystal shutter is used.

[0118] Fig. 4K is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4K differs from the space-floating image display device of Fig. 4G in that it has a transmissive self-luminous image display device 1650 instead of the transparent member 100. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0119] In the space-floating image display device 1000 of FIG. 4K, an image luminous flux passes through the display surface of the transmissive self-luminous image display device 1650, and then a space-floating image 3 is formed outside the space-floating image display device 1000. That is, when an image is displayed on the transmissive self-luminous image display device 1650, which is a two-dimensional flat display, the space-floating image 3 can be displayed as a pop-up image further in front of the image displayed on the transmissive self-luminous image display device 1650. In this case, the user 230 can simultaneously view two images at different depth positions. The transmissive self-luminous image display device 1650 may be configured using existing technology, such as a transmissive organic EL panel, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2014-216761. Although the transmissive self-luminous image display device 1650 is not shown in FIG. 3, it may be configured to be connected to other processing units, such as the control unit 1110, as a component of the space-floating image display device 1000 of FIG. 3.

[0120] Here, if the transmissive self-luminous image display device 1650 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise image experience.

[0121] Furthermore, if the interior of the space-floating image display device 1000 is kept in a light-blocking state, the background of the transmissive self-luminous image display device 1650 will be sufficiently dark. Therefore, when no image is displayed on the display device 1 or the light source of the display device 1 is turned off and an image is displayed only on the transmissive self-luminous image display device 1650, the transmissive self-luminous image display device 1650 appears to the user 230 as a normal two-dimensional flat display rather than a transmissive display (since the space-floating image 3 in the embodiment of the present invention is displayed as a real optical image in a space without a screen, if the light source of the display device 1 is turned off, the intended display position of the space-floating image 3 will become empty space). Therefore, when the transmissive self-luminous image display device 1650 is used to display an image as if it were a normal two-dimensional flat display, characters, objects, etc. can suddenly be displayed in the air as the space-floating image 3, thereby providing the user 230 with a more effective surprise video experience.

[0122] Note that the darker the interior of the space-floating image display device 1000, the more the transmissive self-luminous image display device 1650 appears like a two-dimensional flat display. Therefore, an absorptive polarizer (not shown) that transmits the polarized light of the image light reflected by the polarization separation member 101B and absorbs polarized light that is 90° out of phase with the polarized light may be provided on the interior surface of the transmissive self-luminous image display device 1650 (the surface on which the image light reflected by the polarization separation member 101B enters the transmissive self-luminous image display device 1650, i.e., the surface of the transmissive self-luminous image display device 1650 opposite the space-floating image 3). While this does not have a significant effect on the image light that forms the space-floating image 3, it can significantly reduce the light that enters the space-floating image display device 1000 from the outside through the transmissive self-luminous image display device 1650, making the interior of the space-floating image display device 1000 darker, which is preferable.

[0123] 4L is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of FIG. 4L is a modified example of the space-floating image display device of FIG. 4K. The orientation of the components in the space-floating image display device 1000 is different from that of the space-floating image display device of FIG. 4K, and is closer to the arrangement of the space-floating image display device of FIG. 4F. The functions and operations of each component are the same as those of the space-floating image display device of FIG. 4K, so repeated explanations will be omitted.

[0124] In the space-floating image display device of FIG. 4L, after the luminous flux of image light passes through the transmissive self-luminous image display device 1650, a space-floating image 3 is formed on the user 230 side of the transmissive self-luminous image display device 1650.

[0125] In both the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, the space-floating image 3 is displayed superimposed on the image of the transmissive self-luminous image display device 1650 from the user 230. Here, the position of the space-floating image 3 and the position of the image of the transmissive self-luminous image display device 1650 are configured to have a difference in the depth direction. Therefore, when the user moves their head (the position of the viewpoint), they can recognize the depth of the two images due to parallax. Therefore, by displaying two images at different depth positions, it is possible to provide the user with a more suitable three-dimensional image experience with the naked eye without the need for stereoscopic glasses or the like.

[0126] Fig. 4M is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4M is provided with a second display device 1680 on the rear side as seen from the user relative to the polarization separation member 101B of the space-floating image display device of Fig. 4G. The other configurations are the same as those of the space-floating image display device of Fig. 4G, so repeated explanations will be omitted.

[0127] In the configuration example shown in Fig. 4M, the second display device 1680 is provided behind the display position of the space-floating image 3, and its image display surface faces the space-floating image 3. With this configuration, from the user 230's perspective, the image of the second display device 1680 and the space-floating image 3, which are displayed at two different depth positions, can be viewed superimposed on each other. In other words, the second display device 1680 is positioned so as to display an image in the direction of the user 230 viewing the space-floating image 3. Although the second display device 1680 is not shown in Fig. 3, it may be configured to be connected to other processing units such as the control unit 1110 as one component of the space-floating image display device 1000 of Fig. 3.

[0128] Note that the image light of the second display device 1680 of the space-floating image display device 1000 of FIG. 4M is viewed by the user 230 after passing through the polarization separation member 101B. Therefore, in order for the image light of the second display device 1680 to more suitably pass through the polarization separation member 101B, it is desirable that the image light output from the second display device 1680 be polarized with a polarization in a vibration direction that the polarization separation member 101B more suitably transmits. In other words, it is desirable that the image light be polarized with a polarization in the same vibration direction as the polarization of the image light output from the display device 1. For example, if the image light output from the display device 1 is S-polarized, it is desirable that the image light output from the second display device 1680 is also S-polarized. Furthermore, if the image light output from the display device 1 is P-polarized, it is desirable that the image light output from the second display device 1680 is also P-polarized.

[0129] The example of the space-floating image display device of FIG. 4M also has the same effect as the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L in that a second image is displayed behind the space-floating image 3. However, unlike the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L, in the example of the space-floating image display device of FIG. 4M, the luminous flux of image light for forming the space-floating image 3 does not pass through the second display device 1680. Therefore, the second display device 1680 does not need to be a transmissive self-luminous image display device, but may be a liquid crystal display, which is a two-dimensional flat display. The second display device 1680 may also be an organic EL display. Therefore, in the example of the space-floating image display device of FIG. 4M, it is possible to realize the space-floating image display device 1000 at a lower cost than the example of the space-floating image display device of FIG. 4K and the example of the space-floating image display device of FIG. 4L.

[0130] Here, depending on the polarization distribution of the image light output from the display device 1 and the performance of the polarization separation member 101B, there is a possibility that part of the image light output from the display device 1 will be reflected by the polarization separation member 101B and head toward the second display device 1680. This light (part of the image light) may be reflected again by the surface of the second display device 1680 and may be visually recognized by the user as stray light.

[0131] Therefore, to prevent this stray light, an absorptive polarizer may be provided on the surface of the second display device 1680. In this case, the absorptive polarizer may be an absorptive polarizer that transmits the polarized waves of the image light output from the second display device 1680 and absorbs polarized waves that are 90° out of phase with the polarized waves of the image light output from the second display device 1680. If the second display device 1680 is a liquid crystal display, an absorptive polarizer is also present on the image output side inside the liquid crystal display. However, if a cover glass (cover glass on the image display surface side) is further provided on the output surface of the absorptive polarizer on the image output side inside the liquid crystal display, it is not possible to prevent stray light caused by reflection of the cover glass by light from outside the liquid crystal display. Therefore, it is necessary to separately provide the above-mentioned absorptive polarizer on the surface of the cover glass.

[0132] When an image is displayed on the second display device 1680, which is a two-dimensional flat display, the floating-in-space image 3 can be displayed as an image further in front of the user in relation to the image on the second display device 1680. In this case, the user 230 can simultaneously view two images at different depth positions. By displaying a character on the floating-in-space image 3 and a background on the second display device 1680, it is possible to provide the effect that the user 230 is viewing the space in which the character exists in three dimensions.

[0133] Furthermore, if the second display device 1680 displays both the background and an object such as a character, and then the object such as the character moves to the foreground in the floating image 3, it is possible to provide the user 230 with a more effective surprise visual experience.

[0134] Next, Fig. 4N is a diagram showing an example of the configuration of a space-floating image display device. The space-floating image display device 1000 of Fig. 4N is a space-floating image display device that employs the optical system of Fig. 2D. As with the examples of space-floating image display devices that employ the optical systems of Figs. 2A to 2C, an image is formed in the air as a space-floating image 3 by image light that has passed through a transparent member 100. Furthermore, using sensing light from an aerial operation detection sensor 1351 that is located on the back side of the transparent member 100 as seen from the user, it is possible to detect operation of the space-floating image 3 by the user's finger 9004.

[0135] 2A to 2C, and in the example of the space-floating image display device employing the optical system of Fig. 2D, the space-floating image 3 is formed in front of the transparent member 100, and the operation of the space-floating image 3 by the user's finger can be detected using the sensing light of the mid-air operation detection sensor 1351 arranged on the back side of the transparent member 100 as seen from the user. Therefore, the space-floating image display device employing the optical system of Fig. 2D has a different optical system from the space-floating image display device in which the optical system of Fig. 2A to 2C is arranged on the back side of the transparent member 100 as seen from the user.

[0136] However, from the user's perspective, the usability of the space-floating image display device employing the optical system of FIG. 2D is almost the same as that of the space-floating image display device employing the optical system of FIGS. 2A to 2C.

[0137] Next, Fig. 4O is a diagram showing an example of the configuration of a space-floating image display device. Fig. 4O is a diagram showing the configuration of the internal optical system of the space-floating image display device 1000 of Fig. 4N. The space-floating image display device 1000 shown in Fig. 4O is equipped with an optical system corresponding to the optical system of Fig. 2D. The space-floating image display device 1000 shown in Fig. 4O is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward.

[0138] 4O, the space-floating image display device 1000 has a transparent member 100 installed on the top surface of the device. The space-floating image 3 is formed above the surface of the transparent member 100 of the space-floating image display device 1000. The light of the space-floating image 3 travels diagonally upward. When the mid-air operation detection sensor 1351 is provided as shown in the figure, it can detect the operation of the space-floating image 3 by the finger of the user 230.

[0139] Here, the configuration of Fig. 4O will be compared with the configuration of Fig. 4A to confirm the differences. In Fig. 4A, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the polarization separation member 101. In contrast, in Fig. 4O, the display device 1 and the space-floating image 3 are in a plane-symmetrical relationship with respect to the plane of the retroreflector 5. Furthermore, the configuration of Fig. 4A includes the retroreflector 2 and the λ / 4 plate 21, but these are not present in Fig. 4O. Furthermore, while the presence of the absorbing polarizer 12 is more preferable in Fig. 4A, the absorbing polarizer 12 is not particularly necessary in Fig. 4O.

[0140] That is, to replace the optical system of FIG. 2A in the configuration of FIG. 4A with the optical system of FIG. 2D and to replace it with the configuration of FIG. 4O, the following can be done. That is, the polarization separation member 101 in the configuration of FIG. 4A is replaced with the retroreflector 5, and the retroreflector 2 and the λ / 4 plate 21 are removed from the configuration of FIG. 4A. The absorptive polarizer 12 is optional. By performing a replacement based on this idea, the optical system of FIGS. 2A to 2C mounted in the configuration of the space-floating image display device of FIGS. 4A to 4G can be replaced with the optical system of FIG. 2D, and the space-floating image display device can be replaced with the optical system of FIG. 2D. In this case, the polarization separation member 101 in FIGS. 4A and 4B is replaced with the retroreflector 5, and the polarization separation member 101B in FIGS. 4C to 4G is replaced with the retroreflector 5.

[0141] In this way, it is possible to realize a space-floating image display device in which the optical system in the configuration of the space-floating image display device of Figures 4A to 4G is replaced with the optical system of Figure 2D. Even in these space-floating image display devices in which the optical system of Figure 2D is replaced, it is possible to realize a space-floating image display device that is almost as easy to use as the space-floating image display device of Figures 4A to 4G.

[0142] <Display Device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes the light source of the image display element 11. Fig. 5 shows the light source device 13 together with the liquid crystal display panel as an exploded perspective view.

[0143] As shown by arrow 30 in Figure 5, this liquid crystal display panel (image display element 11) receives an illumination light beam from light source device 13, which is a backlight device, that has narrow-angle diffusion characteristics, i.e., has strong directivity (linearity) and characteristics similar to laser light with a polarization plane aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam in accordance with an input video signal. The modulated image light is reflected by retroreflector 2 and passes through transparent member 100 to form a real image, a floating image in space (see Figure 1).

[0144] 5 also shows a configuration including a liquid crystal display panel 11 constituting the display device 1, a light redirection panel 54 for controlling the directional characteristics of the light beam emitted from the light source device 13, and, if necessary, a narrow-angle diffuser (not shown). Specifically, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarization is emitted with its intensity modulated by a video signal (see arrow 30 in FIG. 5). This allows the desired image to be projected as highly directional (linear) light of a specific polarization via the light redirection panel 54 toward the retroreflector 2, where it is reflected by the retroreflector 2 and transmitted to the eyes of a monitor outside the store (space), forming a floating image 3. A protective cover 50 (see FIGS. 6 and 7) may be provided on the surface of the light redirection panel 54.

[0145] <Display Device Example 1> FIG. 6 shows an example of a specific configuration of the display device 1. In FIG. 6, a liquid crystal display panel 11 and a light direction conversion panel 54 are disposed on the light source device 13 shown in FIG. 5. The light source device 13 is configured on a case shown in FIG. 5, formed of, for example, plastic, and contains LED elements 201 and a light guide 203 therein. As shown in FIG. 5 and other figures, the end surface of the light guide 203 has a lens-like shape whose cross-sectional area gradually increases toward the light receiving section in order to convert the divergent light from each LED element 201 into a substantially parallel beam. The lens-like shape gradually reduces the divergence angle by multiple total reflections during propagation within the light guide 203. The liquid crystal display panel 11 constituting the display device 1 is attached to the top surface of the display device 1. Furthermore, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202 on which their control circuits are mounted are attached to one side of the case of the light source device 13 (the left end surface in this example). A heat sink, which is a member for cooling the heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED board 202 .

[0146] Furthermore, a frame (not shown) for the liquid crystal display panel is attached to the top surface of the case of the light source device 13. The frame includes the liquid crystal display panel 11, which is attached to the frame, and a flexible printed circuit (FPC) (not shown) electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is an image display element, generates a display image by modulating the intensity of transmitted light in conjunction with the LED elements 201, which are solid-state light sources, based on a control signal from a control circuit (image control unit 1160 in FIG. 3 ) constituting the electronic device. The generated image light has a narrow diffusion angle and contains only specific polarization components, resulting in a novel image display device similar to a surface-emitting laser image source driven by a video signal. Currently, it is technically and safety-wise impossible to obtain a laser beam of the same size as the image obtained by the display device 1 described above using a laser device. Therefore, in this embodiment, light similar to the surface-emitting laser image light described above is obtained from a beam of light from a general light source, for example, an LED element.

[0147] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 7 as well as FIG.

[0148] 6 and 7 are cross-sectional views, and only one of the LED elements 201 constituting the light source is shown, and this light is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion on the end surface of the light guide and the LED element are attached while maintaining a predetermined positional relationship.

[0149] Each light guide 203 is formed of a translucent resin such as acrylic. The LED light-receiving surface at the end of the light guide 203 has a cone-shaped outer periphery obtained by rotating a parabolic cross section, and the top of the light guide 203 has a concave portion with a convex portion (i.e., a convex lens surface) formed in the center, and the center of the flat portion has a convex lens surface (or a concave lens surface) that protrudes outward (not shown). The outer shape of the light-receiving portion of the light guide to which the LED element 201 is attached is a parabolic shape that forms a cone-shaped outer periphery, and is set within an angle range that allows total reflection of the light emitted from the LED element toward the periphery within the parabolic shape, or a reflective surface is formed.

[0150] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed to the LED collimator (light-receiving end surface 203 a) so that the LED elements 201 on the surface are positioned in the center of the recessed portion described above.

[0151] With this configuration, the shape of the light-receiving end surface 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as approximately parallel light, thereby improving the utilization efficiency of the generated light.

[0152] As described above, the light source device 13 is configured by attaching a light source unit having a plurality of LED elements 201 arranged as light sources to the light-receiving end surface 203a, which is a light-receiving section provided on the end surface of the light guide 203, and the divergent light beams from the LED elements 201 are converted into approximately parallel light by the lens shape of the light-receiving end surface 203a of the light guide end surface, which is then guided inside the light guide 203 as shown by the arrow (in a direction parallel to the drawing), and emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11, which is disposed approximately parallel to the light guide 203 (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of the light beam direction conversion means 204 depending on the shape inside or on the surface of the light guide.

[0153] The light beam direction conversion means 204 described above emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 (in a direction perpendicular to the front of the drawing) that is disposed substantially parallel to the light guide 203, by changing the shape of the surface of the light guide or by providing a portion with a different refractive index inside the light guide. In this case, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is positioned at the same position as the diagonal dimension of the screen, if the relative brightness ratio between the center of the screen and the periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

[0154] 6 is a cross-sectional layout diagram illustrating the configuration and operation of the light source of this embodiment that performs polarization conversion in light source device 13 including the above-mentioned light guide 203 and LED element 201. In Fig. 6, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface thereof is attached liquid crystal display panel 11 that has polarizing plates on the light source light entrance surface and the image light exit surface.

[0155] In addition, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., P wave) 212 of the natural light beam 210 emitted from the LED element 201. The reflected light is reflected again by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and directed toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice to convert the reflected light beam from P polarized light to S polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam, the light intensity of which is modulated by a video signal in the liquid crystal display panel 11 (arrow 213 in Figure 6), enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0156] 7 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in a light source device 13 including a light guide 203 and an LED element 201, similar to Fig. 6. The light source device 13 is similarly configured with a light guide 203 formed of, for example, plastic, on the surface of which or inside which a light beam direction conversion means 204 is provided, an LED element 201 as a light source, a reflective sheet 205, a retardation plate 206, a lenticular lens, etc. A liquid crystal display panel 11 is attached to the top surface of the light source device 13 as an image display element, and the panel has polarizing plates on the light source light entrance surface and the image light exit surface.

[0157] A film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., S wave) 211 of the natural light beam 210 emitted from the LED element 201. That is, in the example of FIG. 7 , the selective reflection characteristics of the reflective polarizing plate 49 differ from those in FIG. 7 . The reflected light is reflected by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and returns to the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting the reflected light beam from S-polarized light to P-polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam intensity-modulated by the image signal in the liquid crystal display panel 11 (arrow 214 in FIG. 7 ) enters the retroreflector 2. After reflection by the retroreflector 2, a real image, a floating image in space, can be obtained.

[0158] In the light source devices shown in Figures 6 and 7, in addition to the function of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, the reflective polarizer reflects the polarized light component on one side. Therefore, the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizer multiplied by the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In fact, experiments have confirmed that the contrast performance of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those of self-luminous organic EL displays are obtained.

[0159] <Display Device Example 2> Figure 8 shows another example of the specific configuration of the display device 1. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, etc., in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. Also, an LED board on which LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and their control circuits are mounted are attached to one side of the case of the light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer surface of the LED board.

[0160] The liquid crystal display panel frame attached to the top surface of the case is configured to have attached thereto the liquid crystal display panel 11 attached to the frame, and further to have attached thereto an FPC (Flexible Printed Circuits) 403 electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements 14a and 14b, which are solid-state light sources, based on a control signal from a control circuit (not shown here) that constitutes the electronic device.

[0161] <Display Device Example 3> Next, another example of the specific configuration of the display device 1 (Display Device Example 3) will be described with reference to Fig. 9 . The light source device of this display device 1 converts a divergent beam of light (a mixture of P-polarized and S-polarized light) from an LED into a substantially parallel beam by a collimator 18, and reflects the parallel beam toward the liquid crystal display panel 11 by the reflective surface of a reflective light guide 304. The reflected light is incident on a reflective polarizer 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizer 49 transmits light of a specific polarization (e.g., P-polarized light) and causes the transmitted polarized light to be incident on the liquid crystal display panel 11. Here, light polarized other than the specific polarization (e.g., S-polarized light) is reflected by the reflective polarizer 49 and directed again toward the reflective light guide 304.

[0162] The reflective polarizing plate 49 is installed at an angle with respect to the liquid crystal display panel 11 so that the reflective polarizing plate 49 is not perpendicular to the chief ray of light from the reflective surface of the reflective light guide 304. The chief ray of light reflected by the reflective polarizing plate 49 is incident on the transmission surface of the reflective light guide 304. The light that has entered the transmission surface of the reflective light guide 304 passes through the back surface of the reflective light guide 304, passes through the λ / 4 plate 270 which is a retardation plate, and is reflected by the reflector 271. The light reflected by the reflector 271 passes through the λ / 4 plate 270 again, and passes through the transmission surface of the reflective light guide 304. The light that has passed through the transmission surface of the reflective light guide 304 is incident on the reflective polarizing plate 49 again.

[0163] At this time, the light that re-enters the reflective polarizing plate 49 has passed through the λ / 4 plate 270 twice, and therefore its polarization has been converted to a polarization (for example, P-polarized light) that is transmitted through the reflective polarizing plate 49. Therefore, the light whose polarization has been converted passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11. Note that, with regard to the polarization design related to the polarization conversion, the polarization may be configured in reverse from the above explanation (S-polarized light and P-polarized light may be reversed).

[0164] As a result, the light from the LED is aligned to a specific polarization (for example, P polarization), enters the liquid crystal display panel 11, and is brightness-modulated in accordance with the video signal to display an image on the panel surface. As in the above example, multiple LEDs that make up the light source are shown (however, because this is a vertical cross section, only one is shown in Figure 9), and these are attached at predetermined positions relative to the collimator 18.

[0165] Each collimator 18 is formed of a translucent resin such as acrylic or glass. The collimator 18 may have a cone-shaped outer circumferential surface obtained by rotating a parabolic cross section. The collimator 18 may have a concave portion with a convex portion (i.e., a convex lens surface) formed in the center of the apex (the side facing the LED substrate 102). The collimator 18 may have a convex lens surface (or a concave lens surface) protruding outward in the center of the flat surface (the side opposite the apex). The parabolic surface forming the cone-shaped outer circumferential surface of the collimator 18 is set within an angle range that allows total reflection of the light emitted from the LED toward the periphery, or a reflective surface is formed therein.

[0166] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED board 102. The LED board 102 is arranged and fixed to the collimator 18 so that the LEDs on the surface are positioned at the center of the apex of the convex cone shape (or in the recess if the apex has a recess).

[0167] With this configuration, the collimator 18 focuses light emitted from the LED, particularly light emitted from the central portion, into parallel light by the convex lens surface that forms the outer shape of the collimator 18. Light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the conical shape of the collimator 18, and is similarly focused into parallel light. In other words, the collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, makes it possible to extract almost all of the light generated by the LED as parallel light, thereby improving the utilization efficiency of the generated light.

[0168] Furthermore, the light converted into approximately parallel light by the collimator 18 shown in FIG. 9 is reflected by the reflective light guide 304. Of this light, light of a specific polarization passes through the reflective polarizer 49 due to the action of the reflective polarizer 49, while light of the other polarization reflected by the action of the reflective polarizer 49 passes through the light guide 304 again. The light is reflected by the reflector 271 located opposite the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarized and converted by passing twice through the λ / 4 plate 270, which is a retardation plate. The light reflected by the reflector 271 passes through the light guide 304 again and enters the reflective polarizer 49 provided on the opposite surface. Since the incident light has been polarized and converted, it passes through the reflective polarizer 49 and enters the liquid crystal display panel 11 with its polarization direction aligned. As a result, all of the light from the light source can be used, doubling the geometrical optical utilization efficiency of light. Furthermore, since the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, the use of the light source device of this embodiment significantly improves the contrast ratio of the entire display device. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 allows adjustment of the angle of light reflection and diffusion at each reflective surface. Adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 for each design can improve the uniformity of the light incident on the liquid crystal display panel 11.

[0169] It should be noted that the λ / 4 plate 270, which is the retardation plate in Fig. 9, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of Fig. 9, any retardation plate may be used as long as the phase changes by 90° (λ / 2) when polarized light passes through it twice. The thickness of the retardation plate may be adjusted according to the incident angle distribution of the polarized light.

[0170] <Display Device Example 4> Another example (Display Device Example 4) of the configuration of an optical system such as a light source device of a display device will be described with reference to Fig. 10. This is a configuration example in which a diffusion sheet is used instead of the reflective light guide 304 in the light source device of Display Device Example 3. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) that convert diffusion characteristics in the vertical and horizontal directions of the drawing (front-to-back directions not shown in the drawing) are used on the light emission side of collimator 18, and light from collimator 18 is made to enter between the two optical sheets (diffusion sheets).

[0171] The optical sheet may be a single sheet instead of a two-sheet configuration. In a single-sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes on the front and back surfaces of the single optical sheet. Alternatively, multiple diffusion sheets may be used to share the functions. In the example of FIG. 10 , the reflection and diffusion characteristics due to the front and back shapes of optical sheets 207A and 207B can be optimally designed using the number of LEDs, the divergence angle from LED substrate (optical element) 102, and the optical specifications of collimator 18 as design parameters so that the surface density of the light beam emitted from the liquid crystal display panel 11 is uniform. In other words, the diffusion characteristics are adjusted by the surface shapes of multiple diffusion sheets instead of light guides.

[0172] In the example of FIG. 10 , polarization conversion is performed in the same manner as in the display device example 3 described above. That is, in the example of FIG. 10 , the reflective polarizing plate 49 may be configured to have the property of reflecting S-polarized light (transmitting P-polarized light). In this case, the reflective polarizing plate 49 transmits P-polarized light from the light source LED, and the transmitted light enters the liquid crystal display panel 11. The reflective polarizing plate 49 reflects S-polarized light from the light source LED, and the reflected light passes through the retardation plate 270 shown in FIG. 10 . The light that passes through the retardation plate 270 is reflected by the reflector 271. The light reflected by the reflector 271 passes through the retardation plate 270 again and is converted to P-polarized light. The polarization-converted light passes through the reflective polarizing plate 49 and enters the liquid crystal display panel 11.

[0173] It should be noted that the λ / 4 plate 270, which is the retarder in FIG. 10, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of FIG. 10, any retarder that changes the phase by 90° (λ / 2) when polarized light passes through it twice may be used. The thickness of the retarder may be adjusted according to the distribution of incident angles of the polarized light. It should be noted that, in FIG. 10 as well, the polarization design for polarization conversion may be configured in reverse (reversing the S-polarized light and P-polarized light) from the above explanation.

[0174] In a typical TV device, the light emitted from the liquid crystal display panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (shown on the X-axis in FIG. 12(a)) and the vertical direction of the screen (shown on the Y-axis in FIG. 12(b)). In contrast, the diffusion characteristics of the light beam emitted from the liquid crystal display panel of this embodiment are, for example, as shown in Example 1 of FIG. 12, when the viewing angle at which the luminance is 50% of that at a front view (angle of 0 degrees) is set to 13 degrees, this is 1 / 5 of the 62 degrees of a typical TV device. Similarly, the vertical viewing angle is set asymmetrically between the top and bottom, and the reflection angle and the area of ​​the reflective surface of the reflective light guide are optimized to keep the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness being more than 50 times higher.

[0175] Furthermore, assuming the viewing angle characteristics shown in Example 2 of Figure 12, if the viewing angle at which brightness is 50% of that at a front view (angle of 0 degrees) is set to 5 degrees, this is 1 / 12 of the 62 degrees of devices used for general TV applications. Similarly, the vertical viewing angle is set equal at the top and bottom, and the reflection angle and area of ​​the reflective surface of the reflective light guide are optimized to keep the viewing angle to about 1 / 12 of that of devices used for general TV applications. As a result, the amount of image light directed in the monitoring direction is significantly improved compared to conventional LCD TVs, and brightness is more than 100 times higher.

[0176] As described above, by setting the viewing angle to a narrow angle, the amount of luminous flux directed in the monitoring direction can be concentrated, significantly improving the efficiency of light utilization. As a result, even when using a liquid crystal display panel for general TV applications, by controlling the light diffusion characteristics of the light source device, it is possible to achieve a significant improvement in brightness with similar power consumption, making it possible to create a video display device that is compatible with information display systems aimed at bright outdoor areas.

[0177] When using a large LCD panel, the brightness of the screen can be improved by directing the light from the periphery of the screen inward so that it is directed toward the observer when the observer is facing the center of the screen. Figure 11 shows the convergence angle between the long and short sides of the panel when the observer's distance from the panel, L, and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in portrait orientation, the convergence angle can be set to match the short side. For example, when using a 22-inch panel in portrait orientation and the monitoring distance is 0.8 m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the observer.

[0178] Similarly, when monitoring with a 15-inch panel in portrait orientation and a monitoring distance of 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the monitor. As described above, depending on the size of the liquid crystal display panel and whether it is used portrait or landscape, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the monitor who is in the optimum position to monitor the centre of the screen.

[0179] As shown in Figure 9, the basic configuration involves a light source device directing a light beam with a narrow angle of directionality to a liquid crystal display panel 11, which is then luminance-modulated according to a video signal. The video information displayed on the screen of the liquid crystal display panel 11 is then reflected by a retroreflector, and the resulting spatially floating image is displayed indoors or outdoors via a transparent member 100.

[0180] By using the display device and light source device according to the embodiment of the present invention described above, it is possible to realize a space floating image display device with higher light utilization efficiency.

[0181] <Example of Image Display Processing in Space-Floating Image Display Device> Next, an example of a problem solved by the image processing of this embodiment will be described with reference to Fig. 13A. In the space-floating image display device 1000, when the far side of the space-floating image 3 as seen from the user is inside the housing of the space-floating image display device 1000 and is sufficiently dark, the user will visually recognize that the background of the space-floating image 3 is black.

[0182] Here, an example of displaying a character "panda" 1525 in a floating-in-space image 3 will be described with reference to Fig. 13A. First, for an image including a pixel region for drawing the image of the character "panda" 1525 and a transparent information region 1520 that is a background image, as shown in Fig. 13A(1), the image control unit 1160 in Fig. 3 distinguishes and recognizes the pixel region for drawing the image of the character "panda" 1525 from the transparent information region 1520 that is a background image.

[0183] A method for distinguishing and recognizing the character image from the background image is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the character image layer in front of the background image layer can be processed as separate layers, and the character image and background image can be distinguished and recognized based on the superimposition relationship when these layers are combined.

[0184] Here, the image control unit 1160 recognizes black pixels that depict objects such as character images and transparent information pixels as different information. However, it is assumed that both the black pixels that depict objects and the transparent information pixels have a luminance of 0. In this case, when the Space Floating Image 3 is displayed, there is no difference in luminance between the pixels that depict black in the image of the character "panda" 1525 and the pixels of the transparent information region 1520, which is the background image. Therefore, in the Space Floating Image 3, as shown in FIG. 13A (2), neither the pixels that depict black in the image of the character "panda" 1525 nor the pixels of the transparent information region 1520 have luminance, and they are visually perceived by the user as the same optically black space. In other words, the black portions of the image of the character "panda" 1525, which is an object, blend into the background, and only the non-black portions of the character "panda" 1525 are perceived as floating in the display region of the Space Floating Image 3.

[0185] An example of image processing according to this embodiment will be described with reference to FIG. 13B. FIG. 13B is a diagram illustrating an example of image processing that more effectively resolves the issue of the black image region of the object blending into the background, as described in FIG. 13A. In FIGS. 13B(1) and 13B(2), the upper side shows the display state of the floating-in-space image 3, and the lower side shows the input / output characteristics of the image processing of the image of the object. Note that the image of the object (character "panda" 1525) and the corresponding data may be read from the storage unit 1170 or memory 1109 in FIG. 3, or may be input from the video signal input unit 1131, or may be acquired via the communication unit 1132.

[0186] In the state shown in Figure 13B(1), the input / output characteristics of the image processing of the object image are in a linear state with no particular adjustment. In this case, the display state is the same as in Figure 13A(2), and the black image area of ​​the object blends into the background. In contrast, in Figure 13B(2), the video control unit 1160 of this embodiment adjusts the input / output characteristics of the image processing of the image of the object (character "panda" 1525) to the input / output characteristics shown in the lower row.

[0187] That is, the video control unit 1160 performs image processing with input / output characteristics on the image of the object (character "panda" 1525), which has a characteristic of converting pixels in low-brightness areas of the input image into output pixels with increased brightness values. After the image of the object (character "panda" 1525) has been subjected to the image processing with input / output characteristics, a video including the image of the object (character "panda" 1525) is input to the display device 1 and displayed. Then, as shown in the upper part of FIG. 13B (2), the display state of the floating in space image 3 is such that the brightness of pixel areas depicting black in the image of the character "panda" 1525 increases. This allows the user to distinguish the areas depicting black among the areas depicting the image of the character "panda" 1525 without them blending into the black background, making it possible to display the object more appropriately.

[0188] 13B(2), the area displaying the image of the object character "panda" 1525 can be distinguished from the black background inside the housing of the space-floating image display device 1000 through the window, improving the visibility of the object. Therefore, for example, even if the object includes pixels with a brightness value of 0 among the pixels constituting the object before the image processing (i.e., at the time when the image of the object and the corresponding data are read from the storage unit 1170 or memory 1109 in FIG. 3, or at the time when the image of the object is input from the video signal input unit 1131, or at the time when the data of the object is acquired via the communication unit 1132, etc.), the image processing of the input / output characteristics by the video control unit 1160 converts the object into an object with increased brightness values ​​of pixels in low-brightness areas, and then the object is displayed on the display device 1 and converted into a space-floating image 3 by the optical system of the space-floating image display device 1000.

[0189] That is, the pixels that make up the object after image processing of the input / output characteristics are converted to a state in which they do not include pixels with a brightness value of 0, and then they are displayed on the display device 1 and converted into a floating image 3 by the optical system of the floating image display device 1000.

[0190] In the image processing of Figure 13B(2), a method for applying image processing with the input / output characteristics of Figure 13B(2) only to the image area of ​​the object (character "panda" 1525) is, for example, to configure the image processing of the video control unit 1160 so that the background image layer and the layer of the character image in front of the background image layer can be processed as separate layers, and the image processing with the input / output characteristics of Figure 13B(2) is applied to the character image layer while not applying this image processing to the background image layer.

[0191] Then, by combining these layers, image processing with a characteristic of raising the low-brightness areas of the input image is performed only on the character image, as shown in Fig. 13B (2). Alternatively, after the character image layer and background image layer are combined, image processing with the input / output characteristics of Fig. 13B (2) may be performed only on the character image area.

[0192] Furthermore, the input / output image characteristics used in the image processing for enhancing low-luminance regions of the input / output characteristics for the input image are not limited to the example shown in FIG. 13B(2). Any image processing for enhancing low luminance may be used, including so-called brightness adjustment. Alternatively, image processing for improving visibility by controlling the gain that changes the weighting of Retinex processing, as disclosed in International Publication WO 2014 / 162533, may be performed.

[0193] According to the image processing of FIG. 13B(2) described above, it is possible to make the user aware of areas where black is drawn among areas where images of characters, objects, etc. are drawn without blending into the black background, thereby realizing a more suitable display.

[0194] 13A and 13B, the problems and more suitable image processing were explained using a space-floating image display device in which the background appears black (for example, the space-floating image display device 1000 in FIGS. 4A to 4G, or the space-floating image display device 1000 in a state where the rear side window is shaded in FIGS. 4I and 4J). However, this image processing is also effective for devices other than these space-floating image display devices.

[0195] Specifically, in the space-floating image display device 1000 of Fig. 4H and the space-floating image display device 1000 in Fig. 4I and Fig. 4J where the rear window is not shaded, the background of the space-floating image 3 is not black, but the scenery behind the space-floating image display device 1000 through the window. In this case, the problems explained in Fig. 13A and Fig. 13B also exist.

[0196] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the scenery behind the space-floating image display device 1000 through the window. In this case, by using the image processing of Fig. 13B (2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognized as being distinct from the scenery behind the space-floating image display device 1000 through the window, improving the visibility of the object.

[0197] That is, by using the image processing of FIG. 13B(2), the area displaying the image of the character "panda" 1525, which is an object, can be recognized as distinct from the scenery behind the space floating image display device 1000 through the window, and it can be more easily recognized that the character "panda" 1525, which is the object, is in front of the scenery, improving the visibility of the object.

[0198] 4K, 4L, and 4M, as described above, when another image (such as an image from the transmissive self-luminous image display device 1650 or an image from the second display device 1680) is displayed at a position different in depth from the space-floating image 3, the background of the space-floating image 3 is not black, but the other image. In this case, the problems described in FIGS. 13A and 13B also exist.

[0199] That is, the part of the image of the character "panda" 1525, which is an object, that is drawn in black will blend into the other image that is displayed at a different depth from the floating image in space 3. In this case as well, by using the image processing of Fig. 13B(2), the part of the image of the character "panda" 1525, which is an object, that is drawn in black will be recognizable as being distinct from the other image, improving the visibility of the object.

[0200] In other words, by using the image processing of Figure 13B (2), the area displaying the image of the object character "panda" 1525 can be recognized as distinct from the other image, and it can be more easily recognized that the object character "panda" 1525 is in front of the other image, improving the visibility of the object.

[0201] An example of the image display process of this embodiment will be described with reference to Fig. 13C. Fig. 13C shows an example of the image display process of this embodiment, in which the space floating image 3 and a second image 2050, which is another image, are simultaneously displayed. The second image 2050 may correspond to the image displayed by the transmissive self-luminous image display device 1650 of Fig. 4K or 4L. The second image 2050 may also correspond to the image displayed by the second display device 1680 of Fig. 4M.

[0202] That is, the example of the image display in Fig. 13C shows a specific example of the image display examples of the space-floating image display device 1000 in Figs. 4K, 4L, and 4M. In the example of this figure, a bear character is displayed in space-floating image 3. Areas other than the bear character in space-floating image 3 are displayed in black, and become transparent as a space-floating image. In addition, the second image 2050 is a background image in which a plain, a mountain, and a sun are drawn.

[0203] 13C , the floating-in-space image 3 and the second image 2050 are displayed at different depth positions. When the user 230 views the two images, the floating-in-space image 3 and the second image 2050, in the line of sight of the arrow 2040, the user 230 can view the two images superimposed on each other. Specifically, the bear character of the floating-in-space image 3 appears superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0204] Here, since the space floating image 3 is formed as a real image in the air, when the user 230 moves their viewpoint slightly, they can recognize the depth of the space floating image 3 and the second image 2050 due to parallax. Therefore, the user 230 can get a stronger sense of floating in space from the space floating image 3 while viewing the two images in an overlapping state.

[0205] An example of the image display process of this embodiment will be described with reference to Fig. 13D. Fig. 13D(1) is a diagram of the floating in space image 3, from the example of image display of this embodiment in Fig. 13C, as seen from the line of sight of the user 230. Here, a bear character is displayed in the floating in space image 3. The area other than the bear character in the floating in space image 3 is displayed in black, and the floating in space image is transparent.

[0206] 13D(2) is a diagram showing the second image 2050 in the example of the video display of this embodiment shown in FIG. 13C as viewed from the line of sight of the user 230. In the example shown in this figure, the second image 2050 is a background image in which a plain, a mountain, and a sun are depicted.

[0207] 13D(3) is a diagram showing the state in which the second image 2050 and the floating in space image 3 appear superimposed in the line of sight of the user 230, among the example of image display of this embodiment in Fig. 13C. Specifically, the bear character of the floating in space image 3 appears superimposed in front of the background of plains, mountains, and the sun depicted in the second image 2050.

[0208] Here, when the space-floating image 3 and the second image 2050 are displayed simultaneously, it is desirable to pay attention to the balance of brightness between the two images in order to more favorably ensure the visibility of the space-floating image 3. If the second image 2050 is too bright compared to the brightness of the space-floating image 3, the displayed image of the space-floating image 3 will be transparent, and the second image 2050, which is the background, will be strongly visible through it.

[0209] Therefore, the output of the light source of the spatially floating image 3 and the display image brightness of the display device 1, and the output of the light source of the display device displaying the second image 2050 and the display image brightness of the display device should be set so that at least the brightness per unit area of ​​the spatially floating image 3 at the display position of the spatially floating image 3 is greater than the brightness per unit area of ​​the image light that reaches the display position of the spatially floating image 3 from the second image 2050.

[0210] Note that, since this condition only needs to be satisfied when the space-floating image 3 and the second image 2050 are displayed simultaneously, when switching from the first display mode in which the space-floating image 3 is not displayed and only the second image 2050 is displayed to the second display mode in which the space-floating image 3 and the second image 2050 are displayed simultaneously, control may be performed to reduce the brightness of the second image 2050 by lowering the output of the light source of the display device that displays the second image 2050 and / or the display image brightness of the display device. These controls may be realized by the control unit 1110 in Fig. 3 controlling the display device 1 and the display device that displays the second image 2050 (the transmissive self-luminous image display device 1650 in Fig. 4K or 4L or the second display device 1680 in Fig. 4M).

[0211] Note that when switching from the above-described first display mode to the above-described second display mode, if control is performed to reduce the brightness of second image 2050, the brightness may be reduced uniformly across the entire screen of second image 2050. Alternatively, instead of reducing the brightness uniformly across the entire screen of second image 2050, the brightness reduction effect may be greatest in the portion where an object is displayed in space-floating image 3, and the brightness reduction effect may be gradually weakened around that portion. In other words, the visibility of space-floating image 3 can be sufficiently ensured by reducing the brightness of second image 2050 only in the portion where space-floating image 3 is visually recognized as being superimposed on second image 2050.

[0212] Here, since the space floating image 3 and the second image 2050 are displayed at positions with different depths, when the user 230 slightly changes their viewpoint, the parallax causes a change in the superimposed position of the space floating image 3 relative to the second image 2050. Therefore, when switching from the above-mentioned first display mode to the above-mentioned second display mode, if the brightness is to be reduced unevenly across the entire screen of the second image 2050, it is not desirable to reduce the brightness sharply based on the outline of the object displayed in the space floating image 3, and it is desirable to perform gradation processing of the brightness reduction effect, which changes the brightness reduction effect in stages depending on the position as described above.

[0213] In addition, in the space floating image display device 1000 where the position of the object displayed in the space floating image 3 is approximately at the center of the space floating image 3, the position where the brightness reduction effect of the gradation processing of the brightness reduction effect is greatest can be set to the center position of the space floating image 3.

[0214] According to the image display process of this embodiment described above, the user 230 can view the space floating image 3 and the second image 2050 more favorably.

[0215] Note that when displaying the space-floating image 3, control may be performed so as not to display the second image 2050. Since not displaying the second image 2050 increases the visibility of the space-floating image 3, this is suitable for applications such as the space-floating image display device 1000 in which the user must be able to reliably view the space-floating image 3 when it is displayed.

[0216] <Example 2> As Example 2 of the present invention, an example of another configuration example of a space-floating image display device will be described. Note that the space-floating image display device according to this example is obtained by changing the optical system stored in the space-floating image display device described in Example 1 to the optical system shown in Figure 14 (1) or Figure 14 (2). In this example, differences from Example 1 will be described, and repeated explanations of the same configuration as Example 1 will be omitted. Note that in the following description of this example, the predetermined polarized light and the other polarized light are polarized waves whose phases differ by 90° from each other.

[0217] Fig. 14(1) shows an example of an optical system and optical path according to this embodiment. The optical system shown in Fig. 14(1) is configured such that the display device 1 is closer to the polarization separation member 101B in the optical system of Fig. 2C, making the entire optical system more compact. In Fig. 14(1), components that are assigned the same reference numerals as those in Fig. 2C will not be described in detail again.

[0218] 14(1), similar to Fig. 2C, image light of a predetermined polarized light (P-polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, similar to Fig. 2C, polarization separation member 101B selectively transmits the predetermined polarized light (P-polarized light in the figure) emitted from display device 1 and reflects the other polarized light (S-polarized light in the figure).

[0219] Therefore, image light of a predetermined polarization (P-polarized light in the figure) traveling in the vertical direction from the image display surface of the display device 1 passes through the polarization separation member 101B and reaches the retroreflector 2 to which the λ / 4 plate 21 is attached. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separation member 101B has passed through the λ / 4 plate 21 twice, and is converted from the predetermined polarization (P-polarized light in the figure) at the time of emission from the display device 1 to the other polarization (S-polarized light in the figure). The image light that travels again toward the polarization separation member 101B is the other polarization (S-polarized light in the figure), and is therefore reflected by the polarization separation member 101B toward the position where the user should be. The traveling direction of the image reflected by the polarization separation member 101B is determined based on the angle at which the polarization separation member 101B is disposed.

[0220] In the example of Figure 14 (1), the image light traveling toward the polarization separation member 101B is reflected at a right angle by the polarization separation member 101B and travels as shown in the figure. The image light reflected by the polarization separation member 101B forms a floating image 3A in space. The floating image 3A can be viewed by the user from the direction of arrow A.

[0221] Here, due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the position where the space floating image 3A is formed. This relationship determines the position where the space floating image 3A is formed in the traveling direction of the image light reflected by the polarization separating member 101B.

[0222] In the example of Figure 14(1), the display device 1, the polarization separating member 101B, and the retroreflector 2 are arranged closer than in the example of Figure 2C. This allows the entire optical system to be configured more compactly. However, the amount by which the space-floating image 3A protrudes from the optical system of Figure 14(1) is not very large. For example, as an indicator of the amount by which the space-floating image 3A protrudes from the optical system, the figure shows the distance from the position where the central light beam of the image light is reflected by the polarization separating member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)).

[0223] 14(1), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0224] Next, Fig. 14(2) shows another example of an optical system and an optical path according to this embodiment. The optical system of Fig. 14(2) is a modified version of the optical system of Fig. 14(1) in order to increase the amount of the floating image projecting from the optical system while realizing the same compactness as the optical system of Fig. 14(1). In Fig. 14(2), the components with the same reference numerals as Fig. 14(1) will not be described in detail again.

[0225] 14(2), as in FIG. 14(1), image light of a predetermined polarized light (P-polarized light in the figure) emitted from display device 1 travels in a direction perpendicular to the image display surface of display device 1. Here, the polarization characteristics of polarization separation member 101B are arranged 90 degrees differently from those in FIG. 14(1). Image light of a predetermined polarized light (P-polarized light in the figure) traveling in a direction perpendicular to the image display surface of display device 1 passes through polarization separation member 101B.

[0226] 14(1), the image light passing through the polarization separation member 101B is not faced with the retroreflector 2 having the λ / 4 plate 21 attached thereto, but with the specular reflector 4 having the λ / 4 plate 21B attached thereto. Here, the reflection at the specular reflector 4 is specular reflection (also called regular reflection), not retroreflection.

[0227] Therefore, the image light that has passed through polarization separation member 101B is specularly reflected by specular reflector 4 to which λ / 4 plate 21B is attached. The image light that has been specularly reflected by specular reflector 4 and travels again toward polarization separation member 101B has been converted from the predetermined polarization (P-polarized in the figure) at the time of emission from display device 1 to the other polarization (S-polarized in the figure) by having passed through λ / 4 plate 21 twice. The image light that has traveled again toward polarization separation member 101B is the other polarization (S-polarized in the figure), and is therefore reflected by polarization separation member 101B.

[0228] Here, because the orientation of the polarization separation member 101B in Figure 14(2) is different from that in Figure 14(1), the image light reflected by the polarization separation member 101B travels in the opposite direction from the position where the user should be. A retroreflector 2 with a λ / 4 plate 21C attached is disposed at the destination of the image light reflected by the polarization separation member 101B. The image light is retroreflected by the retroreflector 2. The image light that is retroreflected by the retroreflector 2 and travels again toward the polarization separation member 101B has been converted from the other polarized light (S-polarized light in the figure) to the specified polarized light (P-polarized light in the figure) by passing through the λ / 4 plate 21C twice.

[0229] The image light that travels back toward the polarization separation member 101B is of a predetermined polarization (P polarization in the figure), so it passes through the polarization separation member 101B and continues toward the location where the user should be. The image light that has passed through the polarization separation member 101B forms a floating image 3B in space. The floating image 3B can be easily viewed by the user from the direction of arrow A.

[0230] 14(2), as in the case of Fig. 14(1), due to the characteristics of retroreflection by the retroreflector 2, the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is equal to the optical path length of the image light emitted from the retroreflector 2 to reach the formation position of the space floating image 3B. This relationship determines the formation position of the space floating image 3B in the traveling direction of the image light transmitted through the polarization separation member 101B.

[0231] 14(2), the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 is longer than the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2 in FIG. 14(1). This is because in the optical system of FIG. 14(2), an optical path going back and forth between the polarization separation member 101B and the specular reflector 4, which does not exist in the optical system of FIG. 14(1), is added to the optical path length of the image light emitted from the display device 1 to reach the retroreflector 2.

[0232] As a result, in the optical system of Figure 14(2), the distance from the position where the central ray of the image light passes through the polarization separation member 101B to the position where the image light forms the space-floating image 3B (L2 in the example of Figure 14(2)) is significantly longer than the distance from the position where the central ray of the image light is reflected by the polarization separation member 101B to the position where the image light forms the space-floating image 3A (L1 in the example of Figure 14(1)) in the optical system of Figure 14(1).

[0233] 14(2), the characteristics of P-polarized light and S-polarized light may be interchanged. Specifically, the predetermined polarization of the image light emitted from the display device 1 may be S-polarized light, and the characteristics of P-polarized light and S-polarized light may be interchanged with respect to the reflection characteristics of the polarization separation member 101B. In this case, the P-polarized light and S-polarized light shown in the figure are both reversed, but the optical design, such as the optical path, can be realized in exactly the same way.

[0234] As described above, according to the optical systems of Fig. 14(1) and Fig. 14(2) in the second embodiment of the present invention, a more compact optical system can be realized. In particular, according to the optical system of Fig. 14(2), it is possible to increase the amount of the floating image projecting from the optical system, despite the more compact optical system.

[0235] In addition, when the optical system of Fig. 14(1) or Fig. 14(2) is incorporated into a space-floating image display device, it can be realized by replacing the optical system in the space-floating image display device described in Example 1 with the optical system of Fig. 14(1) or Fig. 14(2). Specifically, the optical system of Fig. 14(1) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4H, Fig. 4I, Fig. 4J, Fig. 4K, Fig. 4L, or Fig. 4M. In this case, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0236] Specifically, the optical system of Fig. 14(2) may be replaced with the optical system of the space-floating image display device of Fig. 4E, Fig. 4F, Fig. 4G, Fig. 4K, or Fig. 4L. In this case, it is possible to increase the amount of the space-floating image projecting from the optical system. Also, since the optical system becomes compact, it is possible to make the housing of the space-floating image display device of each figure smaller.

[0237] <Example 3> As Example 3 of the present invention, an example of the configuration of a space-floating image display device will be described. The space-floating image display device of Example 3 can be similarly applied to the configurations of Examples 1 and 2 as a basic configuration. The space-floating image display device of Example 3 can be applied based on, for example, FIG. 2A as the optical system configuration, and based on, for example, FIG. 4A as the housing, in other words, the main body configuration. Example 3 is a system (sometimes referred to as a space-floating image display system, etc.) having a space-floating image display device and external devices connected thereto. The external devices are, for example, mobile devices such as smartphones, tablets, and wearable devices (smartwatches, etc.) owned by the user. The mobile devices may also be referred to as mobile information processing terminal devices.

[0238] The space-floating image display device / system of Example 3 has the following basic functions. This space-floating image display system connects and cooperates with a user's mobile device via communication. This system has the function of displaying an image / video held by the user, such as an image stored in the mobile device and displayed on the screen of the mobile device (sometimes referred to as a target image, etc.), as a space-floating image 3 on the space-floating image display device. To do this, the user performs an operation / motion to bring the mobile device into contact with the space-floating image 3 (a corresponding predetermined imaging position, display range) on the space-floating image display device. This operation / motion is a predetermined operation / motion and may be referred to as an inserting operation, a touching operation, etc. The space-floating image display device determines and detects the touching operation using a sensor, etc. In this example, for example, the mobile device issues a display request based on a user's input operation, while the user performs a touching operation to insert or touch the mobile device into a position on the plane of the space-floating image 3.

[0239] Based on the contact operation, the space-floating image display device or the mobile terminal generates a display request / display instruction to display the target image on the mobile terminal side as the space-floating image 3. In other words, the space-floating image display device / mobile terminal generates such a display request / display instruction when it determines that a predetermined contact operation has been performed that satisfies predetermined conditions regarding contact between the mobile terminal and the space-floating image 3. The space-floating image display device, particularly the image control unit 1160 in FIG. 3, receives the display request and receives and acquires the target image data from the mobile terminal side via communication. Then, the space-floating image display device executes image processing of the display device 1 so as to display the target image as the space-floating image 3.

[0240] This function may be as follows: The user's mobile terminal transmits the display request together with the above-mentioned contact operation, in other words, immediately before, during, or after the contact. The space-floating image display device receives, allows / validates the display request from the mobile terminal based on the above-mentioned contact operation, and similarly executes the process of displaying the target image as space-floating image 3. The display request may be a set with the transmission of the target image data. This function differs in whether the display request is generated by the space-floating image display device or the mobile terminal, but in either case, the target image is automatically displayed as space-floating image 3 in response to a specific contact operation, which is the same.

[0241] This function may be as follows: Initially, the mobile device does not send a display request, and the user touches the mobile device to the space-floating image 3. When the space-floating image display device determines and detects a predetermined contact operation, it sends information such as confirmation to the mobile device via communication that it accepts the display request, in other words, that it is possible to display an image using this function. When the mobile device receives this information such as confirmation, it sends data of the target image to the space-floating image display device. The data of the target image may be accompanied by a display request. When the space-floating image display device receives the data of the target image, it displays the target image as the space-floating image 3.

[0242] [Space-Floating Image Display System] FIG. 15 shows the configuration of system 3000, which is a space-floating image display system according to a third embodiment. This system 3000 is a system in which a space-floating image display device 1000 and a mobile terminal 2000 of a user 230 are connected via a communication network such as a LAN 3020 and / or the Internet 3010. FIG. 15 schematically illustrates a part of the housing 1190 of the space-floating image display device 1000, i.e., the main body, the space-floating image 3, and the communication unit 1132 (FIG. 3). The housing 1190 is equipped with the components shown in FIG. 3 above. The control unit 1110 and communication unit 1132 shown in FIG. 3 may be installed inside or outside the housing 1190. The camera of the imaging unit 1180, the aerial operation detection sensor 1351, a microphone, a speaker, and the like may also be installed outside the housing 1190.

[0243] In this system 3000, communication is performed between the mobile terminal 2000 and the space-floating image display device 1000 via an arbitrary communication interface. The mobile terminal 2000 and the space-floating image display device 1000 may be directly connected to each other via short-range wireless communication, or the mobile terminal 2000 and the space-floating image display device 1000 may be connected to each other via the Internet 3010 or the like. In this system 3000, image data and the like are exchanged between the mobile terminal 2000 and the space-floating image display device 1000, so a certain level of communication speed is required. Therefore, in the example of FIG. 15, the mobile terminal 2000 and the space-floating image display device 1000 are connected via a LAN 3020. In other words, the mobile terminal 2000 and the space-floating image display device 1000 are connected to the same network via a router or the like, and are configured so that they can be identified from each other using IP addresses or the like. Communication is not limited to this, and mirroring or other methods of communication are also possible.

[0244] The mobile terminal 2000 has an application 2010 and an image 2020. The application 2010 is, for example, an arbitrary application program that displays an image 2020 or the like on the display screen of the mobile terminal 2000. The application 2010 may be a general OS or app, or may be a dedicated application for linking and communicating with the space-floating image display device 1000 of the system 3000 of Example 3. Data of the application 2010 may be distributed from the space-floating image display device 1000, or may be distributed from a server or the like on the Internet 3010. The image 2020 is image data that is a target image. The image 2020 may be stored in a server or the like on the Internet 3010, or may be distributed from the space-floating image display device 1000. The image 2020 may be a still image or a video. The image 2020 may be data in the form of a program or the like.

[0245] The application 2010 of the mobile terminal 2000 may create a display request 2030 for displaying the target image (image 2020) as the space-floating image 3, and send it to the space-floating image display device 1000.

[0246] 16 shows an overview of the functions of the space floating image display system of Example 3. (1) First, the user 230 has a target image 1601 on his / her mobile terminal 2000. The target image 1601 is an image to be displayed as the space floating image 3.

[0247] (2) The user 230 performs a contact operation 1600 by inserting the mobile terminal 2000 into the display range 3R of the space-floating image 3 of the space-floating image display device 1000 and bringing it into contact with the display range 3R. At this time, the mobile terminal 2000 may issue a display request 2030. The display request 2030 is a request or instruction to display the target image 1601 as the space-floating image 3.

[0248] (3) When the space-floating image display device 1000 detects the contact operation 1600, it displays the target image 1601 of the mobile terminal 2000 as the image 1602, which is the space-floating image 3. At this time, the space-floating image display device 1000 may receive a display request 2030 and acquire the target image 1601 of the mobile terminal 2000 based on the display request 2030. The space-floating image display device 1000 may send some kind of response to the display request 2030 to the mobile terminal 2000.

[0249] 17 shows an example of the configuration of a mobile terminal 2000. The mobile terminal 2000 includes, in a housing 2090, a control unit 20011, a display panel 20012, an external power input IF (interface) 20013, a power supply 20014, a secondary battery 20015, a storage unit 20016, a video control unit 20017, an attitude sensor 20018, an operation input unit 20019, a communication unit (including an antenna) 20020, an audio output unit (including a speaker) 20021, an audio input unit (including a microphone) 20022, a video signal input unit 20023, an audio signal input unit 20024, an imaging unit 20025, a memory 20026, and a non-volatile memory 20027. These components are connected by an architecture such as a bus.

[0250] The control unit 20011 has a processor and the like that controls the entire portable terminal 2000 and each unit. The control unit 20011 loads data such as programs stored in the non-volatile memory 20027 or the storage unit 20016 into the memory 20026 and executes processing in accordance with the programs. This enables various functions to be realized. The control unit 20011 may cooperate with the programs stored in the memory 20026 to perform arithmetic processing based on information acquired from each unit in the portable terminal 2000. The application 2010 in FIG. 15 is executed by, for example, the control unit 20011 or the video control unit 20017.

[0251] The memory 20026 stores video data to be displayed on the display panel 20012, control data for the mobile terminal 2000, etc. The control unit 20011 may read various software programs from the storage unit 20016, etc., and expand and store them in the memory 20026. The non-volatile memory 20027 and the storage unit 20016 hold various data and information used by the mobile terminal 2000. The data and information stored in the non-volatile memory 20027 and the storage unit 20016 include, for example, data for various operations to be displayed on the display panel 20012, display icons, data and layout information for objects to be operated by the user 230, etc.

[0252] The display panel 20012 is a display unit, such as a touch panel, and serves as both a display means and an input means for accepting touch operation input. Images and the like are displayed on the screen of the display panel 20012. The display panel 20012 is equipped with a touch sensor and accepts touch operation input by the finger of the user 230. The display panel 20012 can be a liquid crystal panel, an organic EL panel, or the like.

[0253] The communication unit 20020 is a device equipped with various communication interfaces. The communication unit 20020 is equipped with, as a communication interface / communication method, a mobile communication interface such as 4G or 5G, a wireless LAN communication interface such as Wi-Fi (registered trademark), or a short-range communication interface such as Bluetooth (registered trademark) or NFC. The communication unit 20020 of the mobile terminal 2000 can communicate with the communication unit 1132 ( FIG. 3 ) of the space-floating image display device 1000 using these communication methods. Similarly, from the perspective of the space-floating image display device 1000, the communication unit 1132 is equipped with these communication interfaces / communication methods. The communication unit 20020 can communicate with a communication device connected to the Internet 3010, such as a wireless base station, using any of these communication methods. This allows the mobile terminal 2000 to communicate with a server or the like connected to the Internet 3010.

[0254] The power supply 20014 converts AC current input from the outside via the external power supply input IF (interface) 20013 into DC current and supplies the DC current required by each unit of the mobile terminal 2000. The secondary battery 20015 stores the power supplied from the power supply 20014. When power is not supplied from the outside via the external power supply input IF 20013, the secondary battery 20015 supplies power to each unit requiring power.

[0255] The video signal input unit 20023 connects to an external video output device and inputs video data. The video signal input unit 20023 can use various digital video input interfaces. For example, a video input interface conforming to the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, a video input interface conforming to the DVI (Digital Visual Interface) standard, or a video input interface conforming to the DisplayPort standard can be used. Alternatively, an analog video input interface such as analog RGB or composite video may be provided. The video signal input unit 20023 may also be various USB interfaces.

[0256] The audio signal input unit 20024 is connected to an external audio output device and inputs audio data. The audio signal input unit 20024 can be an HDMI-standard audio input interface, an optical digital terminal interface, a coaxial digital terminal interface, or the like. The audio signal input unit 20024 may also be any of various USB interfaces. In the case of an HDMI-standard interface, the video signal input unit 20023 and the audio signal input unit 20024 may be configured as an interface in which a terminal and a cable are integrated.

[0257] The audio output unit 20021 can output audio based on audio data input to the audio signal input unit 20024 or audio data stored in the storage unit 20016. The audio output unit 20021 may be configured with a speaker or may have an earphone jack. The audio output unit 20021 may also output built-in operation sounds and error warning sounds. The audio output unit 20021 may be configured to output digital signals to an external device, such as the Audio Return Channel function defined in the HDMI standard.

[0258] The audio input unit 20022, which includes a microphone, collects sounds around the mobile terminal 2000, converts the collected sounds into signals, and generates audio signals. The audio input unit 20022 may generate audio signals by recording the voice of a person, such as the voice of the user 230, with the microphone, and the control unit 20011 or the like may perform voice recognition processing on the audio signals to acquire text information.

[0259] The imaging unit 20025 is, for example, a camera having an image sensor. The mobile terminal 2000 may have a camera (in-camera) on the front side of the housing 2090, which is the side of the display panel 20012, or a camera (out-camera) on the opposite back side. In this embodiment, the imaging unit 20025 has both an in-camera and an out-camera.

[0260] The storage unit 20016 is a storage device that records various data such as video data, image data, and audio data, as well as various information such as programs. The storage unit 20016 may be configured, for example, as a magnetic recording medium recording device such as a hard disk drive (HDD) or a semiconductor device memory such as a solid-state drive (SSD). For example, the storage unit 20016 may be pre-recorded with various data such as video data, image data, and audio data, as well as various information such as programs, at the time of product shipment. The storage unit 20016 may also record various data and information acquired from an external server or the like via the communication unit 20020. The image data and the like recorded in the storage unit 20016 are output and displayed on the display panel 20012 through processing by the video control unit 20017. The mobile terminal 2000 may output and transmit the various data and information recorded in the storage unit 20016 to an external server or the like via the communication unit 20020.

[0261] The video control unit 20017 performs various controls related to the video signal input to the display panel 20012. The video control unit 20017 may also be referred to as a video processing circuit, a video processing unit, an image processing unit, etc. The video control unit 20017 may be configured with hardware such as an ASIC, an FPGA, or a video processor. The video control unit 20017 performs video switching control, such as determining which video signal to input to the display panel 20011 for display, among the video signals to be stored in the memory 20026, etc., and the video signals (video data) input to the video signal input unit 20023, etc. The video control unit 20017 may also perform control to perform image processing on the video signals input from the video signal input unit 20023, the video signals to be stored in the memory 20026, etc. Examples of image processing include scaling processing for enlarging, reducing, or transforming an image, brightness adjustment processing for changing brightness, contrast adjustment processing for changing the contrast curve of an image, Retinex processing for decomposing an image into light components and changing the weighting of each component, etc. The video control unit 20017 may include a video memory or the like for storing video data to be input to the display panel 20012.

[0262] The attitude sensor 20018 is a sensor configured by combining a gyro sensor, a gravity sensor, an acceleration sensor, a geomagnetic sensor, etc., and can detect the attitude of the mobile terminal 2000. The attitude can be expressed, for example, by the directions and angles of three orthogonal axes (X, Y, Z) in space. The control unit 20011 may control the operation of each unit based on the attitude detection result of the attitude sensor 20018. In addition, the mobile terminal 2000 may also include a GPS receiver, a proximity sensor, an illuminance sensor, a distance measurement sensor, etc.

[0263] The operation input unit 20019 is a device for the user 230 to input operations, and includes, for example, a power button and a volume button.

[0264] [Image Display Method] Figure 18 (Figure 18A, etc.) is an explanatory diagram of an image display method by cooperation between the space-floating image display device 1000 and the mobile terminal 2000 in the system 3000 of Example 3. Figure 18A, etc. shows a state in which a user 230 makes a display request 2030 from the mobile terminal 2000 to the space-floating image display device 1000 by touching the mobile terminal 2000 to the display range 3R of the space-floating image 3 of the space-floating image display device 1000. Figure 18A is a perspective view, Figure 18B is a top view (XY plan view) seen from above, Figure 18C is a side view (YZ plan view) seen from the side, and Figure 18D is a front view (XZ plan view) seen from the front of the device.

[0265] In FIG. 18A , the top surface of the housing 1190 has an opening 1200, which is provided with a transparent member 100 (e.g., a glass plate) and a polarization separation member 101, as in FIG. 2A . The display range 3R represents a predetermined position or plane in space where the real image, the floating-in-space image 3, is formed. In FIG. 18A , the spatial coordinate system is represented by (X, Y, Z). The X-axis / X direction is the left-right direction (first horizontal direction) as seen from the user 230 (a viewpoint not shown at the origin of the direction of arrow A). The Y-axis / Y direction is the front-to-back direction (depth direction, second horizontal direction) as seen from the user 230. The Z-axis / Z direction is the up-down direction (vertical direction) as seen from the user 230. The coordinate system for the floating-in-space image 3 is represented by (x, y, z). The x-direction is the horizontal direction of the screen, the y-direction is the vertical direction of the screen, and the z-direction is the depth direction.

[0266] Direction 1801 shows an example of the direction (insertion direction) in which user 230 inserts mobile terminal 2000 into display range 3R of floating-in-space image 3 when performing a contact operation, and in this example, it is the Y direction. In Fig. 18A, an example of the location where mobile terminal 2000 and display range 3R come into contact is shown as contact point 1802. In this example, contact point 1802 is near the bottom right of display range 3R.

[0267] In the embodiment of FIG. 18 , when the mobile terminal 2000 comes into contact with the display range 3R, a display request 2030 is made from the mobile terminal 2000 to the space-floating image display device 1000. In other words, the display request 2030 is transmitted from the mobile terminal 2000 to the space-floating image display device 1000 immediately before, during, or immediately after the contact. For example, the space-floating image display device 1000 determines and detects this contact, and upon detecting this contact, receives the display request 2030 transmitted from the mobile terminal 2000. That is, the space-floating image display device 1000 allows the target image to be displayed on the space-floating image 3 based on the display request 2030. The space-floating image display device 1000 receives image data of the target image accompanying the display request 2030 from the mobile terminal 2000 by transfer. The space-floating image display device 1000 controls the display device 1 based on the acquired image data to display the target image as the space-floating image 3. In other words, the space floating image display device 1000 converts the target image designated by the user 230 into a space floating image and displays it as the space floating image 3.

[0268] The display request 2030 sent from the mobile terminal 2000 can be sent by the various communication methods described above, and the details are not limited thereto. In this example, the display request 2030 is sent by wireless communication in the LAN 3020, but this is not limiting, and a wired connection may be used, or the mobile terminal 2000 and the space floating image display device 1000 may be connected by infrared communication or the like.

[0269] The determination and detection of contact between the portable terminal 2000 and the display range 3R will be described later, but examples of the determination and detection include the aerial operation detection sensor 1351 (FIG. 3) and the imaging unit 1180 (camera). In addition, the determination and detection of contact may be performed using various sensors attached inside and outside the housing 1190, or various sensors included in the portable terminal 2000, such as a gyro sensor, or a combination of these.

[0270] As described above, in this system 3000, when the mobile terminal 2000 determines and detects a contact operation on the display range 3R, the space-floating image display device 1000 recognizes the display request 2030 from the mobile terminal 2000, and the space-floating image display device 1000 starts displaying the target image of the space-floating image. As described above, a feature of the third embodiment is that the target image, which is a space-floating image, can be displayed simply by contacting and placing the mobile terminal 2000 on the display range 3R of the space-floating image 3.

[0271] [Touch Operation] Fig. 19 (Fig. 19A, etc.) shows the orientation change direction when the mobile terminal 2000 is brought into contact with the display range 3R of the floating-in-space image 3. As shown in the figure, the orientation change direction 1901 of the mobile terminal 2000 can be in various directions. Fig. 19A is a perspective view showing the rotation angles of three axes: pitch angle α, roll angle β, and yaw angle γ. In this embodiment, as a rule, the orientation of the mobile terminal 2000 can be arbitrary during the touch operation; that is, the state of these angles (α, β, γ) can be any state. This simplifies the touch operation by the user 230, and provides high convenience.

[0272] FIG. 19B is an XY plane view showing an example in which the orientation of the mobile terminal 2000 is rotated within a horizontal plane (XY plane) during a contact operation. For the sake of explanation, the representative position and orientation of the mobile terminal 2000 are, as shown in FIG. 19A , the front surface of the housing, particularly the center position P1 of the display panel 20012, and the orientation R1 (the same as the axis of the roll angle β) along the long side of the front surface. In this example, the orientation R1 of the mobile terminal 2000 at the time of contact and the corresponding insertion direction 1801 are rotated around the Z axis (yaw angle γ) and tilted diagonally left with respect to the Y axis. The orientation change direction 1901 is the direction of rotation around the Z axis (yaw angle γ). The mobile terminal 2000 is in contact with the display range 3R near the top edge of the housing (contact point 1802). For example, the user 230 holds the mobile terminal 2000 in his right hand and inserts it near the center of the display range 3R.

[0273] 19C is a YZ plane view, showing an example in which the orientation of the portable terminal 2000 is rotated in the YZ plane during a contact operation. In this example, the orientation R1 of the portable terminal 2000 at the time of contact and the corresponding insertion direction 1801 are rotated around the X axis (pitch angle α) and tilted diagonally downward with respect to the Y axis. The orientation change direction 1901 is the direction of rotation around the X axis (pitch angle α). In this example, the insertion direction 1801 is perpendicular to the display range 3R. The vicinity of the top edge of the housing of the portable terminal 2000 is in contact with the display range 3R (contact point 1802).

[0274] 19D is an XZ plane view showing an example in which the orientation of the mobile terminal 2000 is rotated within the XZ plane during a contact operation. In this example, the orientation R1 of the mobile terminal 2000 during contact and the corresponding insertion direction 1801 are rotated around the Y axis (roll angle β), and the flat plate of the housing is oriented vertically. The orientation change direction 1901 is the direction of rotation around the Y axis (roll angle β).

[0275] FIG. 19E is a YZ plane cross-sectional view showing another example of the attitude of the mobile terminal 2000 during a touch operation. This example shows a case where the screen of the display panel 20012 on the front of the mobile terminal 2000 faces downward during a touch operation. FIG. 19E shows an example in which the configuration of FIG. 4A is applied to the configuration of the space-floating image display device 1000. Image light is emitted diagonally upward from the opening 1200 of the transparent member 100 on the top surface (XY plane) of the housing 1190, forming the space-floating image 3. The plane (xy plane) of the display range 3R of the space-floating image 3 is tilted approximately 45 degrees from the XY plane. In FIG. 19E, the mid-air operation detection sensor 1351 and the like are omitted (described below).

[0276] When performing a contact operation, user 230 holds portable terminal 2000 with screen 1902 facing downward (direction 1903) and inserts it toward the back (positive) in the Y direction as insertion direction 1801 relative to display range 3R. Fig. 19E shows a state (portable terminal 2000b) in which portable terminal 2000 has contacted display range 3R and passed through to the back. In this state, screen 1902 of portable terminal 2000b faces downward (direction 1903). Such a contact operation may be performed.

[0277] 19F is a YZ plane cross-sectional view, showing another example of the attitude of the mobile terminal 2000 during a contact operation. This example shows a case where, during a contact operation, the screen 1902 of the display panel 20012 on the front of the mobile terminal 2000 is oriented along the plane (xy plane) of the display range 3R of the floating-in-space image 3. The user 230 operates the screen 1902 of the mobile terminal 2000 so that it faces the display range 3R and overlaps with it. In FIG. 19F, the screen 1902 of the mobile terminal 2000 remains in contact with the display range 3R.

[0278] The predetermined contact operation may be determined by determining the time (contact time) that the mobile terminal 2000 is in contact with the display range 3R. That is, the space floating image display device 1000 counts the contact time, and determines that the predetermined contact operation has been performed when the contact time reaches or exceeds a certain time as a determination condition.

[0279] 20 ( FIG. 20A , etc.) are explanatory diagrams showing that when a contact operation is performed to insert the portable terminal 2000 into the display range 3R, the position of the portable terminal 2000 can basically be any position at each point, including before, during, and after insertion. Insertion of the portable terminal 2000 into the display range 3R is sufficient as long as the portable terminal 2000 and the display range 3R come into contact; in other words, as long as contact detection is possible, any position, insertion direction, or posture angle can be used. Furthermore, contact between the display range 3R and the portable terminal 2000 can basically be made at any point within the display range 3R.

[0280] In this way, broadening the range of permissible touch operations and actions (corresponding determination conditions) has the advantage of facilitating user operation. Conversely, it is also possible to narrow the range of permissible touch operations and actions (determination conditions). In other words, the position, posture, insertion direction, contact time, contact area of ​​the display range 3R, etc. of the portable terminal 2000 that are permissible during a touch operation (before, during, or after insertion) may be specified to be limited.

[0281] Fig. 20A shows examples of the position and insertion direction of the portable terminal 2000 before insertion on the YZ plane. In Fig. 20A, the portable terminal 2000 is not shown, and the position of the portable terminal 2000 is indicated by a dot. Black circles and white circles (p1, etc.) indicate examples of the position of the portable terminal 2000 before insertion. As shown in the figure, various positions and insertion directions are possible as long as the portable terminal 2000 contacts the display range 3R.

[0282] FIG. 20B shows examples of the position and insertion direction of the portable terminal 2000 before insertion on the XY plane. Any insertion direction is possible, allowing the portable terminal 2000 to touch the display area 3R from any position before insertion (e.g., p11). Furthermore, in the case of position p14, before insertion, the portable terminal 2000 is located to the right of the display area 3R and further back in the Y direction than the display area 3R, and the insertion direction is from the back (+Y) to the front (-Y) or left to right relative to the display area 3R. Even in this case, if the portable terminal 2000 touches the display area 3R, it is permitted as a touch operation.

[0283] 20C is a YZ plan view showing an example of a state when the mobile terminal 2000 touches the display range 3R. The touch of the mobile terminal 2000 with the display range 3R may be a touch of any part of the mobile terminal 2000, as long as the touch can be detected and determined. Furthermore, once the mobile terminal 2000 has touched the display range 3R once (after a touch operation has been detected), the mobile terminal 2000 may move out of the display range 3R.

[0284] State 2000c1 shows an example of the state of the portable terminal 2000 at the time of contact, where the contact point (contact position) c1 at that time is near the center of the display range 3R and is, for example, a part near the top edge of the housing on the portable terminal 2000 side. State 2000c2 shows another example of the state of the portable terminal 2000 at the time of contact, where the contact point c2 at that time is at the top of the display range 3R and is, for example, a part near the bottom edge of the housing. State 2000c3 shows another example of the state of the portable terminal 2000 at the time of contact, where the contact point c3 at that time is at the bottom of the display range 3R, and after contact, the portable terminal 2000 has passed to the back and is not touching the display range 3R. State 2000c4 shows a state in which the portable terminal 2000 has been moved forward in the Y direction from the contact state 2000c1.

[0285] [Modification: Judgment Conditions] In the third embodiment, the posture and position of the portable terminal 2000 when inserting or touching the display range 3R are arbitrary, but the present invention is not limited to this. In a modification, the posture and position of the portable terminal 2000 when inserting or touching the display range 3R may be limited to a specific state. That is, the space-floating image display device 1000 may judge and detect a specific posture or other state when the portable terminal 2000 is touching the display range 3R, and only when this is detected, allow the display request 2030. As a modification, when narrowly limiting the allowable predetermined touch operations, the following judgment conditions may be cited, for example.

[0286] (1) Only insertion directions are permitted in the Y direction toward the front, that is, toward the back from the side where the user 230 views the Floating in Space image 3, relative to the plane of the display range 3R of the Floating in Space image 3. In other words, contact with the display range 3R from the back or side is not permitted. Furthermore, only insertion directions within a predetermined angle range are permitted relative to the plane of the display range 3R of the Floating in Space image 3.

[0287] (2) When the mobile terminal 2000 touches the plane of the display range 3R, the contact location, i.e., the position of the mobile terminal 2000, or the area within the display range 3R, is limited to a part. For example, a part of the center, a part of the bottom right, or a part of the top left within the display range 3R.

[0288] (3) The orientation of the mobile terminal 2000 when it comes into contact with the plane of the display range 3R is limited to certain orientations. For example, the angle with respect to the plane of the display range 3R must be within a predetermined angle range. For example, as shown in FIG. 19C , only a case where the flat housing of the mobile terminal 2000 is inserted roughly perpendicular to the plane of the display range 3R may be permitted. Furthermore, for example, as shown in FIG. 19F , only a case where the flat housing of the mobile terminal 2000 comes into contact with the plane of the display range 3R roughly parallel to the plane of the display range 3R may be permitted.

[0289] (4) The attitude and holding time of the mobile terminal 2000 after the mobile terminal 2000 has come into contact with the plane of the display range 3R and then passed through to the rear side are limited. For example, as shown in the example of FIG. 19E , the attitude angle with respect to the plane of the display range 3R or the opening 1200 is within a predetermined angle range.

[0290] 21A shows an example in which, when a touch operation is performed to insert the mobile terminal 2000 into the display range 3R of the space-floating image 3, the space-floating image display system displays guide information for the user 230 regarding the insertion / touch operation. In Example 3, the space-floating image display device 1000 displays the guide information, in other words, information for guiding and guiding the location and direction to insert the mobile terminal 2000, and user interface information, on the space-floating image 3. This allows the user 230 to easily perform the touch operation to insert the mobile terminal 2000 into the display range 3R according to the guide information.

[0291] Examples of guidance displays in this guide information include the following: (a) Displaying the entire display range 3R (the area that accepts touch operations; a partial area is also acceptable) in a predetermined color, etc. Alternatively, displaying a frame representing the entire display range 3R in a color, etc. (b) Explaining the insertion location and insertion direction by displaying arrows, etc. (c) Explaining the insertion location and insertion direction by using natural language, video, images, animation, etc.

[0292] In the example of FIG. 21A , an example of guide information display is shown at the bottom in an xy plane view when the floating-in-space image 3 is viewed from the front. The guide information in FIG. 21A illustrates several guidance display examples, and at least one of these may be applied. Guidance display example 2101 is an example in which a rectangular frame representing the entire display range 3R is displayed in color. Guidance display example 2102 is an example in which an area accepting insertion or contact is indicated to the user by displaying an arrow image (or other predetermined symbol or graphic). The arrow image in this example is an image of multiple arrows pointing inward from the four sides of the display range 3R. Guidance display example 2103 is an example in which a text image in natural language, for example, "Insert here," is displayed to inform the user of insertion or contact into the area of ​​the display range 3R. Guidance display example 2104 is an example in which a video / image, for example, animation, is used to inform the user of insertion or contact into the area of ​​the display range 3R.

[0293] As a modification, as will be described later, guide information may be similarly displayed on the screen of the mobile terminal 2000.

[0294] As a modified example, a guide image for informing the contact status may be displayed on the floating-in-space image 3 when performing an inserting / touching operation on the mobile terminal 2000. In the example of Fig. 21B, only a partial area 2105 of the display range 3R of the floating-in-space image 3 is defined as a predetermined area (insertion area / contact area) that accepts the inserting / touching operation.

[0295] The space-floating image display device 1000 uses a sensor to detect the contact status of the portable terminal 2000 with respect to the display range 3R, and displays a guide image 2106 representing the contact status according to the detected contact location. This guide image 2106 is an image indicating that the portable terminal 2000 is contacting the display range 3R at a spatial position corresponding to the guide image 2106. In this example, the guide image 2106 is an effect image resembling ripples spreading from the contact location. By looking at the guide image 2106, the user 230 can more easily recognize the contact status of the portable terminal 2000 with respect to the display range 3R. Then, the user 230 simply moves the portable terminal 2000 to the insertion area, which is a partial area 2105, based on the guide image 2106. The guide information / feedback information for the user 230 is not limited to the image display as described above, and may also be combined with audio output or the like.

[0296] [Display Restriction Function] The present system 3000 may also have the following function (referred to as a display restriction function). When the mobile terminal 2000 is inserted into the display range 3R during a contact operation, the present system 3000 may hide the space-floating image 3 in a portion of the display range 3R near the row where the mobile terminal 2000 is inserted, and then display something else in the remaining area of ​​the display range 3R. When the user 230 looks at the display range 3R after inserting the mobile terminal 2000, there may be areas where the space-floating image 3, which is the displayed image, is blocked or obscured by the mobile terminal 2000. This function controls the display of the space-floating image 3 in such an invisible area (partial area) so that it is temporarily hidden. In this function, the image control unit 1160 of the space-floating image display device 1000, particularly the image control unit 1160 of FIG. 3, may realize the above-mentioned hiding by controlling the light source device 13 or the image display element (liquid crystal display panel 11) of the display device 1.

[0297] Furthermore, when the above-mentioned contact operation causes a portion of the display range 3R to be hidden, the space-floating image display device 1000 may reduce the display size of the displayed image or change the aspect ratio so that the remaining display area matches the space-floating image 3, which is the displayed image.

[0298] FIG. 22 (including FIG. 22A) shows an example of this display restriction function. FIG. 22A is an XY plan view, similar to FIGS. 18A to 18D, showing a case in which the user 230 inserts and touches the portable terminal 2000 in the Y direction against a portion of the display range 3R. Area 2201 indicates the position and area of ​​the portable terminal 2000 when it is in contact with the display range 3R after insertion. FIG. 22B shows an XZ plan view corresponding to the state of FIG. 22A. Area 2202 is an example of an invisible area caused by the blocking of the image light. Correspondingly, FIG. 22C is a YZ plan view showing the state of the blocking of the image light. Position 2203 indicates the position and area of ​​the portable terminal 2000 when it is in contact with the display range 3R after insertion.

[0299] Image light component 2204 indicates a component of the luminous flux of image light emitted to the outside from opening 1200 that is not blocked by mobile terminal 2000 (position 2203) and enters the viewpoint (eye) of user 230, thereby forming space floating image 3 in display range 3R. Image light component 2205 indicates a component of the luminous flux of image light emitted to the outside from opening 1200 that is blocked by mobile terminal 2000 (position 2203) and does not enter the viewpoint (eye) of user 230, thereby forming invisible region 2202 in display range 3R.

[0300] FIG. 23 (FIG. 23A, etc.) shows an example of the display of a guide image in the space floating image 3, as well as an example of the operation of the display restriction function.

[0301] 23A shows an example in which a guide image 2302 indicating that a display request 2030 is occurring is displayed when a contact operation is detected in a front view (xy plane) of the display range 3R of the floating-in-space image 3. In this example, a frame display 2301 showing the entire display range 3R is provided, and a guide image 2302 indicating that a display request 2030 is occurring is displayed within the frame display 2301. The guide image 2302 is, for example, a text image indicating that a display request 2030 is occurring, such as "Requesting..." or an arrow icon. In this state, an invisible area 2202 is created due to occlusion by the portable terminal 2000.

[0302] 23B shows an example in which the space floating image display device 1000 hides a part of the display range 3R and controls the display of the remaining area based on the detection of the occurrence of the invisible area 2202 as shown in FIG. 23A. The occurrence of the invisible area 2202 can be detected by the aerial operation detection sensor 1351 or the imaging unit 1180.

[0303] 3 of the space-floating image display device 1000, for example, determines and detects the contact operation and also determines and detects the invisible area 2202, and determines a non-display area within the display range 3R that includes the invisible area 2202. In this example, based on detection by the aerial operation detection sensor 1351 (described later), the contact point 2303 in the X and x directions, in other words, the width and column, of the display range 3R, is determined. Therefore, the space-floating image display device 1000 determines the display column corresponding to the contact point 2303 within the display range 3R as the non-display area 2304 shown with a diagonal hatching pattern. The non-display area 2304 includes the invisible area 2202. The area of ​​the display range 3R other than the non-display area 2304 becomes the display area 2305. In other words, the space floating image display device 1000 configures the display area 2305 by reducing the size of the display range 3R in the X direction and the x direction by the width of the non-display area 2304.

[0304] Space-floating image display device 1000 controls the display of display device 1 so that space-floating image 3 is not displayed in non-display area 2304, and space-floating image 3 is displayed only in display area 2305. In this example, in display area 2305 of display range 3R, the original frame image 2301 and guide image 2302 as shown in FIG. 23A are changed to frame image 2306 and guide image 2307 by being adjusted, for example reduced, so as to fit within the width of display area 2305 while maintaining the aspect ratio.

[0305] The above function makes it easier for the user 230 to view the guide image 2307 and the like in the floating image 3. In the above example, a larger area is set as the non-display area 2304 compared to the invisible area 2202, but the present invention is not limited to this. The non-display area may be the same area as the invisible area, or may be a different area that overlaps at least partially with the invisible area. Furthermore, the guide image can be changed or adjusted in various ways, such as changing the aspect ratio, changing the color or brightness, or shifting the display position.

[0306] 24A shows an example of the display of a guide image of the floating-in-space image 3 as feedback such as a notification to the user 230 when a predetermined contact operation is detected, in other words, when the contact operation is determined to be successful and a display request 2030 is received. In this example, a guide image 2401 indicating success is displayed over the entire display range 3R. The guide image 2401 is made up of letters, icons, etc. indicating success. When the operation is successful in this way, the process moves to a phase in which the target image is displayed as the floating-in-space image 3.

[0307] 24B shows an example of the display of a guide image of the floating image 3 in space as feedback such as a notification to the user 230 when a predetermined touch operation cannot be detected, in other words, when the touch operation is determined to have failed and the display request 2030 is not received. In this example, a guide image 2402 indicating failure is displayed over the entire display range 3R. The guide image 2402 is made up of characters, icons, etc. indicating failure or retry.

[0308] If the user 230 does not properly insert or touch the mobile terminal 2000 into the display range 3R, or if the operation is properly performed but the system 3000 cannot properly detect it due to a sensor malfunction or the like, the space-floating image display device 1000 will determine that it is a failure or an error, as described above, and output a guide image 2402 as feedback to the user 230. In this case, the space-floating image display device 1000 may prompt the user 230 to retry the touch operation. The user 230 will start over from the touch operation.

[0309] 24C shows an example in which, when a guide image indicating that the display request 2030 from the mobile terminal 2000 to the space-floating image display device 1000 has been properly made (successful) together with a touch operation of the mobile terminal 2000 to the display range 3R is displayed as the space-floating image 3, the display area is adjusted according to the invisible area 2202. The non-display area 2404 is an area of ​​the display range 3R that includes the invisible area 2202, and the display area 2405 is an area other than the non-display area 2404.

[0310] (A) on the left side of the lower part of Fig. 24C is an example of a successful case, in which guide video 2401b whose size etc. has been adjusted based on guide video 2401 as in Fig. 24A is displayed in display area 2405. Similarly, (B) on the right side of the lower part of Fig. 24C is an example of an unsuccessful case, in which guide video 2402b whose size etc. has been adjusted based on guide video 2402 as in Fig. 24B is displayed in display area 2405.

[0311] The feedback to the user 230 is not limited to visual means such as displaying a guide image on the floating image 3 or lighting a lamp, but may also be auditory means such as audio output, or tactile means such as a device that generates stimuli such as vibration, air, or ultrasonic waves. Such devices, such as an air jet or ultrasonic generator, may be installed inside or outside the housing 1190. Also, a combination of these means may be used simultaneously.

[0312] FIG. 24D shows a case where the floating-in-space image display device 1000 generates a sound 2408 indicating success or failure from the audio output unit 1140 in FIG. 3 , such as a speaker, to the user 230 when the touch operation and display request 2030 results in a success or failure on the YZ plane. (A) shows the sound 2408 (an example of audio notification) in the case of success, such as "Connected." (B) shows the sound 2408 (an example of audio notification) in the case of failure, such as "Connection failed" or "Please try again." The speaker may be installed inside or outside the housing 1190. The speaker may be a superdirectional speaker or the like. While the above example is an example of natural language audio output, the output may also be a predetermined short sound, such as a beep, an alert sound, or music.

[0313] 25 (FIG. 25A, etc.) show implementation examples of an aerial operation detection sensor 1351 and an imaging unit 1180 (camera) as sensors in the space-floating image display device 1000 of Example 3. The aerial operation detection unit 1350, the control unit 1110, or the image control unit 1160 in FIG. 3 of the space-floating image display device 1000 of Example 3, in particular, uses these sensors to determine and detect insertion and contact operations of the mobile terminal 2000 into the display range 3R of the space-floating image 3.

[0314] FIG. 25A is a YZ plan view showing an example of the arrangement of the aerial operation detection sensor 1351 and the imaging unit 1180 (camera) in the housing 1190. The configuration example of the optical system in FIG. 25A is the same as that in FIGS. 4A and 18E. In the example of FIG. 25A, the aerial operation detection sensor 1351 is installed near the transparent member 100 on the top surface of the housing 1190, on the front side closer to the user 230 in the Y direction. The optical axis of this aerial operation detection sensor 1351, indicated by the dashed-dotted arrow, faces diagonally upward, i.e., in the y direction, as shown, and is set to overlap with the display range 3R of the floating-in-space image 3. In addition, in the example of FIG. 25A, the imaging unit 1180 (camera) is installed within the housing 1190 with the optical axis indicated by the dashed-dotted arrow facing upward, i.e., in the Z direction. The imaging range of the imaging unit 1180 (camera) is set to include the display range 3R and the opening 1200, as indicated by the dashed lines.

[0315] FIG. 25B shows an example configuration of the aerial operation detection sensor 1351. FIG. 25B shows an xy plane view aligned with the display range 3R of the floating-in-space image 3. The aerial operation detection sensor 1351 has multiple optical elements 1351c arranged in the x direction (the X direction in FIG. 25A). The optical elements 1351c are pairs of a light-emitting element 1351a and a light-receiving element 1351b. The light-emitting element 1351a is configured, for example, as an infrared element. The emission surface of the light-emitting element 1351a of the aerial operation detection sensor 1351 coincides with the top surface of the transparent member 100 of the housing 1190 in FIG. 25A. The light-emitting element 1351a emits light a1, for example, infrared light, in the y direction. If the light a1 is not blocked by an object, it passes through the display range 3R. If the light a1 is blocked by an object, it is reflected by the object and returns as reflected light a2. The reflected light a2 is received by the light receiving element 1351b.

[0316] For example, if a contact point 2501 made by the user's 230 finger is located within the xy plane of the display range 3R, light a1 is reflected from the contact point 2501 and returns as reflected light a2. A light receiving element 1351b at a certain x-direction position detects the reflected light a2. This allows the aerial operation detection unit 1350 to determine that the contact point 2501 is located at that x-direction position. The aerial operation detection unit 1350 can also calculate the distance using the TOF method from the time it takes for the emitted light a1 to return as reflected light a2. For example, it can calculate a distance 2502 to the contact point 2501. This also allows the position coordinates of the contact point 2501 on the xy plane of the display range 3R to be determined.

[0317] 25B shows the case where the mobile terminal 2000b is placed within the display range 3R by an insertion / touch operation as a contact point 2503. In this case, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 can similarly detect the position coordinates of the contact point 2503. Note that, although there is nothing above the floating-in-space image 3 in the example of FIG. 25A, a part of the housing 1190 may also be provided above the floating-in-space image 3. In that case, light a1 from the aerial operation detection sensor 1351 is reflected by that part of the housing 1190 and returns as reflected light a2.

[0318] By using the aerial operation detection sensor 1351 and the aerial operation detection unit 1350, the floating-in-space image display device 1000 can detect a contact operation when the mobile terminal 2000b contacts the display range 3R as shown in Fig. 25A. Without being limited to the above arrangement example, the aerial operation detection sensor 1351 may be arranged above the xy plane of the display range 3R, or may be arranged at a position shifted forward or backward, i.e., in the z direction or Y direction, or multiple aerial operation detection sensors may be arranged at multiple positions in the forward or backward direction.

[0319] 25A, when the mobile terminal 2000b is placed in the display range 3R, the mobile terminal 2000b appears in the captured image of the imaging unit 1180 (camera). Therefore, the image control unit 1160 and the like linked to the imaging unit 1180 can detect the mobile terminal 2000b from the captured image and determine whether the mobile terminal 2000b has touched the display range 3R. As a result, the space-floating image display device 1000 can detect a contact operation when the mobile terminal 2000b touches the display range 3R as shown in FIG. 25A by using the imaging unit 1180. The space-floating image display device 1000 can determine and detect a contact operation using at least one of the aerial operation detection sensor 1351 and the imaging unit 1180. Using both of them can increase the accuracy of the determination and detection.

[0320] 25A , other possible arrangements for the imaging unit 1180 (camera) include, for example, an arrangement in which the imaging unit 1180b is positioned outside the housing 1190, at the rear in the Y direction, with the optical axis facing the display range 3R, as shown by the dashed line as imaging unit 1180b. The imaging unit 1180b may capture an image of the face of the user 230, or may detect that the user 230 has approached. Furthermore, instead of being limited to the aerial operation detection sensor 1351 that senses parallel to the plane of the floating-in-space image 3 as shown in FIG. 25B , a distance measurement sensor or stereo camera that senses the plane from a direction perpendicular to the plane of the floating-in-space image 3 may also be used.

[0321] FIG. 25C shows another example of the arrangement of the aerial operation detection sensor 1351 and the imaging unit 1180 (camera) in the housing 1190. The example in FIG. 25C shows a case where the optical system is based on FIG. 2B , with the housing 1190 oriented vertically and the floating-in-space image 3 formed to stand vertically (Z direction). The components of the optical system described above are mounted on the rear of the housing 1190 in the Y direction, and the floating-in-space image 3 is formed between the upper and lower housing sections at the front of the housing 1190. In this example, the aerial operation detection sensor 1351 and the imaging unit 1180 (camera) are installed in the upper housing section. The aerial operation detection sensor 1351 faces downward so that its optical axis overlaps with the display range 3R. The imaging range of the imaging unit 1180 (camera) is set to include the display range 3R. Similarly, with this configuration, when the mobile terminal 2000 is inserted into the display range 3R, a contact operation can be detected using at least one of the aerial operation detection sensor 1351 and the imaging unit 1180 (camera).

[0322] [Regarding Distinguishing Between Fingers and Mobile Terminals] The space-floating image display device 1000 may or may not distinguish between the fingers of the user 230 and the mobile terminal 2000 as objects that come into contact with the space-floating image 3 (display range 3R). When distinguishing between them, the following technical means can be cited, for example.

[0323] (1) The space-floating image display device 1000 may recognize the mobile terminal 2000 from communication information such as a connection request when performing any communication between the mobile terminal 2000 and the space-floating image display device 1000. When the mid-air operation detection sensor 1351 detects that some object is in contact with the display range 3R, if there is the above communication information, the space-floating image display device 1000 may infer that the object is the mobile terminal 2000.

[0324] (2) The space floating image display device 1000 may recognize and determine that the object in contact with the space floating image 3 is the mobile terminal 2000 by image recognition processing or the like based on the image captured by the imaging unit 1180 (camera).

[0325] (3) The floating-in-space image display device 1000 may use the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 to determine whether the object in contact with the floating-in-space image 3 is the user's 230 finger or the mobile terminal 2000. For example, as a normal aerial operation / touch operation, the device is configured to accept operations using a single finger (e.g., detection of contact point 2501 in FIG. 25B ). When the mobile terminal 2000 is inserted into the display range 3R, the area of ​​the contact point is larger for the mobile terminal 2000 than for a single finger, and the shape of the contact point is linear. The floating-in-space image display device 1000 detects and determines the area and shape of the contact point, for example, contact point 2503 in FIG. 25B , using the aerial operation detection sensor 1351, and determines whether the object in contact is a finger or the mobile terminal 2000 based on the results.

[0326] (4) The information exchanged between the space-floating image display device 1000 and the mobile terminal 2000 includes information representing the mobile terminal 2000, such as a QR code, which will be described later. The space-floating image display device 1000 recognizes and acquires this information, and thereby distinguishes whether the object touching the space-floating image 3 is the user's finger or the mobile terminal 2000.

[0327] 26A shows an example of a display on the screen of the display unit 20012 by the application 2010 etc. of the mobile terminal 2000 in Example 3. The mobile terminal 2000 displays an application screen 2601 on the screen of the display unit 20012 by processing of the OS, application 2010 etc. based on an input operation of the user 230. In this example, the application 2010 is a dedicated app that cooperates with the space-floating image display device 1000 to allow the user 230 to conveniently use various functions of the space-floating image display device 1000.

[0328] The application screen 2601 in FIG. 26A has a GUI that prompts the user 230 to specify an image 2020 (target image) to be displayed on the space-floating image 3 of the space-floating image display device 1000. For example, a message such as "Please specify the image to be displayed on the aerial display" is displayed. The user 230 selects an image 2020 (FIG. 15) to be displayed on the aerial display from images in the mobile terminal 2000. The selected image is displayed as a preview as a display target image 2602. After checking the display target image 2602, the user 230 presses a display request button 2603. As a result, the mobile terminal 2000 transmits a display request 2030 to the space-floating image display device 1000 via short-range wireless communication or the like, with the display target image 2602 as the target image.

[0329] 26B is a modified example of the display of the guide image in FIG. 21A described above, and is an example in which a guide image is displayed on the screen of the mobile terminal 2000. An application screen 2601 in FIG. 26B displays a guide image 2604 for prompting the user 230 to perform a contact operation. The guide image 2604 displays, for example, a message such as "Please contact the smartphone with the floating image," and an image showing the operation of inserting and contacting the mobile terminal 2000 into the display range 3R.

[0330] Fig. 26C is a modified example of the guide image display in Fig. 23A and the guide image display in Fig. 24A described above, and is an example of displaying a guide image on the screen of the mobile terminal 2000. On the application screen 2601 in Fig. 26C, depending on the situation, a guide image 2605 is displayed to inform the user 230 that the display request 2030 is in progress, or a guide image 2606 is displayed to inform the user 230 that the display request 2030 has been received successfully.

[0331] In Example 3, as one method, during a contact operation, as described above, user 230 specifies target image 2020 on the screen of application 2010 of portable terminal 2000, presses display request button 2603, and portable terminal 2000 transmits display request 2030. Thereafter, user 230 inserts portable terminal 2000 into display range 3R.

[0332] Not being limited to this, in other embodiments, the operation of the user 230 may be kept to a minimum. For example, the user 230 displays a desired target image on the screen of the mobile terminal 2000, and in that state inserts the mobile terminal 2000 into the display range 3R. The space-floating image display device 1000 determines and detects the contact operation, and if successful, generates a display request 2030. The space-floating image display device 1000 communicates with the mobile terminal 2000, obtains a target image associated with the display request 2030, and displays it on the space-floating image 3. The target image is the image displayed on the screen of the mobile terminal 2000 at that time.

[0333] [Display of target image in space-floating image] The image control unit 1160 (FIG. 3) of the space-floating image display device 1000 generates image data to be displayed on the screen of the display device 1 (liquid crystal display panel 11) based on the image data of the target image. For example, if the target image is two-dimensional image data, this image data is two-dimensional image data generated by processing the two-dimensional image data. Also, if the target image is three-dimensional image data (such as a three-dimensional model), this image data is two-dimensional image data generated from the three-dimensional image data by rendering in a virtual three-dimensional space.

[0334] 16 , for example, a target image 1601 (an example of a character image) that a user 230 has displayed and designated on a mobile terminal 2000 can be displayed as image 1602 as a floating-in-space image 3 in response to a touch operation and a display request 2030. This allows the user 230 to enjoy viewing the desired target image 1601 as image 1602 of the floating-in-space image 3.

[0335] There is no particular limitation on the method for canceling the display of the target image in the space floating image 3 after that. For example, the target image may be automatically canceled after a certain time has passed since the display of the target image began. Also, a display cancel button or the like may be provided on the space floating image 3 or the screen of the mobile terminal 2000. A display cancel button or the like may also be provided on the housing 1190.

[0336] As described above, according to the space floating image display system of Example 3, by linking with the mobile terminal 2000, it is possible to provide the user 230 with a service that allows the user 230 to view a desired image by displaying it as a space floating image 3.

[0337] In addition, the data and information exchanged between the mobile terminal 2000 and the space floating image display device 1000 may not be limited to the image to be displayed, and some management and control data and information, such as user information, may also be exchanged along with the image to be displayed.

[0338] As a variant, it is also possible to display a background image, character image, etc. in advance as the floating image 3 in space, and then, from that state, superimpose the target image on the background image, etc., by the user 230 touching the mobile terminal 2000 and making a display request 2030.

[0339] In the third embodiment, when the mobile device 2000 comes into contact with the floating image in space 3, it is deemed a success, the display request 2030 is received, and the target image is displayed, but this is not limiting. In a modified example, it may be deemed a success when the user 230 inserts the mobile device 2000 into the floating image in space 3 and makes contact with it, then the mobile device 2000 passes through the floating image in space 3 and enters the space at the back, and is no longer in contact with the display range 3R (for example, FIG. 19E).

[0340] In addition, the following modification is also possible. The function in this modification is a function to acquire the image displayed as the space-floating image 3 on the mobile terminal 2000 side. The space-floating image display device 1000 displays a character image or the like on the space-floating image 3 in advance (for example, similar to (3) in FIG. 16). When the user 230 wants to acquire the image displayed as the space-floating image 3 on the mobile terminal 2000 side, the user performs a touch operation with the mobile terminal 2000 on the space-floating image 3 in the same manner as described above. If this touch operation is successful, the space-floating image display device 1000 transmits the image displayed as the space-floating image 3 (the corresponding image data) to the mobile terminal 2000. As a result, the user 230 can acquire and display the image on the mobile terminal 2000 and enjoy looking at it (for example, similar to (1) in FIG. 16).

[0341] In the third embodiment, after the target image is displayed by touching the mobile terminal 2000 to the space-floating image 3, the target image continues to be displayed even if the mobile terminal 2000 is no longer in contact with the space-floating image 3, but this is not limiting. In a modified example, the target image may be displayed only while the mobile terminal 2000 continues to be in contact with the space-floating image 3. When the space-floating image display device 1000 detects that the mobile terminal 2000 has moved away from the display range 3R, it cancels the display of the target image on the space-floating image 3.

[0342] <Example 4> Next, an example using a QR code (two-dimensional code) or the like will be described as another example (Example 4) with reference to Fig. 27A and subsequent figures. Example 4 is a modification of Example 3. The mobile terminal 2000 has a function of appropriately displaying a QR code on the screen of the display unit 20012. This QR code may be, for example, a code representing the display request 2030 described above.

[0343] In Example 4, when the mobile terminal 2000 sends a display request 2030 to the space-floating image display device 1000, the following operations and actions may be performed in order to make the mobile terminal 2000 recognize the space-floating image display device 1000, or to make the space-floating image display device 1000 recognize the mobile terminal 2000. The user 230 performs an operation from an application 2010 or the like on the mobile terminal 2000 to the space-floating image display device 1000 to perform a linkage or connection, and to perform a display request 2030 (similar to, for example, FIG. 26A). Then, the user 230 performs a contact operation to insert the mobile terminal 2000 into the display range 3R. Alternatively, for example, the user 230 displays a QR code or the like on the screen of the mobile terminal 2000 before insertion, and then performs a contact operation to insert the mobile terminal 2000 into the display range 3R in a position that allows the QR code or the like of the mobile terminal 2000 to be read by the imaging unit 1180 (camera) on the space floating image display device 1000 side.

[0344] FIG. 27A shows an example in which the mobile terminal 2000 displays a QR code screen 2701 on the screen 1902 of the display unit 20012 based on an input operation by the user 230, and displays image information such as a QR code 2702 on the QR code screen 2701.

[0345] FIG. 27B shows an example in a YZ plane view in which user 230 displays a QR code on screen 1902 of portable terminal 2000a (FIG. 27A) at a position before insertion into display range 3R. Then, portable terminal 2000a is inserted into display range 3R, resulting in portable terminal 2000b at a contact position. In this state, portable terminal 2000b has its screen 1902 facing downward and is within the imaging range of imaging unit 1180 (camera). Therefore, space-floating image display device 1000 can detect portable terminal 2000b from the captured image and the QR code 2702 displayed on screen 1902. Space-floating image display device 1000 recognizes the QR code and extracts the data and information described in the QR code. For example, space-floating image display device 1000 obtains a display request 2030 from the QR code.

[0346] The space-floating image display device 1000 detects the contact of the mobile terminal 2000b with the display range 3R by the aerial operation detection sensor 1351 or the imaging unit 1180, and acquires the display request 2030 from the QR code of the mobile terminal 2000b. Based on the success of these operations, the space-floating image display device 1000 displays the target image associated with the display request 2030 on the space-floating image 3.

[0347] 27B, the portable terminal 2000b is in contact with the display range 3R, but this is not limiting. As described above, the portable terminal 2000 may come into contact with the display range 3R and then move further back than the display range 3R to enter a non-contact state. The orientation of the portable terminal 2000 may be such that the QR code on the screen 1902 can be detected by the imaging unit 1180 (camera). For example, in the case of the imaging unit 1180b in FIG. 25A, the orientation of the portable terminal 2000 may be such that the screen 1902 faces the back in the Y direction (for example, it may be the same as in FIG. 19E).

[0348] In Example 4, as described above, a condition for recognizing a QR code is added to the judgment condition for a predetermined contact operation in Example 3. In other words, in Example 4, when the AND condition that a predetermined contact operation is detected and a QR code is recognized is satisfied, the space-floating image display device 1000 receives and permits the display request 2030 and displays the target image on the space-floating image 3.

[0349] In Example 4 ( FIG. 27B ), user 230 may insert portable terminal 2000 into display range 3R in a horizontal direction parallel to the top surface of housing 1190 with screen 1902 facing downward (similar to FIG. 19E ). In a modified example, user 230 may insert portable terminal 2000 into display range 3R with screen 1902 facing downward, and then place screen 1902 in contact with the top surface of transparent member 100 of housing 1190. In these cases, the QR code can still be recognized.

[0350] [QR Code] The data and information described in the code such as the QR code / barcode is not limited to the example of the display request 2030, but may include the following, for example: These pieces of data and information may be used in combination.

[0351] (1) Connection information: Information for communication connection between the mobile terminal 2000 and the space-floating image display device 1000 in the communication network of Fig. 15. For example, information such as an IP address, ID, and password. The communication network here may be the Internet 3010 or LAN 3020, or may be direct communication between the mobile terminal 2000 and the space-floating image display device 1000.

[0352] (2) Terminal ID / User ID: Information such as the ID of the mobile terminal 2000. Alternatively, information such as the ID of the user 230.

[0353] (3) Image ID: Information such as the ID or URL of the target image that the user 230 wants to display.

[0354] When the connection information is a QR code, the mobile terminal 2000 transmits the connection information to the space-floating image display device 1000 by presenting the QR code, and a wireless connection is established between the communication unit 20020 of the mobile terminal 2000 and the communication unit 1132 of the space-floating image display device 1000 using the connection information.

[0355] When the terminal ID / user ID etc. are a QR code, the space-floating image display device 1000 can recognize a specific mobile terminal 2000 / user 230. The space-floating image display device 1000 may be configured to allow only a specific mobile terminal 2000 / user 230 associated with a specific terminal ID / user ID to be provided with the service of this function.

[0356] When the image ID is a QR code, the target image is not limited to an image stored in the mobile terminal 2000 and can be used.

[0357] As described above, according to the space-floating image display system of Example 4, by using the QR code, it is possible to define a predetermined contact operation between the mobile terminal 2000 and the space-floating image display device 1000, and various controls can be performed using the QR code.

[0358] Incidentally, even in the case of a form such as Example 3 that does not use a QR code, the above-mentioned data and information may be exchanged between the mobile terminal 2000 and the space floating image display device 1000 together with the display request 2030 and used.

[0359] As a modification of the fourth embodiment, the space floating image display device 1000 may transmit the information of the QR code to a nearby mobile terminal 2000, and the mobile terminal 2000 that receives the information may display the QR code.

[0360] In the third and fourth embodiments, the data of the target image to be displayed on the space-floating image 3 is stored in advance in the mobile terminal 2000, but this is not limiting. In a modified example, the data of the target image may be stored on the space-floating image display device 1000 side, and the user 230 or the mobile terminal 2000 side may select and specify the target image and issue the display request 2030. In another modified example, the data of the target image may be stored on a server or the like on a communication network, and the user 230 or the mobile terminal 2000 side may select and specify the target image and issue the display request 2030.

[0361] Furthermore, as a modified example of Example 4, the floating-in-space image display device 1000 may display the information of the QR code on the floating-in-space image 3, and the user 230 may recognize the information of the QR code displayed as the floating-in-space image 3 using the camera of the mobile terminal 2000.

[0362] Fig. 28 (Fig. 28A, etc.) shows this modified example. Fig. 28A is an example in which the space-floating image display device 1000 displays a QR code 2801, which is a space-floating image 3. Fig. 28B is a YZ plan view, and is an example in which the mobile terminal 2000 captures and reads the QR code 2801 of the space-floating image 3 in Fig. 28A using the camera of the imaging unit 20025, for example, the outer camera. The mobile terminal 2000 obtains predetermined information from the recognized QR code 2801. After recognizing the QR code 2801, the user 230 touches the mobile terminal 2000 to the display range 3R.

[0363] 28C is a further modified example, which shows an example in which a QR code 2802 is displayed on the screen of the second display device 1680 of the space-floating image display device 1000. The space-floating image display device 1000 uses the second display device 1680 arranged so as to overlap the back side of the space-floating image 3, similar to Fig. 4M, for example, and displays the QR code 2802 on the screen of the second display device 1680. Furthermore, the space-floating image display device 1000 may use a transmissive self-luminous image display device 1650, similar to Fig. 4L, and display a QR code on the screen of the transmissive self-luminous image display device 1650.

[0364] User 230 captures an image of QR code 2802 displayed on the screen of second display device 1680 using the camera of imaging unit 20025 of portable terminal 2000, and portable terminal 2000 recognizes the QR code to obtain information. After recognizing QR code 2802, user 230 brings portable terminal 2000 into contact with display range 3R.

[0365] <Example 5> As Example 5 of the present invention, an example of the configuration of a space-floating image display device will be described. The space-floating image display device of Example 5 can be similarly configured to Examples 1 to 4 as a basic configuration. Furthermore, Example 5, like Examples 3 and 4, can be configured as a system (space-floating image display system) having a space-floating image display device and external devices connected thereto. The external devices include, for example, mobile devices (mobile information processing terminal devices) such as smartphones, tablets, and wearable devices (smartwatches, etc.) owned by the user.

[0366] [Problems and Solutions Related to Example 5] The problem is that the space-floating image display device 1000 is equipped with various sensors such as the aerial operation detection sensor 1351 and the attitude sensor 1113 ( FIG. 3 ), but the mobile terminal 2000 may also be equipped with similar sensors (touch sensors and attitude sensors). As in Examples 3 and 4, when the mobile terminal 2000 is communicatively connected to the space-floating image display device 1000, various possibilities are considered regarding which device's operation and detection (sensor) to enable or use. If it is unclear which side's operation and detection to enable, this may result in undesirable behavior or effects depending on the application, function, situation, etc. Therefore, it is necessary to clarify which side's operation and detection to enable depending on the application, function, situation, etc.

[0367] Therefore, the space-floating image display system and display method of Example 5 have the following functions, etc. The system of Example 5 targets both a sensor for detecting user operations, etc. on the mobile terminal 2000 side and a sensor for detecting user operations, etc. on the space-floating image display device 1000 side. This system handles a sensor for detecting touch operations and rotation operations on an image displayed on the screen of the mobile terminal 2000 side, and a sensor for detecting touch operations and rotation operations on an image displayed on the space-floating image 3 on the space-floating image display device 1000 side. This system selects, sets, and controls which side's sensor to enable / disable when an image / video is displayed on both the screen of the mobile terminal 2000 side and the space-floating image 3 side (dual display, described below).

[0368] [Example 1 of Problem and Solution] Fig. 37 shows Example 1 of Problem and Solution. First, suppose that an image 3701 (for example, image B) is displayed on the screen 2005 of the mobile terminal 2000, and based on that image B, a similar image 3702 (for example, image B) is also displayed on the space-floating image 3, which is the screen of the space-floating image display device 1000 (dual display function 2910 in Fig. 29). One of the problems is that in this situation, when the user 230 performs a touch operation on, for example, image B on the screen 2005 of the mobile terminal 2000, it is conventionally unclear what will happen to the two images 3701 and 3702.

[0369] In this embodiment, one solution is as follows: In this system, in response to detection of, for example, a touch operation on image 3701 (image B) on screen 2005 of mobile terminal 2000, a display update or the like is performed as a predetermined process associated with image 3701, and synchronization is performed so that the display update or the like is reflected in image 3702 (image B) on the side of the floating in space image 3 (synchronization control function 2920 in FIG. 29 ). Image B on the side of mobile terminal 2000 changes from image B to image Bb as a display update, and image B on the side of the floating in space image 3 changes from image B to image Bb as a display update.

[0370] Conversely, as shown in the lower part of Figure 37, when a touch operation is performed on image 3702 (image B) on the spatial floating image 3 side, the system does not perform a display update on image 3702, nor does it reflect this on image 3701 (image B) on the mobile terminal 2000 side; in other words, no synchronization is performed on the mobile terminal 2000 side.

[0371] The synchronization control in this example is performed based on a setting that enables detection of touch operations on the mobile terminal 2000 side and disables detection of touch operations on the floating image 3 side (operation detection control function 2930 in FIG. 29). When two devices are connected for communication, this system performs settings related to the state after such connection. Each device controls operation, detection, and display according to the settings after connection. This achieves the effect shown in FIG. 37.

[0372] [Example 2 of Problem and Solution] Fig. 38 shows Example 2 of Problem and Solution. In the example of Fig. 38, an image 3802 (image A) is displayed on the space floating image 3, which is the screen of the space floating image display device 1000, and based on that image A, a similar image 3801 (image A) is also displayed on the screen of the mobile terminal 2000 (dual display function 2910 in Fig. 29). One of the problems is that in this state, when the user 230 performs a touch operation on, for example, image A of the space floating image 3, it is conventionally unclear what will happen to the two images 3801 and 3802.

[0373] In this embodiment, one solution is as follows: In this system, in response to detection of, for example, a touch operation on image 3802 (image A) of the floating in space image 3, a display update or the like is performed as a predetermined process associated with the image 3802, and synchronization is performed so that the display update is also reflected in image 3801 (image A) on the mobile terminal 2000 side (synchronization control function 2920 in FIG. 29 ). Image A on the floating in space image 3 side changes to image Ab as a display update, and image A on the mobile terminal 2000 side changes to image Ab as a display update.

[0374] Conversely, as shown in the lower part of Figure 38, when a touch operation is performed on image 3801 (image A) on the mobile terminal 2000 side, the system does not perform a display update on image 3801, nor does it reflect this on image 3802 (image A) on the floating image 3 side.

[0375] The synchronization control in this example is performed based on a setting that disables touch operation detection on the mobile terminal 2000 side and enables touch operation detection on the floating image 3 side (operation detection control function 2930 in FIG. 29). When two devices are connected for communication, this system performs settings related to the state after such connection. Each device controls operation, detection, and display according to the settings after connection. This achieves the effect shown in FIG. 38.

[0376] [Example 3 of Problem and Solution] Fig. 39 shows Example 3 of Problem and Solution. In the example of Fig. 39, first, as in Fig. 37, an image 3901 (for example, image B) is displayed on the screen of the mobile terminal 2000, and based on that image B, a similar image 3902 (for example, image B) is also displayed on the Space Floating Image 3, which is the screen of the Space Floating Image Display Device 1000 (dual display function 2910 in Fig. 29). One of the problems is that in this state, when the user 230 performs a rotation operation on, for example, image B on the screen 2005 of the mobile terminal 2000, it is conventionally unclear what will happen to these two images 3901 and 3902.

[0377] In this embodiment, one solution is as follows. The system detects a rotation operation on image 3901 (image B) on screen 2005 of mobile terminal 2000, for example, rotation of image 3901 due to rotation of the casing. Image B on screen 2005 as seen by user 230 becomes image Bc rotated by 90 degrees, for example. In response to this detection, the system synchronizes image 3902 (image B) on the side of floating in space image 3 so that the image rotation is reflected (synchronization control function 2920 in FIG. 29 ). Image B of floating in space image 3 as seen by user 230 becomes image Bc rotated by 90 degrees.

[0378] Alternatively, as shown in the lower part of Fig. 39 , the system detects a rotation operation, for example, image rotation by a touch operation, on image 3901 (image B) on screen 2005 of mobile terminal 2000. Image B on screen 2005 as seen by user 230 becomes image Bc, rotated by 90 degrees, for example. In response to this detection, the system synchronizes image 3902 (image B) on the floating-in-space image 3 side so that the image rotation is reflected (synchronization control function 2920 in Fig. 29 ).

[0379] The synchronization control in this example is performed based on the settings that enable detection of rotation operations and image rotation on the mobile terminal 2000 side and disable detection of rotation operations and image rotation on the floating image 3 side (operation detection control function 2930 in FIG. 29). When two devices are connected for communication, this system performs settings related to the state after such connection. Each device controls operation, detection, and display according to the settings after connection. This achieves the effects shown in FIG. 39.

[0380] [Example 4 of Problem and Solution] Fig. 40 shows Example 4 of Problem and Solution. In the example of Fig. 40, first, as in Fig. 39, an image 4001 (for example, image B) is displayed on the screen 2005 of the mobile terminal 2000, and based on that image B, a similar image 4002 (for example, image B) is also displayed on the space-floating image 3, which is the screen of the space-floating image display device 1000 (dual display function 2910 in Fig. 29). One of the problems is that in this state, when the user 230 performs a zoom operation by touch operation (particularly a pinch operation) on image B on the screen 2005 of the mobile terminal 2000, it is conventionally unclear what will happen to the two images 4001 and 4002.

[0381] In this embodiment, one solution is as follows. The system detects a touch operation, such as a pinch-out operation, on image 4001 (image B) on screen 2005 of mobile device 2000, and enlarges and displays image 4001 in response to the pinch-out operation. Image B becomes enlarged image Bd. In response to this detection, the system synchronizes image 4002 (image B) on the floating-in-space image 3 side so that the enlarged image display is reflected (synchronization control function 2920 in FIG. 29 ). As shown in the lower part of FIG. 40 , the same applies to a reduced display caused by a pinch-in operation.

[0382] The synchronization control in this example is performed based on the settings that enable the detection of touch operations and the zoom display on the mobile terminal 2000 side, and disable the detection of touch operations and the zoom display on the floating image 3 side (operation detection control function 2930 in FIG. 29). When two devices are connected for communication, this system performs settings related to the state after such connection. Each device controls operation, detection, and display according to the settings after connection. This achieves the effect shown in FIG. 40.

[0383] As in the above examples, when the mobile terminal 2000 and the space-floating image display device 1000 are connected, there are cases where the target image / video is displayed on both the screen of the mobile terminal 2000 and the screen of the space-floating image display device 1000 (space-floating image 3) (dual display). In this case, the user 230 may perform a touch operation or the like on the image displayed on the mobile terminal 2000, or may perform a touch operation or the like on the image displayed on the space-floating image 3. When a user operation is performed on one device, it has traditionally been unclear whether the operation should be reflected in the display image on the other device, and various possible responses have been considered. After communication connection is established, particularly in a dual display state, it is necessary to appropriately configure and control the operation and detection of the hardware and software of one or both devices, for the operation and detection on the mobile terminal 2000 side and the operation and detection on the space-floating image 3 side. By appropriately setting and controlling in accordance with the requests of user 230, it is possible to realize clear user operations on the display image on the mobile terminal 2000 side and the display image on the space floating image 3 side, and a response according to the operations (display control process). In one example, by setting the mobile terminal 2000 side as enabled and the space floating image 3 side as disabled, for example, as shown in Fig. 39, when user 230 rotates the display image on the mobile terminal 2000 side, it is possible to rotate the display image on the space floating image 3 side in synchronization with the operation. In this way, it is possible to clarify user operations and improve convenience for user 230.

[0384] As described above, in a state after communication connection, in a certain time period and mode, only operation detection on the mobile terminal 2000 side is enabled, and in another time period and mode, only operation detection on the space floating image display device 1000 side is enabled. In each mode, a predetermined function is realized. When operation detection on one device is enabled, unnecessary operation detection on the other device can be prevented. Depending on a predetermined mode, operation detection on both devices may be enabled.

[0385] In addition, when the communication connection between the mobile terminal 2000 and the space floating image display device 1000 is terminated, the system returns the settings related to operation detection of each device as described above to the state before the connection, i.e., the normal setting.

[0386] [Space-floating image display system and display method] Fig. 29 is an explanatory diagram of a space-floating image display system and display method in Example 5. A system 2900 in Fig. 29 is a space-floating image display system in which a space-floating image display device 1000 and a mobile terminal 2000 are communicatively connected. An outline of Example 5 will be described using Fig. 29.

[0387] State A shows a case where, after communication connection between the space-floating image display device 1000 and the mobile terminal 2000, for example, image 2901 on the screen 2005 of the mobile terminal 2000 is also displayed as image 2902 on the space-floating image 3 (display range 3R) of the space-floating image display device 1000. State A is a state of dual display of image 2901 on the mobile terminal 2000 side and image 2902 on the space-floating image 3 side. This dual display is a state in which images 2901 and 2902 of different aspects are displayed on the screens of each device based on the same image data.

[0388] State B shows a case where, from state A, for example, user 230 performs a touch operation on image 2901 on screen 2005 of mobile device 2000. In the example of Fig. 29, it is assumed that the system is set, as a post-connection setting (a certain mode described below), to enable operation detection on the mobile device 2000 side and disable operation detection on the floating-in-space image 3 side. Image 2901 is an object that accepts touch operations, and a predetermined process, for example, display update, that is predefined in association with image 2901 is executed. In this example, the touch operation and detection of image 2901 are determined to be valid, so image 2901 changes to image 2903 through display update.

[0389] Next, based on the detection of a touch operation and display update of image 2901 on the mobile terminal 2000 side, this system communicates with the space-floating image display device 1000 side, and as a synchronous control, reflects processing according to the touch operation on image 2903 on the space-floating image 3 side. In other words, as a result of the touch operation, the display of image 2902 on the space-floating image 3 side is also updated. By display update, image 2902 changes to image 2904. In this way, when the mobile terminal 2000 side is set to be valid, if user 230 performs a user operation on the mobile terminal 2000 side, the user operation can be reflected on the space-floating image 3 side.

[0390] On the other hand, when user 230 performs a touch operation on image 2902 displayed on the side of the floating in space image 3, because operation detection on the floating in space image 3 side is disabled, this system does not reflect the touch operation on image 2902 on the side of the floating in space image 3 or image 2901 on the side of the mobile terminal 2000. As a result, no display update is performed on either side, so the state of image 2902 on the floating in space image 3 and image 2901 on the side of the mobile terminal 2000 remains the same.

[0391] The system 2900 in FIG. 29 includes a dual display function 2910, a synchronization control function 2920, and an operation detection control function 2930 as functions for performing settings and controls such as those in the above example.

[0392] The dual display function 2910 is a function for displaying the image / video displayed on the screen of the Space-Floating Image 3 of the Space-Floating Image Display Device 1000 on the screen of the mobile terminal 2000 in accordance with the communication connection, and / or a function for displaying the image / video displayed on the screen of the mobile terminal 2000 on the screen of the Space-Floating Image 3 of the Space-Floating Image Display Device 1000. The synchronization control function 2920 is a function for synchronizing the display images on the screens of the devices in dual display so that when the user 230 performs an operation such as a touch operation on the screen of one device, a predetermined process (in other words, a display control process, a system process) executed in response to the operation is also reflected in the display image on the screen of the other device via communication. Note that the synchronization control function 2920 may be considered as an integrated part of the dual display function 2910.

[0393] Furthermore, the operation detection control function 2930 is a function that sets and controls whether to enable or disable either the operation, detection, and display control on the space floating image display device 1000 side or the operation, detection, and display control on the mobile terminal 2000 side, with regard to operations such as touch operations on the display image on the screen of each device and the predetermined processing executed in response to those operations.

[0394] [Dual Display Function] Fig. 30 is an explanatory diagram of the dual display function 2910 of Fig. 29. State A shows, as a first example of dual display, a case in which image A displayed in space-floating image 3 of space-floating image display device 1000 is also displayed as image A on screen 2005 of mobile terminal 2000. Initially, image A is displayed as image 3002 on space-floating image 3 of space-floating image display device 1000. User 230 connects mobile terminal 2000 to space-floating image display device 1000 via communication. Space-floating image display device 1000 transmits image data corresponding to image A to mobile terminal 2000 via communication, and mobile terminal 2000 acquires image data corresponding to image A from space-floating image display device 1000 via communication and displays image A as image 3001 on screen 2005. User 230 can see image A in space-floating image 3 and the corresponding image A on screen 2005 of mobile terminal 2000.

[0395] State B shows a second example of dual display, in which image B displayed on screen 2005 of mobile terminal 2000 is also displayed as image B in space-floating image 3 of space-floating image display device 1000. Initially, image B is displayed as image 3003 on screen 2005 of mobile terminal 2000. User 230 connects mobile terminal 2000 to space-floating image display device 1000 via communication. Mobile terminal 2000 transmits image data corresponding to image B to space-floating image display device 1000 via communication, and space-floating image display device 1000 obtains image data corresponding to image B from mobile terminal 2000 and displays image B as image 3004 in space-floating image 3. User 230 can see image B on screen 2005 of mobile terminal 2000, and can also see the corresponding image B in space-floating image 3.

[0396] [Modification regarding acquisition of image data] As shown in FIG. 30 and FIG. 32B described later, the present invention is not limited to transmitting target image data in the direction from the space-floating image display device 1000 to the mobile terminal 2000, or transmitting target image data in the opposite direction. In a modification, for example, ID information of the target image is transmitted from the space-floating image display device 1000 to the mobile terminal 2000. The mobile terminal 2000 identifies the target image data stored in the memory of the mobile terminal 2000 from the received ID information, and displays the target image on the screen 2005 based on the identified image data. Alternatively, the mobile terminal 2000 refers to an external server or the like from the received ID information (which may be a URL, etc.), identifies the target image data stored in the external server, etc., acquires the identified image data from the external server, etc., and displays the target image on the screen 2005 based on the image data.

[0397] [Basic Control Flow] Fig. 31 shows the basic control flow of the space-floating image display system (system 2900 in Fig. 29) and display method in Example 5. The main components of each device of the space-floating image display device 1000 and the mobile terminal 2000, in particular the control unit of the mobile terminal 2000 and the image processing unit of the space-floating image display device 1000, perform processing related to the control in this example. Unless otherwise specified below, the main components are the same.

[0398] Step S10 is the state before the space floating image display device 1000 and the mobile terminal 2000 are connected for communication (in other words, the state of non-connection). Each device is in a state of optional use. Note that in step S10, the mode described below may be selected and set in advance as a system setting or user setting.

[0399] In step S11, the mobile terminal 2000 and the space-floating image display device 1000 are communicatively connected from the state before connection in step S10. For example, the mobile terminal 2000 is communicatively connected to the space-floating image display device 1000 based on the operation / motion of the user 230 who holds the mobile terminal 2000. The communication connection here is a connection in any manner, regardless of the details including the communication interface. This communication connection may be a conventional general method in which the mobile terminal 2000 sends a connection request to the space-floating image display device 1000 and establishes a connection with the space-floating image display device 1000. This communication connection may also be a communication connection by touching the mobile terminal 2000 to the space-floating image 3, as a method unique to the above-mentioned third and fourth embodiments.

[0400] Step S12 represents the processing and state immediately after the communication connection. In step S12, each device, the mobile terminal 2000 and the space-floating image display device 1000 of this system, performs connection settings, in other words, settings immediately after connection, settings related to the state of this system after connection, and settings after connection. Each device performs its own "post-connection settings" based on the communication link with the other device. This setting includes settings related to hardware, software, functions, etc. This setting mainly relates to enabling / disabling user operations and sensor detection on each device in the operation detection function 2930 of FIG. 29. This setting may also include settings related to the images displayed on the screen of each device in the dual display function 2910 and synchronization control function 2920 of FIG. 29. Details of the "post-connection settings" will be described later (FIG. 32B).

[0401] This setting, as a setting for detecting operation on the mobile terminal 2000 side, includes a setting for enabling / disabling a sensor for detecting touch operations and rotation operations on the screen of the mobile terminal 2000. Also, this setting, as a setting for detecting operation on the space-floating image display device 1000 side, includes a setting for enabling / disabling a sensor for detecting touch operations and rotation operations on the screen (display range 3R) of the space-floating image 3. The "setting after connection" continues until the communication connection, which will be described later, is released.

[0402] In step S12, the system may select and set a mode, which will be described later. For example, a GUI for selecting and setting a mode may be displayed on the screen of one of the devices, and the user 230 may select and set the mode.

[0403] Step S13 shows the post-connection setting state, in other words, the state of use of each device of this system by user 230 in a communication connection state. For example, as a dual display, the display image on the Space Floating Image 3 side is also displayed on the screen of the mobile device 2000 side. Alternatively, the image on the mobile device 2000 side is also displayed on the Space Floating Image 3 side. User 230 uses the display image on the screen of the mobile device 2000 and also uses the display image on the Space Floating Image 3 of the Space Floating Image Display Device 1000. In this case, user operations such as touch operations and rotation operations on the display image on the screen of each device are enabled / disabled, synchronized, etc., based on the post-connection setting in step S12. For example, in a certain setting mode, touch operations on the display image on the mobile device 2000 side are enabled, and in response to such operations, reactions such as display updates of the display images on each device occur, but touch operations on the display image on the Space Floating Image 3 side are disabled. This makes operation and detection clear during dual display, making it easier for user 230 to operate. Details of each mode will be described later.

[0404] In step S14, the communication connection between the mobile terminal 2000 and the space-floating image display device 1000 is released. For example, the communication connection between the mobile terminal 2000 and the space-floating image display device 1000 is released based on the operation or movement of the user 230 who owns the mobile terminal 2000. The release of the connection includes not only an intentional disconnection or release operation by the user 230, but also an unintentional disconnection or release due to an error or the like. In step S14, the system returns to the state after the connection is released, in other words, to a non-connected state, and returns to the state before the connection.

[0405] After step S14, in step S15, each device of the mobile terminal 2000 and the space floating image display device 1000 of this system performs the settings at the time of disconnection, in other words, the settings immediately after disconnection, the settings related to the state before connection of this system, and the settings before connection. This setting includes settings related to the hardware, software, functions, etc. of each device. Each device performs its own "pre-connection settings". This setting corresponds to returning to the state before connection in step S10. For example, it is returned to the settings shown in FIG. 32A described below. Note that this system may store the "pre-connection settings" in memory when connecting for communication in steps S11 and S12, and may return to the "pre-connection settings" based on the information in the memory when disconnecting in steps S14 and S15.

[0406] [Examples of displays and settings before and after connection] Figures 32A and 32B are explanatory diagrams summarizing in table form examples of displays and settings (in other words, modes and patterns) before and after communication connection between the mobile terminal 2000 and the space-floating image display device 1000 in this system.

[0407] FIG. 32A shows an example of the display and settings before connection. This setting corresponds to the state of step S10 in FIG. 31. The table shows the settings of the mobile terminal 2000 (abbreviated as MT in the table) and the space-floating image display device 1000 (abbreviated as AD in the table) before connection of this system. The space-floating image display device 1000 displays a predetermined image, for example, image A (in other words, content A) selected by the user 230, on the screen of the space-floating image 3. The mobile terminal 2000 displays a predetermined image, for example, image B (in other words, content B) selected by the user 230, on the screen of the display panel. Images A and B are different images / contents. On the space-floating image display device 1000 side, a touch sensor (described below) provided in the AD is used for touch operation and detection of image A on the screen, and an orientation sensor (described below) provided in the AD is used for rotation operation and detection of image A on the screen. On the mobile terminal 2000 side, a touch sensor (described later) provided in the MT is used for touch operations and detection of image B on the screen, and an attitude sensor (described later) provided in the MT is used for rotation operations and detection of image B on the screen.

[0408] Before connection, the two devices (MT, AD) are independent, so FIG. 32A simply illustrates the application of general sensor and other technologies in each device.

[0409] Fig. 32B shows an example of display and settings after connection. This setting corresponds to the setting in step S12 in Fig. 31. In the table, as the setting after connection of this system, the display image of each screen of each device of the mobile terminal 2000 (MT) and the space floating image display device 1000 (AD), the touch sensor used for touch operation and detection (in other words, which device's operation and detection should be enabled), and the attitude sensor used for rotation operation and detection (in other words, which device's operation and detection should be enabled) are shown.

[0410] The settings at the time of connection in step S12 in Fig. 31 correspond to the "settings after connection" as shown in Fig. 32B. This setting includes the setting for selecting the image data storage source, which will be described later.

[0411] The settings of the enabled devices / sensors in the "Touch Operation Detection Touch Sensor" column and the "Rotation Operation Detection Orientation Sensor" column specify which device's user operations and the responses to those operations are enabled, and do not prevent the normal use of various sensor devices (touch sensors and orientation sensors). The sensor devices of the device set as disabled still perform normal detection operations (e.g., generation and output of touch detection signals) and are used for various existing functions. When set as disabled, for example, predetermined processing in response to the detection and determination of touch operations is not executed.

[0412] Note that the terms "touch sensor" and "posture sensor" in the table refer to any type of sensor for detecting touch operations or rotation operations on the display image of each screen of each device, and include specific hardware and software that vary depending on the implementation, without limiting the details.

[0413] Depending on the implementation of this system, the housing 1190 of the space-floating image display device 1000 may not be equipped with an orientation sensor 1113 (FIG. 3). This is because, when the housing 1190 and the space-floating image 3 (display range 3R) are fixedly installed in a specific orientation, detecting the orientation of the housing 1190 and the space-floating image 3 is not essential. In this case, it can be considered that the control of the rotation operation related to the "orientation sensor" column in the table of FIG. 32B is not applicable. In other words, in this case, the space-floating image display device 1000 (AD) does not perform control related to the rotation operation of the displayed image.

[0414] In Fig. 32B, the multiple examples (in other words, modes) can be broadly categorized as follows. Examples 1 to 6 are examples in which, like state A in Fig. 30, image A displayed on the space floating image 3 side is also displayed as image A on the screen of the mobile terminal 2000 side. User 230 can see both of these images A. Examples 7 to 12 are examples in which, like state B in Fig. 30, image B displayed on the screen of the mobile terminal 2000 side is also displayed as image B on the space floating image 3 side. User 230 can see both of these images B.

[0415] In each example and mode, the operation and detection of which device is enabled is set as shown in the right column. In this embodiment, user operations to be controlled are broadly divided into touch operations and rotation operations (see FIG. 42, described below). Each device has a touch sensor used to detect touch operations and an orientation sensor (in other words, a rotation sensor) used to detect rotation operations. For each operation and detection, the "Touch Operation Detection Touch Sensor" column in the table indicates which of the devices, the AD-side or the MT-side, the touch sensor is enabled. The "Rotation Operation Detection Orientation Sensor" column in the table indicates which of the devices, the AD-side or the MT-side, the orientation sensor is enabled. For example, in the row for Example 1, the touch sensor on the AD side is enabled, and the orientation sensor on the AD side is enabled. In other words, this setting means that the touch sensor on the MT side is disabled, and the orientation sensor on the MT side is disabled. Furthermore, for example, in the row for Example 4, the touch sensor on the MT side is enabled, and the orientation sensor on the MT side is enabled. In other words, this setting means that the touch sensor on the AD side is disabled and the attitude sensor on the AD side is disabled. Also, in Examples 5 and 6, the "Touch Operation Detection Touch Sensor" column in the table states "AD or MT." This indicates that both the touch sensor on the AD side and the touch sensor on the MT side are enabled. Specific operations for each example will be described later.

[0416] [Regarding the Use of Each Mode] At least one of the various examples and modes shown in FIG. 32B is implemented and made available in the design or configuration of this system. The system may be implemented and configured so that only one mode is fixed, or may be implemented and configured so that multiple modes can be selected and used. In the latter case, this system predefines and prepares the various setting examples and modes shown in the table of FIG. 32B and controls the system to switch between these modes as appropriate. In this system, the user 230 may be able to select and set the mode to be used through user settings, etc.

[0417] [Example of a Mode Selection / Setting Screen] Fig. 33 shows an example of a screen display when a user setting GUI is provided so that the user 230 can select and set the above modes. The example of Fig. 33 is an example in which a GUI 3300 that allows mode selection is displayed on the screen 2005 of the mobile terminal 2000. This is not a limitation, and a GUI using floating-in-space images 3 may also be provided in a similar manner. Also, while this example shows a case in which modes 1 to 4 in Fig. 32B are selected, this is not a limitation, and the GUI screen may display a table or detailed information such as that in Fig. 32B to allow each mode to be selected.

[0418] In the example of Fig. 33, when image A on the side of the floating image 3 is also acquired on the side of the mobile terminal 2000 and displayed in double, as in state A of Fig. 30, modes 1 to 4 relating to the enable / disable of touch operation (touch sensor) and rotation operation (posture sensor) corresponding to examples 1 to 4 of Fig. 32B are displayed as options for "setting the operation / detection mode for the displayed image", allowing mode selection by operation of the user 230. For example, mode 4 is selected, and mode 4 is set as the current mode.

[0419] Fig. 34 shows an example of a GUI with a different expression from that of Fig. 33 for selecting and setting the above modes. In the example of Fig. 34, in the GUI 3400 on the screen 2005 of the mobile terminal 2000, as a "setting of the operation / detection mode for the displayed image," it is possible to select and set, using a GUI such as a list box, whether to enable or use operation / detection of either the mobile terminal 2000 (MT) side or the space floating image 3 (AD) side, or both, for touch operation and rotation operation. These selection settings correspond to mode selection settings. For example, only MT is selected for touch operation and only MT for rotation operation, which corresponds to mode 4.

[0420] In this example, user operations can be divided into touch operations and rotation operations and can be set for each, but it is also possible to set only touch operations or only rotation operations. Also, in this example, with regard to rotation operations, only the rotation operations of one device can be selected as valid, but this is not limiting, and it is also possible to select rotation operations of both devices.

[0421] [Example of Functional Block Configuration] FIG. 35 shows an example of the functional block configuration of this system. An example configuration for realizing each function as shown in FIG. 29 is shown. The mobile terminal 2000 includes a control unit (processor) 2001, memory 2002, communication interface 2003, sensor 2004, display screen (screen) 2005, etc., which are interconnected by a predetermined architecture such as a bus. The control unit (processor) 2001 realizes a predetermined control function 2011 based on the execution of processing such as a program by the processor. This control function 2011 includes the dual display function 2910, s...

Claims

1. A floating-in-the-air image display device comprising: an image processing unit; a display unit that displays an image that has been image-processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; a detection unit that detects contact of an object with the display area of ​​the floating-in-the-air image; and a communication unit that communicates with a user's mobile device, wherein when contact of the mobile device with the display area is detected, a target image specified on the mobile device is displayed as the floating-in-the-air image.

2. A floating-in-the-air image display device as claimed in claim 1, wherein a display request is generated from the mobile terminal to the floating-in-the-air image display device before, during or after the mobile terminal comes into contact with the display area, and when the contact of the mobile terminal with the display area is detected, the display request is received, a target image specified on the mobile terminal side is obtained via communication from the mobile terminal or an external device, and displayed as the floating-in-the-air image.

3. A floating-in-the-air image display device as claimed in claim 1, wherein, when contact of the mobile terminal with a specific area of ​​the display range is detected, a target image specified on the mobile terminal side is displayed as the floating-in-the-air image.

4. A floating-in-the-air image display device as claimed in claim 1, wherein, when a specific attitude of the portable terminal is detected when the portable terminal comes into contact with the display area, a target image specified on the portable terminal side is displayed as the floating-in-the-air image.

5. A floating-in-the-air image display device as claimed in claim 1, wherein when it is detected that the portable terminal has entered a space behind the display range after the portable terminal has come into contact with the display range, a target image specified on the portable terminal side is displayed as the floating-in-the-air image.

6. A floating-in-the-air image display device as claimed in claim 1, wherein when the mobile terminal is brought into contact with the display range, if the contact time exceeds a certain period of time, a target image specified on the mobile terminal side is displayed as the floating-in-the-air image.

7. A floating-in-the-air image display device according to claim 1, wherein a guide image for encouraging the user to touch the display area with the mobile terminal is displayed as the floating image.

8. A floating-in-the-air image display device according to claim 1, further comprising: a guide image for prompting the user to touch the display area with the mobile terminal on the screen of the mobile terminal.

9. A floating-in-the-air image display device as described in claim 1, wherein, when an invisible area caused by the mobile terminal touching the display range is detected, a non-display area including the invisible area within the display range is determined, and the floating-in-the-air image is displayed in a display area other than the non-display area.

10. A floating-in-the-air image display device according to claim 1, wherein the detection unit is at least one of an aerial operation detection sensor that detects an aerial operation in a range including the display range, or an imaging unit that images a range including the display range.

11. A floating image display device as claimed in claim 1, which recognizes a code displayed on the screen of the mobile device to acquire information, and when the code is recognised and contact of the mobile device with the display area is detected, displays a target image specified on the mobile device as the floating image.

12. A floating-in-the-air image display device as claimed in claim 1, wherein the mobile terminal recognizes a code displayed on the floating image in the space and acquires information, and when the code is recognized and contact of the mobile terminal with the display area is detected, a target image specified on the mobile terminal is displayed as the floating-in-the-air image.

13. In the air-floating image display device according to claim 11 or 12, the code information is at least one of the following information: a display request from the mobile terminal to the air-floating image display device; connection information for communication between the mobile terminal and the air-floating image display device; an ID of the mobile terminal or an ID of the user; an ID or URL of the target image; and an air-floating image display device.

14. A floating-in-the-air image display device as claimed in claim 1, wherein, when contact of the mobile terminal with the display range is detected, the image displayed on the floating-in-the-air image is acquired by the mobile terminal.

15. A floating-in-the-air image display system having a floating-in-the-air image display device and a user's portable terminal, the floating-in-the-air image display device comprising: an image processing unit; a display unit that displays an image that has been image-processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; a detection unit that detects contact of an object with the display area of ​​the floating-in-the-air image; and a communication unit that communicates with the user's portable terminal, and when the floating-in-the-air image display device detects contact of the portable terminal with the display area, it displays a target image specified on the portable terminal as the floating-in-the-air image.

16. A portable terminal in a floating-in-the-air image display system having a floating-in-the-air image display device and a user's portable terminal, wherein the floating-in-the-air image display device comprises: an image processing unit; a display unit that displays an image that has been image-processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; a detection unit that detects contact of an object with the display area of ​​the floating-in-the-air image; and a communication unit that communicates with the user's portable terminal, wherein when the floating-in-the-air image display device detects contact of the portable terminal with the display area, it displays a target image specified on the portable terminal as the floating image, and the portable terminal specifies the target image based on operation by the user, and transmits image data or information of the specified target image to the floating-in-the-air image display device.

17. A floating-in-the-air image display device, comprising: an image processing unit that performs image processing; a display unit that displays an image processed by the image processing unit; an optical system that generates a floating-in-the-air image based on the image displayed by the display unit; a user operation detection mechanism that detects an operation by a user on a display range of the floating-in-the-air image; and a communication unit that communicates with a mobile terminal of the user, wherein, based on a communication connection with the mobile terminal of the user, a first image displayed on the floating-in-the-air image is also displayed as a first image on the screen of the mobile terminal, and / or a second image displayed on the screen of the mobile terminal is also displayed as a second image on the floating-in-the-air image, a predetermined first process associated with the first image of the floating-in-the-air image is executed based on detection and determination of an operation by the user on the first image displayed in the floating-in-the-air image, and control is performed so that the first process is also reflected in the first image on the screen of the portable device, and / or a predetermined second process associated with the second image on the screen of the portable device is executed based on detection and determination of an operation by the user on the second image displayed on the screen of the portable device, on the portable device side, and control is performed so that the second process is also reflected in the second image of the floating-in-the-air image.

18. A floating-in-the-air image display device as described in claim 17, wherein, as a setting after said communication connection, enable / disable is set for the user's operation and detection of the displayed image of the floating image on said floating-in-the-air image display device side and for the user's operation and detection of the displayed image on the screen of said mobile terminal side, and said operation, detection, display and processing are controlled in accordance with said setting.

19. A floating-in-the-air image display device as claimed in claim 18, wherein, at the time of said communication connection, the enable / disable setting after said communication connection is performed, and, at the time of said communication connection being released, the enable / disable setting is returned to the setting before said communication connection.

20. A floating-in-the-air image display device as claimed in claim 18, wherein the user's operations on the floating-in-the-air image or the image displayed on the screen of the mobile terminal include a touch operation and a rotation operation, the touch operation is detected using a touch sensor in the user operation detection mechanism provided on the floating-in-the-air image display device or a touch sensor provided on the mobile terminal, and the rotation operation is detected using an attitude sensor provided on the floating-in-the-air image display device or an attitude sensor provided on the mobile terminal.

21. A floating-in-the-air image display device as claimed in claim 18, wherein in a certain mode, the user's operation and detection of the image displayed on the screen of the floating-in-the-air image display device is set to be enabled, and the user's operation and detection of the image displayed on the screen of the mobile terminal is set to be disabled.

22. A floating-in-the-air image display device as claimed in claim 18, wherein in a certain mode, the user's operation and detection of the image displayed on the screen of the floating-in-the-air image display device is set to be disabled, and the user's operation and detection of the image displayed on the screen of the mobile terminal is set to be enabled.

23. A floating-in-the-air image display device as claimed in claim 18, wherein in a certain mode, the user's operation and detection of the image displayed on the screen of the floating-in-the-air image display device is enabled, and the user's operation and detection of the image displayed on the screen of the mobile terminal is also enabled.

24. A floating-in-the-air image display device as claimed in claim 18, wherein mode options corresponding to enable / disable settings of the user's operation and detection of the displayed image of the floating-in-the-air image on the floating-in-the-air image display device and the user's operation and detection of the displayed image on the screen of the portable terminal are displayed on the floating-in-the-air image display device and / or on the screen of the portable terminal, and the enable / disable setting is set in accordance with the mode selection made by the user.

25. A floating-in-the-air image display device according to claim 20, wherein the touch sensor provided on the floating-in-the-air image display device has an air-operation detection sensor or a camera that detects the position of the user's fingers or an object to be operated relative to the display range of the floating-in-the-air image.

26. A floating-in-the-air image display device as claimed in claim 20, wherein the rotation operation includes a rotation operation of a housing of the floating-in-the-air image display device or a housing of the mobile terminal, and the predetermined processing responsive to the determination of the rotation operation includes displaying an image in the floating-in-the-air image or an image on the screen of the mobile terminal so that the image appears rotated as seen by the user.

27. A floating-in-the-air image display device as claimed in claim 17, wherein, upon detection of contact of the mobile terminal with the display range of the floating-in-the-air image during said communication connection, a first image displayed on said floating-in-the-air image is also displayed as a first image on the screen of said mobile terminal, and / or a second image displayed on the screen of said mobile terminal is also displayed as a second image on said floating-in-the-air image.

28. A display method in a system in which a floating-in-the-air image display device and a user's mobile terminal are communicatively connected, the floating-in-the-air image display device comprising: an image processing unit that performs image processing; a display unit that displays an image processed by the image processing unit; an optical system that generates a floating image based on the image displayed by the display unit; a user operation detection mechanism that detects an operation by a user on the display range of the floating-in-the-air image; and a communication unit that communicates with the user's mobile terminal, the method comprising the steps of: establishing a communication connection between the floating-in-the-air image display device and the user's mobile terminal; and, based on the communication connection, displaying a first image displayed on the floating-in-the-air image also on the screen of the mobile terminal as a first image and / or displaying a second image displayed on the screen of the mobile terminal as a second image on the floating-in-the-air image. a step of executing a predetermined first process associated with the first image of the floating-in-the-air image based on detection and determination of an operation by the user on the first image displayed on the screen of the portable device, and controlling so that the first process is also reflected in the first image on the screen of the portable device, and / or a step of executing a predetermined second process associated with the second image on the screen of the portable device based on detection and determination of an operation by the user on the second image displayed on the screen of the portable device, on the portable device side, and controlling so that the second process is also reflected in the second image of the floating-in-the-air image.

29. A portable terminal in a system in which a floating-in-the-air image display device and a user's portable terminal are communicatively connected, the floating-in-the-air image display device comprising: an image processing unit that performs image processing; a display unit that displays an image processed by the image processing unit; an optical system that generates a floating-in-the-air image based on the image displayed by the display unit; a user operation detection mechanism that detects an operation by a user on the display range of the floating-in-the-air image; and a communication unit that communicates with the user's portable terminal, the portable terminal communicatively connects with the floating-in-the-air image display device, and based on the communication connection, displays a first image displayed on the floating-in-the-air image also on the screen of the portable terminal as a first image, and / or displays a second image displayed on the screen of the portable terminal as a second image on the floating-in-the-air image, A mobile terminal that, based on detection and determination of a user's operation on the first image displayed in the floating-in-the-air image, executes a predetermined first process associated with the first image of the floating-in-the-air image and controls to reflect the first process in the first image on the screen of the mobile terminal, and / or, based on detection and determination of a user's operation on the second image displayed on the screen of the mobile terminal, executes a predetermined second process associated with the second image on the screen of the mobile terminal and controls to reflect the second process in the second image of the floating-in-the-air image.

Images