Space-floating image display device

Abstract

To provide a more suitable space floating video display device.SOLUTION: A space floating video display device continuously displays a virtual shadow of a user's finger on a display surface of a space floating video until a distance in a normal direction between the position of a tip of the user's finger and the display surface of the space floating video becomes zero when the position of a tip of the user's finger approaches the display surface of the space floating video such that the distance in the normal direction to the display surface of the space floating video becomes small on a near side of the display surface of the space floating video viewed from the user.SELECTED DRAWING: Figure 27

Description

[Technical Field]

[0001] The present invention relates to a space floating image display device. [Background technology]

[0002] As a space-floating information display system, an image display device that displays an image directly to the outside and a display method that displays it as a spatial screen are already known. In addition, a detection system that reduces false detections of operations on the operation surface of a displayed spatial image is also disclosed, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-128722 Summary of the Invention [Problem to be solved by the invention]

[0004] However, touch operations on a floating image in space are not performed on a physical button, touch panel, etc. Therefore, there are cases where the user cannot recognize whether or not a touch operation has been performed. Therefore, an object of the present invention is to provide a more suitable space floating image display device. [Means for solving the problem]

[0005] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problem, and as an example, a space-floating image display device includes a display device that displays an image, a retroreflecting member that reflects image light from the display device and forms a space-floating image in the air with the reflected light, a sensor that detects the position of a user's finger that performs a touch operation on one or more objects displayed in the space-floating image, and a control unit, and the control unit controls image processing for the image displayed on the display device based on the position of the user's finger detected using the sensor, so that the space-floating image display device displays a virtual shadow of the user's finger on the display surface of the space-floating image, which has no physical contact surface. The position of the virtual shadow displayed on the display surface of the space-floating image is a position identified from the positional relationship between the position of the virtual light source and the position of the user's finger detected using the sensor, and the position of the virtual light source is set at an infinite distance from the display surface of the space-floating image, and when the position of the tip of the user's finger approaches the display surface of the space-floating image so that the normal distance to the display surface of the space-floating image becomes small on the front side of the display surface of the space-floating image as seen from the user, the virtual shadow of the user's finger continues to be displayed on the display surface of the space-floating image until the normal distance between the position of the tip of the user's finger and the display surface of the space-floating image becomes zero. [Effects of the Invention]

[0006] According to the present invention, a more suitable space floating image display device can be realized. Other problems, configurations, and effects will become clear in the following description of the embodiments. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram showing an example of a usage form of a space floating image display device according to an embodiment of the present invention; [Figure 2] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 3A] 10A and 10B are diagrams illustrating an example of a method for installing a space floating image display device. [Figure 3B] 10A and 10B are diagrams showing other examples of installation methods for the space floating image display device. [Figure 3C] FIG. 1 is a diagram illustrating an example of the configuration of a space floating image display device. [Figure 4] FIG. 10 is a diagram showing another example of the main part configuration of the space floating image display device according to an embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram for explaining the function of a sensing device used in the space floating image display device. [Figure 6] FIG. 1 is an explanatory diagram of the principle of 3D image display used in a space floating image display device. [Figure 7] FIG. 1 is an explanatory diagram of a measurement system for evaluating the characteristics of a reflective polarizing plate. [Figure 8] FIG. 10 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the angle of incidence of light. [Figure 9] FIG. 10 is a characteristic diagram showing transmittance characteristics of the reflection axis of a reflective polarizing plate relative to the angle of incidence of light. [Figure 10] FIG. 10 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the angle of incidence of light. [Figure 11] FIG. 10 is a characteristic diagram showing the transmittance characteristics of the reflection axis of a reflective polarizing plate versus the angle of incidence of light. [Figure 12] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 13] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 14] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 15] 1 is a layout diagram showing a main part of a space floating image display device according to an embodiment of the present invention; [Figure 16] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 17] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 18] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 19] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 20] 10A and 10B are explanatory diagrams for explaining the light source diffusion characteristics of the image display device. [Figure 21] FIG. 10 is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Figure 22] FIG. 10 is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Figure 23] 1 is a cross-sectional view showing a configuration of a video display device. [Figure 24] 1 is an explanatory diagram for explaining the principle of occurrence of a ghost image in the prior art; [Figure 25] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 26] FIG. 2 is a diagram illustrating an example of a display on a display device according to an embodiment of the present invention. [Figure 27] 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Figure 28] 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Figure 29] 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Figure 30] 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Figure 31] 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Figure 32] 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Figure 33] FIG. 10 is a diagram illustrating a method for setting a virtual light source. [Figure 34] FIG. 10 is a configuration diagram showing an example of a method for detecting the position of a finger. [Figure 35] FIG. 10 is a configuration diagram showing another example of a method for detecting the position of a finger. [Figure 36] FIG. 10 is a configuration diagram showing another example of a method for detecting the position of a finger. [Figure 37] 10A and 10B are diagrams illustrating a method of assisting a touch operation by displaying input contents. [Figure 38]10A and 10B are diagrams illustrating a method of highlighting input content to assist touch operations. [Figure 39] 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation by vibration. [Figure 40] 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation by vibration. [Figure 41] 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation by vibration. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, components having the same functions are given the same reference numerals, and repeated explanations thereof may be omitted.

[0009] The following embodiments relate to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member.

[0010] According to the following embodiments, an image display device suitable for, for example, bank ATMs, train station ticket machines, digital signage, and the like can be realized. For example, currently, bank ATMs, train station ticket machines, and the like typically use touch panels. However, by using a transparent glass surface or a light-transmitting plate, high-resolution image information can be displayed in a floating state on the glass surface or light-transmitting plate. In this case, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflective material. This improves light utilization efficiency and suppresses the ghost images that occur in addition to the main floating image, which is a problem with conventional retroreflective systems, thereby achieving a clear floating image. Furthermore, by using a device including the light source of this embodiment, a novel and highly usable floating image display device (floating image display system) can be provided that can significantly reduce power consumption. Furthermore, a floating image display device for a vehicle can be provided that can display a so-called unidirectional floating image that can be viewed inside and / or outside the vehicle.

[0011] On the other hand, in conventional technology, an organic EL panel or a liquid crystal panel is combined with a retroreflective member 151 as a high-resolution color display image source 150. In conventional technology, image light is diffused over a wide angle, so in addition to the light reflected normally by the retroreflective member 151, ghost images 301 and 302 are generated by image light incident obliquely on the retroreflective member 2a as shown in FIG. 24, impairing the image quality of the floating image in space. Furthermore, as shown in FIG. 23, in addition to the normal floating image in space 300, multiple ghost images such as a first ghost image 301 and a second ghost image 302 are generated. For this reason, the same floating image in space, which is a ghost image, can be observed by people other than the observer, posing a major security issue. <Space-floating image display device 1>

[0012] Fig. 1 is a diagram showing an example of a usage form of a space-floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using Fig. 2 etc., but light with a narrow-angle directional characteristic and specific polarization is emitted from the image display device 1 as an image light beam, once enters the retroreflective member 2, is retroreflected and passes through a transparent member 100 (glass etc.), and forms a real aerial image (space-floating image 3) on the outside of the glass surface.

[0013] In addition, in a store or the like, a space is partitioned by a show window (also called "window glass") 105, which is a translucent member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit such a transparent member and display a floating image in one direction to the outside and / or inside of the store (space).

[0014] 1(A), the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing a means for reflecting specific polarized waves in the window glass 105, it is possible to reflect the waves and form an aerial image at a desired position inside the store.

[0015] FIG. 1(B) is a schematic block diagram showing the configuration of the above-mentioned video display device 1. The video display device 1 includes a video display unit that displays the original aerial image, a video control unit that converts the input video to match the resolution of the panel, and a video signal receiving unit that receives the video signal. The video signal receiving unit supports wired input signals such as HDMI (High-Definition Multimedia Interface) input and wireless input signals such as Wi-Fi (Wireless Fidelity), and can function as a standalone video receiver / display device, and can also display video information from tablets, smartphones, etc. Furthermore, by connecting a stick PC or the like, it can be equipped with capabilities such as calculation processing and video analysis processing.

[0016] Fig. 2 is a diagram showing an example of the configuration of the main components and the retroreflector of a space-floating image display device according to one embodiment of the present invention. The configuration of the space-floating image display device will be described in more detail using Fig. 2. As shown in Fig. 2(A), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light with a narrow-angle diffusion characteristic.

[0017] Image light of a specific polarization from the display device 1 is reflected by a polarization separator 101 (in the figure, the polarization separator 101 is formed into a sheet and adhered to the transparent member 100) that has a film that selectively reflects image light of a specific polarization and is provided on a transparent member 100, and then enters the retroreflective member 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflective member. The image light is polarized and converted from the specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when entering the retroreflective member and once when exiting. Here, the polarization separator 101 that selectively reflects image light of a specific polarization has the property of transmitting the polarized light of the other polarization that has been polarization-converted, so the image light of the specific polarization after polarization conversion passes through the polarization separator 101. The image light that has passed through the polarization separator 101 forms a space-floating image 3, which is a real image, outside the transparent member 100.

[0018] The light that forms the floating image 3 is a collection of light rays that converge from the retroreflective member 2 to the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is a highly directional image, unlike the diffused image light formed on a screen by a general projector or the like. Therefore, in the configuration of Figure 2, when a user views the floating image 3 from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views the floating image 3 from the direction of arrow B, the floating image 3 cannot be perceived as an image at all. This characteristic is very suitable for use in systems that display images that require high security or highly confidential images that should be kept secret from people directly facing the user.

[0019] Depending on the performance of the retroreflective member 2, the polarization axis of the reflected image light may become misaligned. In this case, some of the image light with misaligned polarization axes is reflected by the polarization separation member 101 described above and returns to the display device 1. This light may be re-reflected on the image display surface of the liquid crystal display panel 11 constituting the display device 1, generating ghost images and potentially degrading the image quality of the spatially floating image. Therefore, in this embodiment, an absorbing polarizer 12 is provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorbing polarizer 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorbing polarizer 12, thereby suppressing the re-reflection. This prevents degradation of image quality due to ghost images of the spatially floating image.

[0020] The polarization separation member 101 may be formed of, for example, a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

[0021] Next, Figure 2(B) shows the surface shape of a typical retroreflective member 2 manufactured by Nippon Carbide Industries Co., Ltd., used in this study. Light rays incident on the regularly arranged hexagonal prisms are reflected by the walls and bottom of the hexagonal prisms and exit as retroreflected light in a direction corresponding to the incident light. This results in a real, floating image based on the image displayed on the display device 1. The resolution of this floating image depends not only on the resolution of the LCD panel 11 but also on the outer diameter D and pitch P of the retroreflective portion of the retroreflective member 2 shown in Figure 2(B). For example, when using a 7-inch WUXGA (1920 × 1200 pixels) LCD panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, then one pixel of the floating image will be equivalent to 300 μm. This reduces the effective resolution of the floating image to approximately one-third. Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, to suppress the occurrence of moire caused by the retroreflective material and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Also, it is advisable to arrange the shape of the retroreflective portion so that none of its sides overlaps any of the sides of one pixel of the liquid crystal display panel.

[0022] On the other hand, to manufacture retroreflective members at low cost, it is recommended to use the roll press method. Specifically, this method aligns the retroreflective parts and forms them on a film. The reverse shape of the shape to be formed is formed on the surface of a roll, and a UV-curable resin is applied to a base material for fixing. The resin is then passed between the rolls to form the required shape, and UV light is applied to cure the resin, resulting in the retroreflective member 2 of the desired shape. <<How to install the space floating image display device>>

[0023] Next, we will explain how to install the space-floating image display device. The installation method of the space-floating image display device can be freely changed depending on the usage form. Fig. 3A is a diagram showing an example of how to install the space-floating image display device. The space-floating image display device shown in Fig. 3A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in Fig. 3A, the space-floating image display device is installed so that the transparent member 100 faces upward, and the space-floating image 3 is formed above the space-floating image display device.

[0024] Fig. 3B is a diagram showing another example of a method for installing a space-floating image display device. The space-floating image display device shown in Fig. 3B is installed vertically so that the surface on which the space-floating image 3 is formed faces sideways (towards the user 230). That is, in Fig. 3B, the space-floating image display device is installed so that the transparent member 100 faces sideways, and the space-floating image 3 is formed on the side of the space-floating image display device (towards the user 230). <<Configuration of the Space Floating Image Display Device>>

[0025] Next, we will explain the configuration of space-floating image display device 1000. Fig. 3C is a block diagram showing an example of the internal configuration of space-floating image display device 1000.

[0026] The space floating image display device 1 includes a retroreflecting unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power supply 1106, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, an image signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc.

[0027] Each component of the space floating image display device 1000 is housed in a housing 1190. Note that the imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0028] The retroreflecting portion 1101 in Fig. 3C corresponds to the retroreflecting member 2 in Fig. 2. The retroreflecting portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected from the retroreflecting portion 1101, the light output to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0029] Image display unit 1102 in Fig. 3C corresponds to liquid crystal display panel 11 in Fig. 2. Light source 1105 in Fig. 3C corresponds to light source device 13 in Fig. 2. Image display unit 1102, light guide 1104, and light source 1105 in Fig. 3C correspond to display device 1 in Fig. 2.

[0030] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 (described later). The video display unit 1102 corresponds to the liquid crystal display panel 11 in FIG. 2. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0031] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power supply 1106 converts AC current input from the outside into DC current and supplies power to the light source 1105. The power supply 1106 also supplies the necessary DC current to each section within the space floating image display device 1000.

[0032] The light guide 1104 guides light generated by the light source 1105 and irradiates it onto the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called a backlight for the image display unit 1102. There are various possible combinations of the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0033] The aerial operation detection sensor 1351 is a sensor that detects an operation on the floating in space image 3 by the finger of the user 230. The aerial operation detection sensor 1351 senses, for example, an area that overlaps with the entire display area of ​​the floating in space image 3. Note that the aerial operation detection sensor 1351 may only sense an area that overlaps with at least a portion of the display area of ​​the floating in space image 3.

[0034] Specific examples of the aerial operation detection sensor 1351 include a distance sensor that uses invisible light such as infrared light, an invisible laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may also be configured to detect coordinates on a two-dimensional plane by combining multiple sensors. The aerial operation detection sensor 1351 may also be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) or an image sensor.

[0035] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect touch operations, etc., made by the user with their finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0036] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and based on the sensing signal, determines whether or not the finger of the user 230 has made contact with an object in the floating in space image 3, and calculates the position (contact position) where the finger of the user 230 has made contact with the object. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array), for example. Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0037] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally as a separate entity from the space-floating image display device 1000. When provided as a separate entity from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to be able to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or a video signal transmission path.

[0038] Also, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the space-floating image display device 1000 without the aerial operation detection function is used as the main body, and only the aerial operation detection function can be added as an option. Also, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the space-floating image display device 1000 may be used. In cases such as when it is desired to more freely position the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0039] The imaging unit 1180 is a camera with an image sensor, and captures images of the space near the floating in space image 3, and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180, or by using an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in the detection process of the touch operation on the floating in space image 3 by the user 230.

[0040] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that targets a plane (intrusion detection plane) including the display surface of the spatial floating image 3 and detects whether or not an object has intruded into this intrusion detection plane, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object (e.g., a user's finger) that has not intruded into the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0041] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the captured images of the multiple imaging units 1180 and object depth information from the depth sensor. These pieces of information and various pieces of information such as the distance between the object and the intrusion detection plane are used for various display controls for the floating in space image 3.

[0042] Furthermore, without using the mid-air operation detection sensor 1351, the mid-air operation detection unit 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging unit 1180.

[0043] Furthermore, the imaging unit 1180 may capture an image of the face of the user 230 operating the space-floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not there is another person standing around or behind the user 230 operating the space-floating image 3 and peeking at the operation of the user 230 on the space-floating image 3, the imaging unit 1180 may capture an image of a range including the user 230 operating the space-floating image 3 and the surrounding area of ​​the user 230.

[0044] The operation input unit 1107 is, for example, an operation button or a light receiving unit of a remote controller, and inputs a signal for an operation different from the air operation (touch operation) by the user 230. Apart from the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may also be used by, for example, an administrator to operate the space floating image display device 1000.

[0045] The video signal input unit 1131 is connected to an external video output device and inputs video data. The audio signal input unit 1133 is connected to an external audio output device and inputs audio data. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may also output built-in operation sounds and error warning sounds.

[0046] The nonvolatile memory 1108 stores various data used by the space floating image display device 1000. The data stored in the nonvolatile memory 1108 includes, for example, data for various operations to be displayed on the space floating image 3, display icons, data and layout information for objects to be operated by user operations, etc. The memory 1109 stores image data to be displayed as the space floating image 3, data for controlling the device, etc.

[0047] The control unit 1110 controls the operation of each connected unit. The control unit 1110 may also work in cooperation with a program stored in the memory 1109 to perform calculations based on information acquired from each unit in the space floating image display device 1000. The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless interface. Various types of data such as video data, image data, and audio data are sent and received through communication via the communication unit 1132.

[0048] The storage unit 1170 is a storage device that records various types of data and information such as video data, image data, and audio data. For example, various types of information such as video data, image data, and audio data may be recorded in advance in the storage unit 1170 at the time of product shipment. Furthermore, the storage unit 1170 may record various types of information such as video data, image data, and audio data acquired from an external device, an external server, or the like via the communication unit 1132.

[0049] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflecting unit 1101. The video data, image data, etc. of display icons, objects for the user to operate, etc., which are displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0050] Layout information of display icons, objects, etc. displayed as the space floating image 3, and various metadata information related to the objects, etc. are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0051] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. For example, the video control unit 1160 controls video switching, such as which video signal is input to the video display unit 1102 between the video signal stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131.

[0052] In addition, the video control unit 1160 may generate a superimposed video signal by superimposing the video signal to be stored in the memory 1109 and the video signal input from the video signal input unit 1131, and input the superimposed video signal to the video display unit 1102, thereby performing control to form the composite video as the floating-in-space video 3.

[0053] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing to enlarge, reduce, deform, etc. the image, brightness adjustment processing to change the brightness, contrast adjustment processing to change the contrast curve of the image, and Retinex processing to decompose the image into light components and change the weighting of each component.

[0054] Furthermore, the video control unit 1160 may perform special effect video processing or the like to assist the aerial operation (touch operation) of the user 230 on the video signal input to the video display unit 1102. The special effect video processing is performed based on, for example, the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 and the image of the user 230 captured by the imaging unit 1180.

[0055] As explained above, various functions are installed in the space-floating image display device 1000. However, the space-floating image display device 1000 does not need to have all of these functions, and any configuration is acceptable as long as it has the function of forming the space-floating image 3. <Space-floating image display device 2>

[0056] 4 is a diagram showing another example of the configuration of the main components of a space-floating image display device according to one embodiment of the present invention. Image display element 11 constituting display device 1 includes light source device 13 that generates light of a specific polarization having diffusion characteristics that form a narrow angle with liquid crystal display panel 11, and is composed of a liquid crystal display panel ranging from a small one with a screen size of about 5 inches to a large one exceeding 80 inches, for example. Polarization separation member 101 such as a reflective polarizer is provided on the surface of folding mirror 22, and reflects the image light from liquid crystal display panel 11 toward retroreflective member 2. The image light of the specific polarization from display device 1 is reflected by a film (sheet 101 is adhesively attached in the figure) provided on transparent member 100 that selectively reflects image light of the specific polarization, and enters retroreflective member 2.

[0057] A λ / 4 plate 21 is provided on the light incident surface of the retroreflective member. The image light is polarized and converted from a specific polarization to another polarization by passing through the polarization separation member 101 twice. The image light then passes through the polarization separation member 101, resulting in a real image, a floating-in-space image 3, displayed on the outside of the transparent member 100. An absorptive polarizer is provided on the external light incident surface of the transparent member 100. Because the polarization separation member 101 retroreflects the image, its polarization axis becomes misaligned, causing some of the image light to reflect back to the display device 1. This light is then reflected again by the image display surface of the liquid crystal display panel 11 that constitutes the display device 1, generating ghost images and significantly degrading the image quality of the floating-in-space image. Therefore, in this embodiment, an absorptive polarizer 12 is provided on the image display surface of the display device 1, which transmits the image light and absorbs the reflected light, thereby preventing image quality degradation due to ghost images of the floating-in-space image. Furthermore, an absorptive polarizer 12 is preferably provided on the surface of the transparent member 100 to reduce image quality degradation caused by sunlight or external illumination. The polarization separation member 101 is formed from a reflective polarizing plate or a metal multilayer film that reflects a specific polarized wave.

[0058] Next, sensors 44 with time-of-fly (TOF) functionality are arranged in multiple layers as shown in Figure 5 to sense the distance and position relationship between the object and the sensors 44 in relation to the space-floating image obtained by the space-floating image display device described above. This makes it possible to sense not only the object's coordinates in the plane direction, but also its coordinates in the depth direction and the object's movement direction and speed. To read two-dimensional distance and position, multiple combinations of ultraviolet light emitters and light receivers are arranged in a straight line, and light from the light emitter is irradiated onto the object, and the reflected light is received by the light receiver. The distance to the object is determined by the product of the difference between the time of light emission and the time of light reception and the speed of light. Furthermore, coordinates on the plane can be read from the coordinates at the point where the difference between the time of light emission and the time of light reception is smallest using multiple light emitters and light receivers. As described above, three-dimensional coordinate information can also be obtained by combining the coordinates of the object on the plane (two-dimensional) and multiple sensors described above.

[0059] Furthermore, a method for obtaining a three-dimensional space-floating image using the above-mentioned space-floating image display device will be described with reference to FIG. 6. FIG. 6 is an explanatory diagram of the principle of three-dimensional image display used in the space-floating image display device. A horizontal lenticular lens is arranged to match the pixels of the image display screen of the liquid crystal display panel 11 of the display device 1 shown in FIG. 4. As a result, to display motion parallax from three directions, P1, P2, and P3 in the horizontal direction of the screen as shown in FIG. 6, images from the three directions are grouped into blocks of three pixels, and image information from three directions is displayed for each pixel. The light emission direction is controlled by the action of the corresponding lenticular lens (shown by vertical lines in FIG. 6) to separate and emit the light in three directions. As a result, a three-parallax stereoscopic image can be displayed. <Reflective polarizing plate>

[0060] In the space-floating image display device of this embodiment, the polarization separation member 101 is used to improve the contrast performance, which determines the image quality, compared to a general half mirror. The characteristics of a reflective polarizer will be described as an example of the polarization separation member 101 of this embodiment. FIG. 7 is an explanatory diagram of a measurement system used to evaluate the characteristics of a reflective polarizer. The transmission and reflection characteristics of the reflective polarizer of FIG. 7 relative to the angle of incidence of light from a direction perpendicular to the polarization axis are shown in FIGS. 8 and 9, respectively, as V-AOI. Similarly, the transmission and reflection characteristics of the reflective polarizer relative to the angle of incidence of light from a direction horizontal to the polarization axis are shown in FIGS. 10 and 11, respectively, as H-AOI.

[0061] As shown in Figures 8 and 9, the reflective polarizer with a grid structure exhibits reduced characteristics for light coming from a direction perpendicular to the polarization axis. For this reason, specifications aligned with the polarization axis are desirable, and the light source of this embodiment, which can emit the image light from the liquid crystal display panel at a narrow angle, is an ideal light source. Similarly, the horizontal characteristics also deteriorate for light coming from an oblique angle. Taking these characteristics into consideration, the following describes an example configuration of this embodiment, in which a light source capable of emitting the image light from the liquid crystal display panel at a narrower angle is used as the backlight for the liquid crystal display panel. This makes it possible to provide high-contrast floating images. <Display device>

[0062] Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes its light source. In Fig. 12, the light source device 13 is shown together with the liquid crystal display panel as an exploded perspective view.

[0063] As shown by arrow 30 in FIG. 12, this liquid crystal display panel (image display element 11) receives light from light source device 13, which serves as a backlight device, and receives an illumination light beam with narrow-angle diffusion characteristics, i.e., a laser-like characteristic with strong directivity (linearity) and a polarization plane aligned in one direction. The image light is modulated according to the input video signal, and is reflected by retroreflective member 2 and transmitted through transparent member 100 to form a real, floating image (see FIG. 1). FIG. 12 also shows a configuration including liquid crystal display panel 11, which constitutes display device 1, a light direction conversion panel 54 that controls the directional characteristics of the light beam emitted from light source device 13, and, if necessary, a narrow-angle diffusion plate (not shown). Polarizing plates are provided on both sides of liquid crystal display panel 11, and image light of a specific polarization is emitted with its intensity modulated according to the video signal (see arrow 30 in FIG. 12). As a result, the desired image is projected as highly directional (straight-line) light of a specific polarization via light direction conversion panel 54 toward retroreflective member 2, and after being reflected by retroreflective member 2, it is transmitted toward the eyes of an observer outside the store (space) to form space-floating image 3. Note that a protective cover 50 (see Figures 13 and 14) may be provided on the surface of light direction conversion panel 54 described above.

[0064] In this embodiment, in order to improve the utilization efficiency of the luminous flux 30 emitted from the light source device 13 and significantly reduce power consumption, in a display device 1 including the light source device 13 and the liquid crystal display panel 11, light from the light source device 13 (see arrow 30 in FIG. 12 ) is projected toward the retroreflective member 2. After being reflected by the retroreflective member 2, a transparent sheet (not shown) attached to the surface of the transparent member 100 (such as the window glass 105) can be used to control the directivity so that a floating image is formed at a desired position. Specifically, this transparent sheet controls the imaging position of the floating image while providing high directivity using optical components such as a Fresnel lens or a linear Fresnel lens. This allows the image light from the display device 1 to efficiently reach an observer outside the display window 105 (e.g., on a sidewalk) with high directivity (straightness) like laser light. As a result, it is possible to display high-quality floating images at high resolution and significantly reduce the power consumption of the display device 1 including the LED elements 201 of the light source device 13. <Display device example 1>

[0065] FIG. 13 shows an example of a specific configuration of the display device 1. In FIG. 13, a liquid crystal display panel 11 and a light direction conversion panel 54 are arranged on the light source device 13 of FIG. 12. This light source device 13 is configured on a case shown in FIG. 12, formed of, for example, plastic, and contains LED elements 201 and a light guide 203. As shown in FIG. 12 and other figures, the end surface of the light guide 203 is provided with a lens shape whose cross-sectional area gradually increases toward the light receiving section in order to convert the divergent light from each LED element 201 into a substantially parallel beam. The lens shape has an effect of gradually reducing the divergence angle by multiple total reflections as the light propagates inside. The liquid crystal display panel 11 constituting the display device 1 is attached to its upper surface. Furthermore, an LED (Light Emitting Diode) element 201, which is a semiconductor light source, and an LED board 202, which has its control circuit mounted thereon, are attached to one side of the case of the light source device 13 (the left end face in this example), and a heat sink, which is a component for cooling the heat generated by the LED element and the control circuit, may be attached to the outer surface of the LED board 202.

[0066] The liquid crystal display panel frame (not shown) is attached to the top surface of the case of the light source device 13. The liquid crystal display panel 11 is attached to the frame, and an FPC (Flexible Printed Circuit) (not shown) electrically connected to the liquid crystal display panel is also attached. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light in conjunction with the LED elements 201, which are solid-state light sources, based on a control signal from a control circuit (not shown) constituting the electronic device. The generated image light has a narrow diffusion angle and contains only specific polarization components, resulting in a novel image display device similar to a surface-emitting laser image source driven by a video signal. Currently, it is technically and safety-wise impossible to obtain a laser beam of the same size as the image obtained by the display device 1 described above using a laser device. Therefore, in this embodiment, light similar to the surface-emitting laser image light described above is obtained from a beam of light from a general light source, such as an LED element.

[0067] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 13 as well as FIG.

[0068] 13 and 14 are cross-sectional views, and only one of the LED elements 201 constituting the light source is shown. This LED element is converted into a nearly collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion of the light guide end surface and the LED element are attached while maintaining a predetermined positional relationship. Each light guide 203 is formed of a translucent resin such as acrylic. The LED light-receiving surface at the end of the light guide has a convex outer surface of a cone obtained by rotating a parabolic cross section, and at its apex, a concave portion with a convex portion (i.e., a convex lens surface) formed in its central portion. The central portion of the flat portion has a convex lens surface (alternatively, a concave lens surface) that protrudes outward (not shown). The outer shape of the light receiving part of the light guide to which the LED element 201 is attached is a paraboloid that forms a conical outer surface, and is set within a range of angles that allows the light emitted from the LED element in the peripheral direction to be totally reflected within it, or a reflective surface is formed.

[0069] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed to the LED collimator (light-receiving end surface 203a) so that the LED elements 201 on the surface are positioned in the center of the recessed portion described above.

[0070] According to this configuration, the shape of the light-receiving end surface 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as approximately parallel light, thereby improving the efficiency of use of the generated light.

[0071] As described above, light source device 13 is configured by attaching a light source unit in which a plurality of LED elements 201 serving as light sources are arranged to light-receiving end surface 203a, which is a light-receiving section provided on the end surface of light guide 203, and divergent light beams from the LED elements are converted into approximately parallel light by the lens shape of light-receiving end surface 203a of the light guide end surface, which is then guided inside light guide 203 (in a direction parallel to the drawing) as shown by the arrow, and emitted by light beam direction conversion means 204 toward liquid crystal display panel 11, which is arranged approximately parallel to the light guide (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of this light beam direction conversion means depending on the shape inside or on the surface of the light guide. The light beam direction conversion means 204 described above emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 arranged substantially parallel to the light guide (in a direction perpendicular to the front of the drawing) by changing the shape of the surface of the light guide or by providing a portion with a different refractive index inside the light guide. In this case, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is positioned at the same position as the diagonal dimension of the screen, if the relative brightness ratio between the center of the screen and the periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

[0072] 13 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED element 201 described above. In Fig. 13, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface of light source device 13 is attached liquid crystal display panel 11 that has polarizing plates on the light source light entrance surface and the image light exit surface.

[0073] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one polarized wave (e.g., P wave) 212 of the natural light beam 210 emitted from the LED element 201, and the reflected light beam is reflected by a reflective sheet 205 provided on one surface (the lower surface in the figure) of the light guide 203, and directed again toward the liquid crystal display panel 52. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, thereby converting the reflected light beam from P polarized light to S polarized light, thereby improving the efficiency of use of the light source light as image light. The image light beam, whose light intensity is modulated by the image signal in the liquid crystal display panel 11 (arrow 213 in Figure 13), enters the retroreflective member 2, and after reflection, as shown in Figure 1, passes through the window glass 105 to produce a real image, a floating image in space, inside or outside the store (space).

[0074] 13, Fig. 14 is a cross-sectional layout diagram illustrating the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED elements 201. Light source device 13 is also similarly composed of light guide 203 formed of, for example, plastic and having light beam direction conversion means 204 on its surface or inside, LED elements 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface thereof is attached liquid crystal display panel 11 as an image display element, which has polarizing plates on the light source light entrance surface and the image light exit surface.

[0075] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one polarized wave (for example, S wave) 211 of the natural light beam 210 emitted from the LED light source 201, which is reflected by a reflective sheet 205 provided on one surface (the lower surface in the figure) of the light guide 203, and then directed again toward the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, thereby converting the reflected light beam from S polarized light to P polarized light, thereby improving the efficiency of use of the light source light as image light. The image light beam whose light intensity is modulated by the image signal on the liquid crystal display panel 11 (arrow 214 in Figure 14) enters the retroreflective member 2, and after reflection, as shown in Figure 1, passes through the window glass 105 to produce a real image, which is a floating image in space, inside or outside the store (space).

[0076] In the light source devices shown in Figures 13 and 14, in addition to the action of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, the reflective polarizer reflects the polarized light component on one side, so the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizer multiplied by the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In fact, experiments have confirmed that the contrast performance of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those of self-luminous organic EL displays are obtained. <Display device example 2>

[0077] Fig. 15 shows another example of a specific configuration of the display device 1. The light source device 13 in Fig. 15 is similar to the light source device in Fig. 17 and the like. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, and the like in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. Also, an LED substrate 102 on which LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and their control circuits are mounted are attached to one side of the case of the light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer surface of the LED substrate 102 (see also Figs. 17, 18, and the like).

[0078] The liquid crystal display panel frame attached to the top surface of the case is configured to have attached thereto a liquid crystal display panel 11 attached to the frame, and further to have attached thereto an FPC (Flexible Printed Circuits) 403 (see FIG. 7) electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements 14a and 14b, which are solid-state light sources, based on a control signal from a control circuit (not shown here) that constitutes the electronic device. <Display device example 3>

[0079] Next, another example of the specific configuration of the display device 1 will be described with reference to Fig. 16. The light source device of this display device 1 converts a divergent luminous flux of natural light (a mixture of P-polarized and S-polarized waves) emitted from an LED into a substantially parallel luminous flux by an LED collimator 18, and reflects it toward the liquid crystal display panel 11 by a reflective light guide 304. The reflected light is incident on a wave plate and a reflective polarizer 49 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (for example, S-polarized wave) is reflected by the reflective polarizer, has its phase converted by the wave plate, returns to the reflecting surface, passes through the retardation plate again, and is converted into a polarized wave (for example, P-polarized wave) that transmits through the reflective polarizer.

[0080] As a result, the natural light from the LEDs is aligned to a specific polarization (e.g., P polarization), enters the liquid crystal display panel 11, and is brightness-modulated in accordance with the video signal to display an image on the panel surface. Similar to the above example, multiple LEDs constituting the light source are shown (however, because FIG. 16 shows a vertical cross section, only one LED is shown), which are attached at predetermined positions relative to the LED collimator 18. Each LED collimator 18 is formed of a translucent resin such as acrylic or glass. The LED collimator 18 has a convex cone-shaped outer surface obtained by rotating a parabolic cross section, and its apex has a concave portion with a convex portion (i.e., a convex lens surface) formed in its central portion. The center of its flat portion also has a convex lens surface protruding outward (or a concave lens surface recessed inward). The parabolic surface forming the conical outer surface of the LED collimator 18 is set within an angle range that allows the light emitted from the LEDs in the peripheral direction to be totally reflected within it, or a reflective surface is formed thereon.

[0081] The above configuration is the same as that of the light source device of the image display device shown in Figures 17 and 18, etc. Furthermore, the light converted into approximately parallel light by the LED collimator 15 shown in Figure 16 is reflected by the reflective light guide 304 and transmits light of a specific polarization due to the action of the reflective polarizer 49, while the other reflected light of the polarization passes through the light guide 304 again and is reflected by the reflector 271 provided on the other surface of the light guide that is not in contact with the liquid crystal display panel 11. At this time, the polarization is converted by passing twice through the retardation plate (λ / 4 plate) 270 arranged between the reflector 271 and the liquid crystal display panel 11, and the light passes through the light guide 304 again and the reflective polarizer 49 provided on the opposite surface, and is made incident on the liquid crystal display panel 11 with its polarization direction aligned. As a result, all of the light from the light source can be used, thereby doubling the light utilization efficiency.

[0082] In conventional TV sets, the light emitted from the LCD panel has similar diffusion characteristics in both the horizontal direction of the screen (shown on the X-axis in Figure 22(a)) and the vertical direction of the screen (shown on the Y-axis in Figure 22(b)). In contrast, the diffusion characteristics of the light beam emitted from the LCD panel of this embodiment are, for example, as shown in Example 1 of Figure 22, reduced to 1 / 5 of the conventional 62-degree viewing angle by setting the viewing angle at which the luminance is 50% of that when viewed from the front (angle of 0 degrees) to 13 degrees. Similarly, the vertical viewing angle is made uneven between the top and bottom, and the reflection angle and the area of ​​the reflective surface of the reflective light guide are optimized to keep the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness being more than 50 times higher.

[0083] Furthermore, assuming the viewing angle characteristics shown in Example 2 of Figure 22, the viewing angle at which brightness is 50% of that at a frontal view (angle of 0 degrees) is set to 5 degrees, reducing the brightness to 1 / 12 of the conventional 62 degrees. Similarly, the vertical viewing angle is kept equal from top to bottom, and the reflection angle and reflective surface area of ​​the reflective light guide are optimized to reduce the viewing angle to approximately 1 / 12 of the conventional viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness being more than 100 times higher. As described above, by setting the viewing angle to a narrow angle, the amount of light flux directed toward the monitoring direction can be concentrated, significantly improving light utilization efficiency. As a result, even when using a conventional TV LCD panel, significant brightness improvement can be achieved with similar power consumption by controlling the light diffusion characteristics of the light source device, making it possible to create a video display device suitable for information display systems aimed at bright outdoor environments.

[0084] When using a large LCD display panel, the overall brightness of the screen can be improved by directing the light from the periphery of the screen inward so that it is directed toward the observer when the observer is facing the center of the screen. Figure 20 shows the convergence angle of the long and short sides of the panel when the observer's distance from the panel, L, and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in portrait orientation, the convergence angle can be set to match the short side. For example, when using a 22" panel in portrait orientation and the monitoring distance is 0.8m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the observer.

[0085] Similarly, when monitoring with a 15" panel in portrait orientation and the monitoring distance is 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the monitor. As mentioned above, depending on the size of the LCD panel and whether it is used portrait or landscape, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the monitor who is in the optimum position to monitor the center of the screen.

[0086] The basic configuration is as shown in Figure 16, in which a light source device emits a light beam with a narrow angle of directivity onto a liquid crystal display panel 11, and the brightness is modulated according to the video signal.The video information displayed on the screen of the liquid crystal display panel 11 is then reflected by a retroreflective member, and the resulting spatially floating image is displayed indoors or outdoors via a transparent member 100. <Light source device example 1>

[0087] Next, the configuration of the optical system, such as the light source device housed in the case, will be described in detail with reference to FIG. 17 as well as FIGS. 18(a) and (b).

[0088] 17 and 18 show LEDs 14a and 14b constituting the light source, which are attached at predetermined positions relative to an LED collimator 15. Each of the LED collimators 15 is made of a translucent resin, such as acrylic. As shown in FIG. 18(b), the LED collimator 15 has a cone-shaped outer peripheral surface 156 obtained by rotating a parabolic cross section, and a recess 153 at its apex, with a convex portion (i.e., a convex lens surface) 157 formed in its central portion. The central portion of the flat portion has a convex lens surface 154 protruding outward (or may be a concave lens surface recessed inward). The parabolic surface 156 forming the cone-shaped outer peripheral surface of the LED collimator 15 is set within an angle range that allows total reflection of the light emitted from the LEDs 14a and 14b in the peripheral direction, or a reflective surface is formed thereon.

[0089] The LEDs 14a and 14b are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 102. The LED substrate 102 is arranged and fixed to the LED collimator 15 so that the LEDs 14a and 14b on the surface are located at the center of the recess 153.

[0090] According to this configuration, the LED collimator 15 described above condenses light emitted from the LED 14a or 14b, particularly light emitted upward from the central portion (toward the right in the figure), into parallel light by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. Light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the conical outer surface of the LED collimator 15, and is similarly condensed into parallel light. In other words, the LED collimator 15, which has a convex lens in its center and a parabolic surface formed on its periphery, can extract almost all of the light generated by the LED 14a or 14b as parallel light, thereby improving the utilization efficiency of the generated light.

[0091] 18, the polarization conversion element 21 is provided on the light exit side of the LED collimator 15. As is clear from FIG. 18, the polarization conversion element 21 is configured by combining a columnar translucent member having a parallelogram cross section (hereinafter referred to as a parallelogram prism) with a columnar translucent member having a triangular cross section (hereinafter referred to as a triangular prism), and arranging a plurality of these in an array parallel to a plane perpendicular to the optical axis of the collimated light from the LED collimator 15. Furthermore, a polarization beam splitter (hereinafter referred to as a "PBS film") 211 and a reflective film 212 are alternately provided at the interface between adjacent translucent members arranged in the array, and a λ / 2 phase plate 213 is provided on the exit surface from which light incident on the polarization conversion element 21 and transmitted through the PBS film 211 exits.

[0092] 18(a) is further provided on the exit surface of this polarization conversion element 21. That is, the light emitted from LED 14a or 14b is converted into parallel light by the action of LED collimator 15, enters synthesizing diffusion block 16, is diffused by texture 161 on the exit side, and then reaches light guide 17.

[0093] The light guide 17 is a rod-shaped member made of a translucent resin such as acrylic and having an approximately triangular cross section (see Figure 18(b)). As is clear from Figure 17, the light guide 17 comprises a light incident portion (surface) 171 that faces the exit surface of the synthetic diffusion block 16 via a first diffuser plate 18a, a light guide light reflecting portion (surface) 172 that forms an inclined surface, and a light guide light exit portion (surface) 173 that faces the liquid crystal display panel 11, which is a liquid crystal display element, via a second diffuser plate 18b.

[0094] 17, which is a partially enlarged view, a number of reflective surfaces 172a and connecting surfaces 172b are alternately formed in a sawtooth pattern on the light guide light reflecting portion (surface) 172 of this light guide 17. The reflective surfaces 172a (line segments sloping upward to the right in the figure) form angles αn (n: natural number, for example, 1 to 130 in this example) with respect to the horizontal plane indicated by the dashed dotted line in the figure, and as an example, αn is set to 43 degrees or less (but 0 degree or more).

[0095] Light guide light incident portion (surface) 171 is formed in a curved convex shape inclined toward the light source. Accordingly, parallel light from the exit surface of synthetic diffusion block 16 is diffused and incident via first diffuser plate 18a, and as is clear from the figure, is bent (deflected) slightly upward by light guide light incident portion (surface) 171 before reaching light guide light reflecting portion (surface) 172, where it is reflected and reaches liquid crystal display panel 11 provided on the exit surface at the top of the figure.

[0096] The display device 1 described above in detail can further improve light utilization efficiency and uniform illumination characteristics, and can also be manufactured compactly and at low cost, including a modularized S-polarized light source device. In the above description, the polarization conversion element 21 is described as being attached after the LED collimator 15, but the present invention is not limited to this, and similar effects can be obtained by providing the polarization conversion element 21 in the optical path leading to the liquid crystal display panel 11.

[0097] The light guide light reflection portion (surface) 172 has a large number of reflective surfaces 172a and connecting surfaces 172b formed alternately in a sawtooth pattern, and the illumination light beam is totally reflected by each reflective surface 172a and directed upward, and then enters light direction conversion panel 54, which is provided with a narrow-angle diffuser plate on light guide light exit portion (surface) 173 to control the directivity as a substantially parallel diffused light beam, and then enters liquid crystal display panel 11 from an oblique direction. In this embodiment, light direction conversion panel 54 is provided between light guide exit portion (surface) 173 and liquid crystal display panel 11, but the same effect can be obtained by providing light direction conversion panel 54 on the exit surface of liquid crystal display panel 11. <Light source device example 2>

[0098] FIG. 19 shows another example of the configuration of an optical system such as the light source device 13. Similar to the example shown in FIG. 18, a plurality of (two in this example) LEDs 14a and 14b constituting the light source are shown, and these are attached at predetermined positions relative to the LED collimator 15. Each of the LED collimators 15 is formed of a translucent resin such as acrylic. Similar to the example shown in FIG. 18, the LED collimator 15 has a convex convex outer surface 156 obtained by rotating a parabolic cross section, and a recess 153 at its apex with a convex portion (i.e., a convex lens surface) 157 formed in its central portion. The central portion of the flat portion has a convex lens surface 154 protruding outward (or may be a concave lens surface recessed inward). The parabolic surface 156 forming the conical outer surface of the LED collimator 15 is set within an angle range that allows the light emitted from the LED 14a in the peripheral direction to be totally reflected therein, or a reflective surface is formed thereon.

[0099] The LEDs 14a and 14b are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 102. The LED substrate 102 is arranged and fixed to the LED collimator 15 so that the LEDs 14a and 14b on the surface are located at the center of the recess 153.

[0100] According to this configuration, the LED collimator 15 described above condenses light emitted from the LED 14a or 14b, particularly light emitted upward from the central portion (toward the right in the figure), into parallel light by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. Light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the conical outer surface of the LED collimator 15, and is similarly condensed into parallel light. In other words, the LED collimator 15, which has a convex lens in its center and a parabolic surface formed on its periphery, can extract almost all of the light generated by the LED 14a or 14b as parallel light, thereby improving the utilization efficiency of the generated light.

[0101] A light guide 170 is provided on the light emission side of the LED collimator 15 via a first diffusion plate 18a. The light guide 170 is a rod-shaped member made of a translucent resin such as acrylic and having a substantially triangular cross section (see FIG. 19(a)), and as is clear from FIG. 19(a), the light guide 170 includes a light incident portion (surface) 171 that faces the emission surface of the diffusion block 16 via the first diffusion plate 18a, a light guide light reflecting portion (surface) 172 that forms an inclined surface, and a light guide light emission portion (surface) 173 that faces the liquid crystal display panel 11, which is a liquid crystal display element, via a reflective polarizing plate 200.

[0102] If this reflective polarizer 200 is selected to have the property of reflecting P-polarized light (and transmitting S-polarized light), for example, it will reflect the P-polarized light from the natural light emitted from the LED light source, pass through the λ / 4 plate 202 provided in the light guide light reflecting section 172 shown in Figure 19(b), reflect at the reflecting surface 201, and pass through the λ / 4 plate 202 again to be converted into S-polarized light, and all the light beams incident on the liquid crystal display panel 11 will be unified into S-polarized light.

[0103] Similarly, if a material having the property of reflecting S-polarized light (and transmitting P-polarized light) is selected as reflective polarizer 200, the S-polarized light in the natural light emitted from the LED light source will be reflected, pass through λ / 4 plate 202 provided on light guide light reflecting section 172 shown in Figure 19(b), be reflected by reflective surface 201, and be converted into P-polarized light by passing through λ / 4 plate 202 again, so that all light beams incident on liquid crystal display panel 52 are unified into P-polarized light. Polarization conversion can also be achieved with the configuration described above. <Light source device example 3>

[0104] Another example of the configuration of an optical system such as a light source device will be described with reference to Fig. 16. In the third example, as shown in Fig. 16, a divergent beam of natural light (a mixture of P-polarized and S-polarized light) from an LED 102 is converted into a substantially parallel beam by a collimator lens 18 and reflected toward a liquid crystal display panel 11 by a reflective light guide 304. The reflected light is incident on a reflective polarizer 206 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (e.g., S-polarized wave) is reflected by the reflective polarizer 206, passes through a surface connecting the reflective surfaces of the light guide 304, is reflected by a reflector 271 arranged facing the opposite surface of the light guide 304, and is polarized and converted by passing twice through a phase plate (λ / 4 wave plate) 270. The light passes through the light guide and the reflective polarizer, enters the liquid crystal display panel 11, and is modulated into image light. In this case, by combining the specific polarized wave with the polarization-converted polarization plane, the light utilization efficiency becomes twice as high as usual, and the polarization degree (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, so that by using the light source device of this embodiment, the contrast ratio of the information display system is significantly improved.

[0105] As a result, the natural light from the LEDs is aligned to a specific polarization (e.g., P polarization). Similar to the above-described example, multiple LEDs are provided as a light source (however, because FIG. 16 shows a longitudinal section, only one LED is shown), and these are attached to a predetermined position relative to the LED collimator 18. Each LED collimator 18 is made of a translucent resin such as acrylic or glass. The LED collimator 18 has a cone-shaped outer surface obtained by rotating a parabolic cross section, and a concave portion with a convex portion (i.e., a convex lens surface) formed in the center at its apex. The center of the flat portion has a convex lens surface protruding outward (or may be a concave lens surface recessed inward). The parabolic surface forming the cone-shaped outer surface of the LED collimator 18 is set within an angle range that allows the light emitted from the LEDs 18 toward the periphery to be totally reflected therein, or a reflective surface is formed therein.

[0106] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 102. The LED substrate 102 is arranged and fixed to the LED collimator 18 so that the LEDs on the surface are positioned in the center of the recesses.

[0107] With this configuration, the light emitted from the LEDs by the LED collimator 18, particularly the light emitted from the central portion, is collected and converted into parallel light by the two convex lens surfaces that form the outer shape of the LED collimator 18. Meanwhile, the light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the conical shape of the LED collimator 18, and is similarly collected and converted into parallel light. In other words, the LED collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, can extract almost all of the light generated by the LEDs as parallel light, thereby improving the utilization efficiency of the generated light. <Light source device example 4>

[0108] Another example of the configuration of an optical system such as a light source device will be described with reference to FIG. 25 . Two optical sheets 207 that convert the diffusion characteristics in the vertical and horizontal directions (front and back directions in the drawing, not shown) are used on the light output side of the LED collimator 18, and the light from the LED collimator 18 is incident between the two optical sheets 207 (diffusion sheets). When this optical sheet 207 is configured with a single sheet, the vertical and horizontal diffusion characteristics are controlled by the fine shapes on the front and back surfaces. Alternatively, multiple diffusion sheets may be used to share the function. The front and back shapes of the optical sheet 207 can be optimized to match the diffusion angle of the light from the LED collimator 18 in the vertical direction of the screen to the width of the vertical plane of the reflective surface of the diffusion sheet, and the number of LEDs and the divergence angle from the LED substrate (optical element) 102 can be used as design parameters to ensure a uniform surface density of the light beam emitted from the liquid crystal display panel 11 in the horizontal direction. In other words, the diffusion characteristics are controlled by the surface shapes of multiple diffusion sheets instead of light guides. In this example, polarization conversion is performed in the same manner as in Example 3 of the light source device described above. Alternatively, a polarization conversion element 21 may be provided between the LED collimator 18 and the diffusion film 207, and the light from the light source may be made incident on the diffusion sheet 207 after polarization conversion.

[0109] If the above-mentioned reflective polarizer 206 is selected to have the property of reflecting S-polarized light (and transmitting P-polarized light), it will reflect the S-polarized light in the natural light emitted from the LED light source, pass through retardation plate 270 shown in Fig. 25, be reflected by reflective surface 271, and pass through retardation plate 270 again to be converted into P-polarized light, which then enters liquid crystal display panel 11. The optimum value for the thickness of this retardation plate must be selected depending on the angle of incidence of the light beam on the retardation plate, and the optimum value is in the range of λ / 16 to λ / 4. <Lenticular lens>

[0110] In order to control the diffusion distribution of the image light from the liquid crystal display panel 11, it is possible to control the emission characteristics in one direction by optimizing the lens shape by providing a lenticular lens between the light source device 13 and the liquid crystal display panel 11 or on the surface of the liquid crystal display panel 11. Furthermore, by arranging a microlens array in a matrix, it is possible to control the emission characteristics of the image light beam from the display device 1 in the X-axis and Y-axis directions, resulting in an image display device with desired diffusion characteristics.

[0111] The effect of the lenticular lens will now be described. By optimizing the lens shape, the lenticular lens can efficiently obtain a spatially floating image by transmitting or reflecting light emitted from the above-described display device 1 through the transparent member 100. That is, by combining two lenticular lenses or by providing a sheet that controls the diffusion characteristics by arranging a microlens array in a matrix, the brightness (relative brightness) of the image light from the display device 1 can be controlled in the X-axis and Y-axis directions according to its reflection angle (vertical direction is 0 degrees). In this embodiment, such a lenticular lens makes the vertical brightness characteristic steeper than conventional ones as shown in Figure 22(b), and furthermore, by changing the balance of the directional characteristics in the up and down directions (positive and negative directions of the Y axis), the brightness (relative brightness) of the light due to reflection and diffusion is increased, so that the image light has a narrow diffusion angle (high straightness) and contains only specific polarization components, like the image light from a surface-emitting laser image source, and ghost images that were generated by the retroreflective material when using an image display device using conventional technology are suppressed, and it is possible to control the spatial floating image due to retroreflection so that it reaches the observer's eyes efficiently.

[0112] Furthermore, by using the light source device described above, it is possible to realize an image display device that emits light of a specific polarization that emits an image light beam that is nearly parallel to a specific direction by providing a directional characteristic that is significantly narrower in both the X-axis direction and the Y-axis direction than the diffusion characteristic of light emitted from a typical liquid crystal display panel shown in Figure 22 (a) and (b) (labeled as conventional in the figure).

[0113] FIG. 21 shows an example of the characteristics of the lenticular lens employed in this embodiment. This example particularly shows the characteristics in the X direction (vertical direction), with characteristic O indicating a luminance characteristic that is symmetrical vertically, with the peak of the light emission direction occurring at an angle of approximately 30 degrees upward from the vertical direction (0 degrees). Characteristics A and B in FIG. 21 also show examples of characteristics in which the image light above the peak luminance is condensed near 30 degrees, thereby increasing the luminance (relative luminance). Therefore, in characteristics A and B, the luminance (relative luminance) of light drops sharply at angles exceeding 30 degrees compared to characteristic O.

[0114] That is, with the optical system including the lenticular lens described above, when the image light beam from the display device 1 is incident on the retroreflective member 2, the light source device 13 can control the exit angle and viewing angle of the image light aligned at a narrow angle, significantly improving the flexibility of installation of the retroreflective sheet (retroreflective member 2). As a result, the flexibility of the image position of the space-floating image, which is reflected or transmitted through the transparent member 100 and focused at the desired position, can be significantly improved. As a result, it is possible to efficiently deliver light with a narrow diffusion angle (high linearity) and only specific polarization components to the eyes of an observer indoors or outdoors. This allows the observer to accurately recognize the image light and obtain information even if the intensity (brightness) of the image light from the image display device is reduced. In other words, by reducing the output power of the image display device, it is possible to realize a space-floating image display device with low power consumption. <Touch operation assistance function>

[0115] Next, the assist function for touch operation for the user will be described. First, the touch operation when the assist function is not provided will be described. Note that here, the case where the user selects and touches one of two buttons (objects) will be described as an example, but the following content can also be suitably applied to, for example, ATMs at banks, ticket vending machines at stations, digital signage, etc.

[0116] FIG. 26 is a diagram illustrating a display example and touch operation of the space-floating image display device 1000. The space-floating image 3 shown in FIG. 26 includes a first button BUT1 displayed as "YES" and a second button BUT2 displayed as "NO." The user moves his / her finger 210 toward the space-floating image 3 and touches the first button BUT1 or the second button BUT2 to select "YES" or "NO." In the examples of FIG. 26 and FIGS. 27 to 29, the first button BUT1 and the second button BUT2 are displayed in different colors. Here, the areas of the space-floating image 3 other than the first button BUT1 and the second button BUT2 may be transparent and not display any images. In that case, the range of the virtual shadow effect, which will be described later, will be limited to the areas of the displayed buttons (the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2). Therefore, in the following explanation, as a more suitable example, in the area of ​​the floating image 3 other than the first button BUT1 and the second button BUT2, an image of a different color or brightness than that of the first button BUT1 and the second button BUT2 is displayed in a wider area including the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2.

[0117] In a general image display device with a touch panel that is not a space-floating image display device, the button selected by the user is configured as an image button displayed on the touch panel surface. Therefore, the user can recognize the sense of distance between the object (e.g., a button) displayed on the touch panel surface and their finger by visually checking the touch panel surface. However, in a space-floating image display device, since the space-floating image 3 is floating in the air, it may not be easy for the user to recognize the depth of the space-floating image 3. Therefore, when performing a touch operation on the space-floating image 3, it may not be easy for the user to recognize the sense of distance between the button displayed on the space-floating image 3 and their finger. Furthermore, in a general image display device with a touch panel that is not a space-floating image display device, the user can easily determine whether or not they have touched a button by the feeling they get when they touch it. However, when performing a touch operation on the space-floating image 3, there is no feeling when touching an object (e.g., a button), so the user may not be able to determine whether or not they have touched the object. Taking the above situation into consideration, this embodiment is provided with a function to assist the user in touch operations.

[0118] In the following description, processing based on the position of the user's finger will be explained, and a specific method for detecting the position of the user's finger will be described later. <<Touch Operation Assistance Using Virtual Shadows (1)>>

[0119] 27 to 29 are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. In the examples of FIGS. 27 to 29, the user touches the first button BUT1 to select "YES." The space-floating image display device 1000 of this embodiment assists the user's touch operation by displaying a virtual shadow on the display image of the space-floating image 3. Here, "displaying a virtual shadow on the display image of the space-floating image 3" refers to an image display process in which the brightness of the image signal is reduced for a portion of the image displayed as the space-floating image 3 in a shape resembling a finger, thereby making it appear as if a shadow is being projected on the image. Specifically, this process can be performed by calculations of the image control unit 1160 or the control unit 1110. In the virtual shadow display process, the brightness of the image signal for a portion of the shape resembling a finger may be completely set to zero. However, rather than completely setting the brightness of the image signal for a portion of the shape resembling a finger to zero, it is preferable to display an image with reduced brightness in that portion of the shape resembling a finger, because this is perceived as a more natural shadow. In this case, in the process of displaying the virtual shadow, not only the luminance of the video signal may be reduced for a part of the region of the shape resembling a finger, but also the saturation of the video signal may be reduced.

[0120] The floating-in-space image 3 exists in the air where there is no physical contact surface, and in a normal environment, the shadow of a finger is normally projected. However, according to the virtual shadow display processing of this embodiment, even in the air where the shadow of a finger is not normally projected, it is possible to make it appear as if a shadow exists in the floating-in-space image 3, thereby improving the user's perception of the depth of the floating-in-space image 3 and improving the sense of reality of the floating-in-space image 3.

[0121] Fig. 27 shows the state at a first point in time when the user attempts to touch the first button BUT1 on the display surface 3a of the space-floating image 3 with the finger 210, Fig. 28 shows the state at a second point in time when the finger 210 is closer to the space-floating image 3 than in Fig. 27, and Fig. 29 shows the state at a third point in time when the finger 210 has touched the first button BUT1 on the display surface 3a of the space-floating image 3. Also, (A) of Fig. 27 to Fig. 29 shows the state when the display surface 3a of the space-floating image 3 is viewed from the front (normal direction of the display surface 3a), and (B) of Fig. 27 to Fig. 29 shows the state when the display surface 3a of the space-floating image 3 is viewed from the side (direction parallel to the display surface 3a). 27 to 29, the x direction is the horizontal direction on the display surface 3a of the space-floating image 3, the y direction is the direction perpendicular to the x axis within the display surface 3a of the space-floating image 3, and the z direction is the normal direction of the display surface 3a of the space-floating image 3 (the height direction relative to the display surface 3a). In the explanatory diagrams of FIGS. 27 to 33, the space-floating image 3 is illustrated as having a thickness in the depth direction for ease of explanation, but in reality, if the image display surface of the display device 1 is flat, the space-floating image 3 is also flat and has no thickness in the depth direction. In this case, the space-floating image 3 and the display surface 3a are on the same plane. In the description of this embodiment, the display surface 3a refers to the surface on which the space-floating image 3 can be displayed, and the space-floating image 3 refers to the portion on which the space-floating image is actually displayed.

[0122] 27, 28, and 29, the detection process of the finger 210 is performed using, for example, an imaged image generated by the imaging unit 1180 and a sensing signal of the aerial operation detection sensor 1351. In the detection process of the finger 210, for example, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the space floating image 3, the height position (z coordinate) of the tip of the finger 210 relative to the display surface 3a, etc. are detected. Here, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the space floating image 3 is the position coordinate on the display surface 3a of the intersection of a perpendicular line from the tip of the finger 210 to the display surface 3a of the space floating image 3. Note that the height position of the tip of the finger 210 relative to the display surface 3a is also depth information that represents the depth of the finger 210 relative to the display surface 3a. Note that the arrangement of the imaging unit 1180 and the aerial operation detection sensor 1351 that detect the finger 210 etc. will be described in detail later.

[0123] At the first time point shown in Fig. 27, finger 210 is at a position farthest from display surface 3a of space-floating image 3 compared to the second time point shown in Fig. 28 and the third time point shown in Fig. 28. The distance (height position) between the tip of finger 210 and display surface 3a of space-floating image 3 at this time is taken as dz1. In other words, distance dz1 indicates the height of finger 210 relative to display surface 3a of space-floating image 3 in the z direction.

[0124] 27 and distance dz2 shown in Fig. 28 described later are defined with the user side as the positive side and the opposite side of the display surface 3a of the floating in space image 3 as the negative side. In other words, if the finger 210 is on the user side of the display surface 3a, the distances dz1 and dz2 will be positive values, and if the finger 210 is on the opposite side of the user from the display surface 3a, the distances dz1 and dz2 will be negative values.

[0125] In the present embodiment, it is assumed that virtual light source 1500 is located on the user side with respect to display surface 3a of space-floating image 3. Here, the setting of the installation direction of virtual light source 1500 may be actually stored as information in nonvolatile memory 1108 or memory 1109 of space-floating image display device 1000. Furthermore, the setting of the installation direction of virtual light source 1500 may be a parameter that exists only in design. Even if the setting of the installation direction of virtual light source 1500 is a parameter that exists only in design, the design installation direction of virtual light source 1500 is uniquely determined based on the relationship between the position of the user's finger and the display position of the virtual shadow, which will be described later. Here, in the examples of FIGS. 27 to 29 , virtual light source 1500 is located on the user side with respect to display surface 3a, on the right side of display surface 3a as seen from the user. Then, virtual shadow 1510, which resembles the shadow of finger 210 formed by light irradiated from virtual light source 1500, is displayed on space-floating image 3. 27 to 29, a virtual shadow 1510 is displayed on the left side of the finger 210. This virtual shadow 1510 assists the user in performing a touch operation.

[0126] In the state of FIG. 27(B), the tip of finger 210 is farthest in the normal direction from display surface 3a of space floating image 3 compared to the states of FIG. 28(B) and FIG. 29(B). Therefore, in FIG. 27(A), the tip of virtual shadow 1510 is formed at a position farthest in the horizontal direction from first button BUT1 to be touched compared to the states of FIG. 28(A) and FIG. 29(A). Therefore, in FIG. 27(A), the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space floating image 3 is viewed from the front is greatest compared to the states of FIG. 28(A) and FIG. 29(A). In FIG. 27(A), the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space floating image 3 is set to dx1.

[0127] 28(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 28(B), distance dz2 in the normal direction between the tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz1. At this time, in FIG. 28(A), virtual shadow 1510 is displayed at a position where the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx2, which is smaller than dx1. That is, in the example of FIG. 28, virtual light source 1500 is provided on the user side with respect to display surface 3a and on the right side of display surface 3a as seen from the user, and therefore the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of space-floating image 3.

[0128] Then, when the tip of finger 210 comes into contact with the tip of virtual shadow 1510, the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 becomes zero, as shown in Fig. 29. At this time, virtual shadow 1510 is displayed so that the distance between finger 210 and virtual shadow 1510 in the horizontal direction of display surface 3a of space floating image 3 becomes zero. This allows the user to recognize that finger 210 has touched display surface 3a of space floating image 3. At this time, if the tip of finger 210 is touching the area of ​​first button BUT1, the user can recognize that first button BUT1 has been touched. 29 example, virtual light source 1500 is provided on the user side of display surface 3a and to the right of display surface 3a as seen from the user, and therefore the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space floating image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3. In other words, the display position of the tip of virtual shadow 1510 is a position specified by the positional relationship between the position of virtual light source 1500 and the position of the tip of user's finger 210, and changes in conjunction with the change in the position of the tip of user's finger 210.

[0129] According to the configuration and processing of "Touch Operation Assistance Using Virtual Shadow (1)" explained above, when performing a touch operation, the user can more appropriately recognize the distance (depth) in the normal direction between the finger 210 and the display surface 3a of the Space Floating Image 3 from the horizontal positional relationship between the finger 210 and the virtual shadow 1510 on the display surface 3a of the Space Floating Image 3. Furthermore, when the finger 210 touches an object (e.g., a button) that is the Space Floating Image 3, the user can recognize that he or she has touched the object. This makes it possible to provide a more appropriate Space Floating Image Display Device. <<Touch Operation Assistance Using Virtual Shadows (2)>>

[0130] Next, as another example of a method for assisting a touch operation using a virtual shadow, a case where virtual light source 1500 is provided on the left side of display surface 3a as seen from the user will be described. FIGS. 30 to 32 are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. FIG. 30 corresponds to FIG. 27 and shows a state at a first point in time when a user attempts to touch first button BUT1 on display surface 3a of floating-in-space image 3 with finger 210. FIG. 31 corresponds to FIG. 28 and shows a state at a second point in time when finger 210 is closer to floating-in-space image 3 than in FIG. 30. FIG. 32 corresponds to FIG. 29 and shows a state when finger 210 touches floating-in-space image 3. For ease of explanation, (B) of FIGS. 30 to 32 shows a view from the opposite direction to (B) of FIGS. 27 to 29.

[0131] 30 to 32, virtual light source 1500 is provided on the user side of display surface 3a, on the left side of display surface 3a as seen from the user. Then, virtual shadow 1510, which resembles the shadow of finger 210 formed by light emitted from virtual light source 1500, is displayed in space floating image 3. In FIGS. 30 to 32, virtual shadow 1510 is displayed to the right of finger 210. This virtual shadow 1510 assists the user in performing a touch operation.

[0132] In the state of Fig. 30(B), compared to the states of Fig. 31(B) and Fig. 32(B), the tip of finger 210 is the furthest away in the normal direction from display surface 3a of space floating image 3. In Fig. 30(B), the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 at this time is dz10. Also, in Fig. 30(A), the distance in the horizontal direction of display surface 3a of space floating image 3 at this time between the tip of finger 210 and the tip of virtual shadow 1510 is dx10.

[0133] 31(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 31(B), distance dz20 in the normal direction between the tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz10. At this time, in FIG. 31(A), virtual shadow 1510 is displayed at a position where the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx20, which is smaller than dx10. That is, in the example of FIG. 31, virtual light source 1500 is provided on the user side with respect to display surface 3a, on the left side of display surface 3a as seen from the user, and therefore the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of space-floating image 3.

[0134] Then, when the tip of finger 210 comes into contact with the tip of virtual shadow 1510, the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 becomes zero, as shown in Fig. 32. At this time, virtual shadow 1510 is displayed so that the distance between finger 210 and virtual shadow 1510 in the horizontal direction of display surface 3a of space floating image 3 becomes zero. This allows the user to recognize that finger 210 has touched display surface 3a of space floating image 3. At this time, if the tip of finger 210 is touching the area of ​​first button BUT1, the user can recognize that they have touched first button BUT1. That is, in the example of FIG. 32, virtual light source 1500 is provided on the user side of display surface 3a, on the left side of display surface 3a as seen from the user, and therefore the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space floating image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3.

[0135] The above-described configuration and processing of "assistance of touch operation using virtual shadow (2)" also provides the same effects as the configurations of FIGS.

[0136] Here, when implementing the above-mentioned "Touch operation assistance using virtual shadow (1)" process and / or "Touch operation assistance using virtual shadow (2)" process in the space floating image display device 1000, there are several possible implementation examples as follows.

[0137] The first implementation example is a method of implementing only "touch operation assistance using a virtual shadow (1)" in the space-floating image display device 1000. In this case, since the virtual light source 1500 is provided on the user side of the display surface 3a and on the right side of the display surface 3a as seen from the user, the virtual shadow 1510 is displayed on the left side of the tip of the user's finger 210 as seen from the user. Therefore, if the user's finger 210 is a finger of the right hand, the visibility of the display of the virtual shadow 1510 is not obstructed by the user's right hand or right arm, which is preferable. Therefore, considering the statistical tendency that there are more right-handed users, even if only "touch operation assistance using a virtual shadow (1)" is implemented in the space-floating image display device 1000, the probability that the display of the virtual shadow 1510 can be clearly seen is sufficiently high, which is preferable.

[0138] As a first implementation example, both the "touch operation assistance using a virtual shadow (1)" process and the "touch operation assistance using a virtual shadow (2)" process may be implemented, and the process to be performed may be switched depending on whether the user performs the touch operation with their right hand or left hand. In this case, it is possible to further increase the probability that the display of the virtual shadow 1510 can be clearly seen, improving user convenience.

[0139] Specifically, when a user performs a touch operation with his / her right hand, the configurations of FIGS. 27 to 29 are used to display a virtual shadow 1510 to the left of the finger 210. In this case, the visibility of the displayed virtual shadow 1510 is not obstructed by the user's right hand or right arm, which is preferable. On the other hand, when a user performs a touch operation with his / her left hand, the configurations of FIGS. 30 to 32 are used to display a virtual shadow 1510 to the right of the finger 210. In this case, the visibility of the displayed virtual shadow 1510 is not obstructed by the user's left hand or left arm, which is preferable. As a result, whether the user performs a touch operation with his / her right hand or left hand, the virtual shadow 1510 is displayed in a position that is easily visible to the user, improving user convenience.

[0140] Here, the determination of whether the touch operation is being performed with the right hand or the left hand may be made, for example, based on a captured image generated by the imaging unit 1180. For example, the control unit 1110 performs image processing on the captured image and detects the user's face, arms, hands, and fingers from the captured image. The imaging unit 1180 then estimates the user's posture or movement from the arrangement of these detected elements (face, arms, hands, and fingers) and determines whether the user is performing the touch operation with the right hand or the left hand. Note that, in this determination, capturing an image of the face is not necessarily required as long as the vicinity of the center of the user's body in the horizontal direction can be determined from other parts. Alternatively, the above determination may be made only from the arrangement of the arms. Alternatively, the above determination may be made only from the arrangement of the hands. Alternatively, the above determination may be made from a combination of the arrangement of the arms and the arrangement of the hands. Furthermore, the determination may be made in combination with the arrangement of the face.

[0141] 27 to 29 and 30 to 32 show virtual shadow 1510 extending at an angle corresponding to the extending direction of actual finger 210. The extending direction of actual finger 210 may be calculated by capturing an image of the finger using any of the imaging units already described. Here, virtual shadow 1510 may be displayed with the extending direction fixed at a predetermined angle without reflecting the angle corresponding to the extending direction of finger 210. This reduces the load on image control unit 1160 or control unit 1110 that controls the display of virtual shadow 1510.

[0142] For example, if finger 210 is a finger of the right hand, it is natural for the user to extend their arm from the right side in front of display surface 3a of space floating image 3, and try to touch display surface 3a of space floating image 3 with finger 210 pointing to the upper left as they face display surface 3a of space floating image 3. Therefore, if finger 210 is a finger of the right hand, by configuring it so that the shadow of the finger indicated by virtual shadow 1510 is displayed in a predetermined direction pointing to the upper right as they face display surface 3a of space floating image 3, a natural display will be achieved without reflecting the angle corresponding to finger 210.

[0143] Furthermore, for example, if finger 210 is a finger of the left hand, it is natural for the user to extend their arm from the left side in front of display surface 3a of space floating image 3, and attempt to touch display surface 3a of space floating image 3 with finger 210 pointing to the upper right as they face display surface 3a of space floating image 3. Therefore, if finger 210 is a finger of the left hand, by configuring it so that the shadow of the finger indicated by virtual shadow 1510 is displayed in a predetermined direction pointing to the upper left as they face display surface 3a of space floating image 3, a natural display will be achieved without reflecting the angle corresponding to finger 210.

[0144] Note that when the user's finger 210 is on the opposite side of the display surface 3a of the space floating image 3 from the user, a display that allows the user to recognize that the finger 210 is behind the space floating image 3 and cannot be touched may be displayed. For example, a message that tells the user that the finger 210 is behind the space floating image 3 and cannot be touched may be displayed on the space floating image 3. Alternatively, for example, the virtual shadow 1510 may be displayed in a different color from normal, such as red. This makes it possible to more suitably prompt the user to return the finger 210 to the appropriate position. <<Example of virtual light source setting conditions>>

[0145] Here, a method for setting virtual light source 1500 will be described. Fig. 33 is a diagram illustrating a method for setting a virtual light source. Fig. 33 shows a situation in which a user performs a touch operation with his / her left hand, but the content described below is also preferably applicable to a case in which a user performs a touch operation with his / her right hand.

[0146] 33 shows normal line L1 of display surface 3a extending from point C at the center of display surface 3a of space floating image 3 toward the user, line L2 connecting virtual light source 1500 and point C where normal line L1 intersects with display surface 3a, and virtual light source installation angle α defined by the angle between normal line L1 and line L2. For ease of explanation, FIG. 33 shows the moment when the tip of user's finger 210 is on line L2.

[0147] 27 to 33, for ease of explanation, virtual light source 1500 is illustrated as being placed at a position not far from display surface 3a of space-floating image 3 and user's finger 210. While virtual light source 1500 may be set at such a position, the most preferable setting example is as follows. That is, it is desirable to set the distance between virtual light source 1500 and point C at the center of display surface 3a of space-floating image 3 at infinity. The reason for this is as follows. If an object plane having a contact surface is in the same coordinate system as display surface 3a of space-floating image 3 in FIGS. 27 to 32, and the light source is the sun rather than a virtual light source, the distance of the sun can be approximated as almost infinity, and therefore the horizontal direction (x direction) position of the tip of the shadow of the user's finger on the real object plane changes linearly with respect to the change in the distance (z direction) between the tip of the user's finger and the object plane. 27 to 33 of this embodiment, if the distance between virtual light source 1500 and center point C of display surface 3 a of space-floating image 3 is set to infinity and the horizontal (x-direction) position of the tip of virtual shadow 1510 in space-floating image 3 changes linearly with changes in the distance (z-direction) between the tip of user's finger 210 and display surface 3 a of space-floating image 3, a virtual shadow that can be perceived more naturally by the user can be expressed. If virtual light source 1500 is set to be located not far from display surface 3 a of space-floating image 3 and user's finger 210, the horizontal (x-direction) position of the tip of virtual shadow 1510 in space-floating image 3 changes nonlinearly with changes in the distance (z-direction) between the tip of user's finger 210 and display surface 3 a of space-floating image 3, and the calculation for calculating the horizontal (x-direction) position of the tip of virtual shadow 1510 becomes somewhat complicated. On the other hand, if the distance between virtual light source 1500 and center point C of display surface 3a of space floating image 3 is set to infinity, the horizontal position (x direction) of the tip of virtual shadow 1510 in space floating image 3 changes linearly with the change in the distance (z direction) between the tip of user's finger 210 and display surface 3a of space floating image 3, which has the effect of simplifying the calculation for calculating the horizontal position (x direction) of the tip of virtual shadow 1510.

[0148] When virtual light source installation angle α is small, the angle between normal line L1 and a line connecting virtual light source 1500 and finger 210 cannot be increased as viewed from the user, and therefore the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction (x direction) of display surface 3 a of floating-in-space image 3 becomes short. This makes it difficult for the user to visually recognize the change in the position of virtual shadow 1510 when the tip of finger 210 performs a touch operation, which may reduce the effectiveness of the user's depth perception in the touch operation. To avoid this, it is desirable to install virtual light source 1500 so that the angle between normal line L1 and line L2 connecting virtual light source 1500 and point C is, for example, 20° or more.

[0149] On the other hand, when the angle between normal line L1 and the line connecting virtual light source 1500 and finger 210 approaches 90°, the distance between the tip of finger 210 and the tip of virtual shadow 1510 becomes very long. This increases the probability that the display position of virtual shadow 1510 will be outside the range of floating-in-space image 3, and increases the probability that virtual shadow 1510 will not be displayed in floating-in-space image 3. For this reason, it is desirable that installation angle α of virtual light source 1500 be 70° or less so that the angle between normal line L1 and line L2 connecting virtual light source 1500 and point C does not approach too close to 90°, for example.

[0150] That is, it is desirable that virtual light source 1500 be placed at a position that is not too close to a plane that includes a normal line passing through finger 210 and is not too close to a plane that includes display surface 3a of space floating image 3.

[0151] The space-floating image display device 1000 of this embodiment can display a virtual shadow as described above. This is an image processing that produces a physically more natural effect than when a predetermined mark is superimposed on an image to assist the user's touch operation. Therefore, the touch operation assistance technology of the space-floating image display device 1000 of this embodiment by displaying the above-mentioned virtual shadow can provide a situation in which the user can recognize the depth of the touch operation more naturally. <<How to detect finger position>>

[0152] Next, a description will be given of a method for detecting the position of finger 210. A configuration for detecting the position of finger 210 of user 230 will be specifically described below. <<<Finger position detection method (1)>>>

[0153] Fig. 34 is a configuration diagram showing an example of a method for detecting the position of a finger. In the example shown in Fig. 34, the position of a finger 210 is detected using one imaging unit 1180 and one air operation sensor 1351. Note that each imaging unit in the embodiment of the present invention has an imaging sensor.

[0154] The first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the space floating image 3. The first imaging unit 1180a may be installed in the housing 1190 as shown in FIG. 34, or may be installed in a location away from the housing 1190.

[0155] The imaging area of ​​the first imaging unit 1180a is set to include, for example, the display area of ​​the space-floating image 3, as well as the fingers, hands, arms, face, etc. of the user 230. The first imaging unit 1180a images the user 230 performing a touch operation on the space-floating image 3, and generates a first captured image. Note that even if the display area of ​​the space-floating image 3 is captured by the first imaging unit 1180a, the image is captured from the opposite side of the traveling direction of the directional light beam of the space-floating image 3, so the space-floating image 3 itself cannot be seen as an image. Here, in the example of finger position detection method (1), the first imaging unit 1180a is not just an imaging unit, but has a built-in depth sensor in addition to an imaging sensor. The configuration and processing of the depth sensor may use existing technology. The depth sensor of the first imaging unit 1180a detects the depth of each part (e.g., the user's fingers, hands, arms, face, etc.) in the captured image of the first imaging unit 1180a and generates depth information.

[0156] The aerial operation sensor 1351 is installed in a position where it can sense the display surface 3a of the space floating image 3 as the sensing target surface. In Fig. 34, the aerial operation sensor 1351 is installed below the display surface 3a of the space floating image 3, but it may also be installed to the side or above the display surface 3a. The aerial operation sensor 1351 may be installed in the housing 1190 as shown in Fig. 34, or may be installed in a location away from the housing 1190.

[0157] 34 is a sensor that detects the position where the finger 210 contacts or overlaps with the display surface 3a of the floating in space image 3. In other words, when the tip of the finger 210 approaches the display surface 3a of the floating in space image 3 from the user's side of the display surface 3a of the floating in space image 3, the floating in space operation detection sensor 1351 can detect the contact of the finger 210 with the display surface 3a of the floating in space image 3.

[0158] 3C reads out from the non-volatile memory 1108 a program for performing image processing and a program for displaying a virtual shadow 1510. The control unit 1110 performs first image processing on a first captured image generated by the imaging sensor of the first imaging unit 1180a, detects the finger 210, and calculates the position (x coordinate, y coordinate) of the finger 210. The control unit 1110 calculates the position (z coordinate) of the tip of the finger 210 relative to the floating in space image 3 based on the first captured image generated by the imaging sensor of the first imaging unit 1180a and depth information generated by the depth sensor of the first imaging unit 1180a.

[0159] 34, the image sensor and depth sensor of the first imaging unit 1180a, the aerial operation sensor 1351, the aerial operation detection unit 1350, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects a touch on an object in the floating in space image 3. This calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210. Furthermore, the touch detection result is calculated by combining the detection result of the aerial operation detection unit 1350 or the detection result of the aerial operation detection unit 1350 with information generated by the first imaging unit 1180a.

[0160] Then, control unit 1110 calculates the position (display position) at which virtual shadow 1510 is to be displayed based on the position (x coordinate, y coordinate, z coordinate) of finger 210 and the position of virtual light source 1500, and generates video data of virtual shadow 1510 based on the calculated display position.

[0161] Note that the control unit 1110 may calculate the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated. Calculation of the display position of the virtual shadow 1510 in the video data does not have to be performed each time the position of the finger 210 is calculated. Instead, display position map data in which the display positions of the virtual shadow 1510 corresponding to each of a plurality of positions of the finger 210 are calculated may be stored in advance in the nonvolatile memory 1108, and after calculating the position of the finger 210, video data of the virtual shadow 1510 may be generated based on the display position map data stored in the nonvolatile memory 1108. Furthermore, the control unit 1110 may calculate the tip of the finger 210 and the extension direction of the finger 210 in the first image processing, calculate the display position of the tip of the finger 210 and the extension direction of the virtual shadow 1510 corresponding to the extension direction, and generate video data of the virtual shadow 1510 adjusted to a display angle corresponding to the actual orientation of the finger 210 based on these.

[0162] The control unit 1110 outputs the generated video data of the virtual shadow 1510 to the video control unit 1160. The video control unit 1160 generates video data (superimposed video data) in which the video data of the virtual shadow 1510 is superimposed on other video data such as an object, and outputs the superimposed video data including the video data of the virtual shadow 1510 to the video display unit 1102.

[0163] The video display unit 1102 displays a video based on the superimposed video data including the video data of the virtual shadow 1510, thereby displaying a space floating video 3 in which the virtual shadow 1510 and an object or the like are superimposed.

[0164] Detection of a touch on an object is performed, for example, as follows. The aerial operation detection unit 1350 and the aerial operation detection sensor 1351 are configured as described in Fig. 3, and when the finger 210 touches or is superimposed on a plane including the display surface 3a of the space floating image 3, they detect the position and output touch position information indicating the position where the finger 210 touches or is superimposed on the display surface 3a to the control unit 1110. Then, when the touch position information is input, the control unit 1110 determines whether or not the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing is included in the display range of each object displayed on the display surface 3a of the space floating image 3. Then, when the position of the finger 210 is included in the display range of any one of the objects, the control unit 1110 determines that a touch has been made on this object.

[0165] According to the detection method described above, it is possible to detect the position of the finger 210 and the touch operation with a simple configuration that combines one imaging unit 1180 (first imaging unit 1180a) having an imaging sensor and a depth sensor with one aerial operation detection sensor 1351.

[0166] As a variant of the finger position detection method (1), the control unit 1110 may detect a touch operation by the finger 210 solely based on the first captured image generated by the image sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, without using the detection results by the aerial operation detection unit 1350 and the aerial operation detection sensor 1351. For example, during normal operation, the system is configured to be in a mode in which touch operations by the finger 210 are detected by combining the captured image of the image sensor of the first imaging unit 1180a, the detection results of the depth sensor, and the detection results of the aerial operation detection sensor 1351, and if there is any malfunction in the operation of the aerial operation detection sensor 1351 or the aerial operation detection unit 1350, the control unit 1110 may switch to a mode in which touch operations by the finger 210 are detected solely based on the first captured image generated by the image sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, without using the detection results of the aerial operation detection unit 1350 and the aerial operation detection sensor 1351. <<Finger position detection method (2)>>

[0167] Fig. 35 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in Fig. 35, the position of the finger 210 is detected using two imaging units. The second imaging unit 1180b (1180) and the third imaging unit 1180c (1180) are both provided on the opposite side of the user 230 with respect to the floating image 3 in space.

[0168] The second imaging unit 1180b is installed, for example, on the right side as seen from the user 230. The imaging area of ​​the second imaging unit 1180b is set to include, for example, the floating in space image 3, the fingers, hands, arms, face, etc. of the user 230. The second imaging unit 1180b images the user 230 performing a touch operation on the floating in space image 3 from the right side of the user 230, and generates a second captured image.

[0169] The third imaging unit 1180c is installed, for example, on the left side as seen from the user 230. The imaging area of ​​the third imaging unit 1180c is set to include, for example, the floating in space image 3, the fingers, hands, arms, face, etc. of the user 230. The third imaging unit 1180c images the user 230 performing a touch operation on the floating in space image 3 from the left side of the user 230, and generates a third captured image. In this way, in the example of FIG. 35, the second imaging unit 1180b and the third imaging unit 1180c configure a so-called stereo camera.

[0170] 35, the second imaging unit 1180b and the third imaging unit 1180c may be installed in the housing 1190, or may be installed at a location away from the housing 1190. Also, one imaging unit may be installed in the housing 1190, and the other imaging unit may be installed at a location away from the housing 1190.

[0171] The control unit 1110 performs second image processing on the second captured image and third image processing on the third captured image, and then calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210 based on the result of the second image processing (second image processing result) and the result of the third image processing (third image processing result).

[0172] 35, the second imaging unit 1180b, the third imaging unit 1180c, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects a touch on an object in the space floating image 3. Then, the position (x coordinate, y coordinate, z coordinate) of the finger 210 is calculated as a position detection result or a touch detection result.

[0173] 35, a virtual shadow 1510 is generated based on the position of the finger 210 calculated based on the second image processing result and the third image processing result. Also, based on the position of the finger 210 calculated based on the second image processing result and the third image processing result, it is determined whether or not the object has been touched.

[0174] According to this configuration, there is no need to employ an imaging unit having a depth sensor. Also, according to this configuration, by using the second imaging unit 1180b and the third imaging unit 1180c as a stereo camera, it is possible to improve the detection accuracy of the position of the finger 210. In particular, compared to the example of FIG. 34, it is possible to improve the detection accuracy of the x coordinate and the y coordinate. Therefore, it is possible to more accurately determine whether or not an object has been touched.

[0175] Furthermore, as a modified example of the finger position detection method (2), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the second captured image by the second imaging unit 1180b and the third captured image by the third imaging unit 1180c, as described above, and the display of the virtual shadow 1510 is controlled based on this, and the presence or absence of a touch on the object of the space floating image 3 can be detected by the aerial operation detection unit 1350 or the control unit 1110 based on the detection result by the aerial operation detection sensor 1351. According to this modified example, since the aerial operation sensor 1351 that senses the display surface 3a of the space floating image 3 is used as the sensing target surface, it is possible to detect the contact of the user's finger 210 with the display surface 3a of the space floating image 3 with higher accuracy than the detection accuracy in the depth direction by the stereo camera of the second imaging unit 1180b and the third imaging unit 1180c. <<<Finger position detection method (3)>>>

[0176] FIG. 36 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in FIG. 36, the position of the finger 210 is also detected using two imaging units. The example in FIG. 36 differs from the example in FIG. 35 in that the fourth imaging unit 1180d (1180), which is one of the imaging units, is positioned to capture an image of the display surface 3a of the space floating image 3 from the side. Also, as in the example in FIG. 34, the first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the space floating image 3. In the example in FIG. 36, the first imaging unit 1180a (1180) only needs to be able to capture an image, and does not need to be equipped with a depth sensor.

[0177] Therefore, the fourth imaging unit 1180d is installed around the display surface 3a of the space floating image 3. In Fig. 36, the fourth imaging unit 1180d is installed below the side of the display surface 3a of the space floating image 3, but it may also be installed to the side or above the display surface 3a. The fourth imaging unit 1180d may be installed in the housing 1190 as shown in Fig. 36, or may be installed in a location away from the housing 1190.

[0178] The imaging area of ​​the fourth imaging unit 1180d is set to include, for example, the space floating image 3, and the fingers, hands, arms, face, etc. of the user 230. The fourth imaging unit 1180d images the user 230 performing a touch operation on the space floating image 3 from around the display surface 3a of the space floating image 3, and generates a fourth captured image.

[0179] The control unit 1110 performs fourth image processing on the fourth captured image, and calculates the distance (z coordinate) between the display surface 3a of the space floating image 3 and the tip of the finger 210. Then, the control unit 1110 performs processing related to the virtual shadow 1510 and determines whether or not an object has been touched, based on the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing on the first captured image by the first imaging unit 1180a described above and the position (z coordinate) of the finger 210 calculated by the fourth image processing.

[0180] 36, a touch detection unit that detects the position of a user's finger and detects a touch on an object is configured by the first imaging unit 1180a, the fourth imaging unit 1180d, and the control unit 1110. Then, the position (x coordinate, y coordinate, z coordinate) of the finger 210 is calculated as a position detection result or a touch detection result.

[0181] According to this configuration, it is possible to improve the accuracy of detecting the distance between the display surface 3a of the floating image 3 in space and the tip of the finger 210, i.e., the depth of the finger 210 relative to the display surface 3a of the floating image 3 in space, compared to the example of the stereo camera configuration in FIG.

[0182] Furthermore, as a modified example of the finger position detection method (3), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the first captured image by the first imaging unit 1180a and the fourth captured image by the fourth imaging unit 1180d, as described above, and the display of the virtual shadow 1510 is controlled based on this, and the presence or absence of a touch on the object of the space floating image 3 can be detected by the aerial operation detection unit 1350 or the control unit 1110 based on the detection result by the aerial operation detection sensor 1351. According to this modified example, the aerial operation sensor 1351 that senses the display surface 3a of the space floating image 3 is used, so that the touch of the user's finger 210 on the display surface 3a of the space floating image 3 can be detected with higher accuracy than the detection accuracy using the fourth captured image by the fourth imaging unit 1180d. <<How to display input content and assist touch operation>>

[0183] An example of assisting the user's touch operation in another way will be described. For example, it is possible to assist the touch operation by displaying the input content. Fig. 37 is a diagram illustrating a method of assisting the touch operation by displaying the input content. Fig. 37 shows a case where numbers are input by touch operation.

[0184] The floating image 3 in FIG. 37 includes a key input UI (user interface) display area 1600 containing multiple objects including, for example, multiple objects for inputting numbers, etc., an object 1601 for erasing the input content, and an object 1603 for determining the input content, and an input content display area 1610 for displaying the input content.

[0185] In the input content display area 1610, the content (for example, numbers) input by touch operation is displayed sequentially on the floating in space image 3 from the left end to the right. The user can check the content input by touch operation while looking at the input content display area 1610. Then, when the user has input all the desired numbers, he or she touches the object 1603. This registers the input content displayed in the input content display area 1610. Unlike physical contact with the surface of the display device, the touch operation on the floating in space image 3 does not give the user a tactile sensation. Therefore, by separately displaying the input content in the input content display area 1610, the user can proceed with the operation while checking whether or not their touch operation has been performed effectively, which is preferable.

[0186] On the other hand, if the user touches the wrong object or otherwise inputs content different from what is desired, the user can erase the last input content (here, "9") by touching object 1601. Then, the user continues to perform touch operations on objects for inputting numbers, etc. When the user has input all the desired numbers, he or she touches object 1603.

[0187] In this way, by displaying the input contents in the input content display area 1610, the user can confirm the input contents, thereby improving convenience. Furthermore, if the user touches the wrong object, the input contents can be corrected, thereby improving convenience. <<How to highlight input content to assist touch operation>>

[0188] Next, it is also possible to assist the touch operation by highlighting the input content. Figure 38 is a diagram for explaining a method of highlighting the input content to assist the touch operation.

[0189] An example in which a number entered by a touch operation is highlighted is shown in Figure 38. Referring to Figure 38, when the object corresponding to the number "6" is touched, the touched object is erased, and the input number "6" is displayed in the area where this object was displayed.

[0190] In this way, the number corresponding to the touched object is displayed in place of the object, which allows the user to recognize that the object has been touched, thereby improving convenience. The number corresponding to the touched object may be called a replacement object, which replaces the touched object.

[0191] As another method for highlighting the input content, for example, the object touched by the user may be brightly lit or may blink. Although not shown here, by recognizing the distance between the finger 210 and the display surface 3a as described in the embodiment of Figures 27 and 28, the object being touched by the finger may become brighter than the surrounding objects as the finger approaches the display surface, and finally, when the finger is moved toward the display surface, the object may become most highlighted, or may be lit even brighter, or may blink. Even with such a configuration, it is possible for the user to recognize that an object has been touched, thereby improving convenience. <<Method to assist touch operation with vibration (1)>>

[0192] Next, a method of assisting a touch operation by vibration will be described. Fig. 39 is a diagram illustrating an example of a method of assisting a touch operation by vibration. Fig. 39 shows a case where a touch operation is performed using a touch pen (touch input device) 1700 instead of a finger 210. The touch pen 1700 is equipped with a communication unit that transmits and receives various information such as signals and data to and from a device such as a space floating image display device, and a vibration mechanism that vibrates based on an input signal.

[0193] Assume that the user operates the touch pen 1700 and touches an object displayed in the key input UI display area 1600 of the floating-in-space image 3 with the touch pen 1700. At this time, for example, the control unit 1100 transmits a touch detection signal indicating that a touch on the object has been detected from the communication unit 1132. When the touch pen 1700 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the touch pen 1700 to vibrate. The vibration of the touch pen 1700 is then transmitted to the user, who recognizes that they have touched the object. In this way, the vibration of the touch pen 1700 assists the touch operation.

[0194] According to this configuration, it is possible to make the user aware that he or she has touched the object by means of vibration.

[0195] Here, the case where the touch pen 1700 receives the touch detection signal transmitted from the space-floating image device has been described, but other configurations are also possible. For example, when detecting a touch on an object, the space-floating image display device notifies the host device that a touch on the object has been detected. Then, the host device transmits a touch detection signal to the touch pen 1700.

[0196] Alternatively, the space-floating image display device and the higher-level device may transmit the touch detection signal via a network. In this way, the touch pen 1700 may receive the touch detection signal indirectly from the space-floating image display device. <<Method to assist touch operation with vibration (2)>>

[0197] Next, another method of assisting a touch operation by vibration will be described. Here, the user is made aware that he or she has touched an object by vibrating the terminal owned by the user. FIG. 40 is a diagram illustrating another example of a method of assisting a touch operation by vibration. In the example of FIG. 40, a user 230 wearing a wristwatch-type wearable terminal 1800 performs a touch operation.

[0198] The wearable terminal 1800 is equipped with a communication unit that transmits and receives various information such as signals and data to and from devices such as a space floating image display device, and a vibration mechanism that vibrates based on an input signal.

[0199] Assume that the user performs a touch operation with the finger 210 and touches an object displayed in the key input UI display area 1600 of the floating-in-space image 3. At this time, for example, the control unit 1100 transmits a touch detection signal indicating that a touch on the object has been detected from the communication unit 1132. When the wearable device 1800 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the wearable device 1800 to vibrate. The vibration of the wearable device 1800 is then transmitted to the user, who recognizes that he or she has touched the object. In this way, the vibration of the wearable device 1800 assists the touch operation. Here, a wristwatch-type wearable device has been described as an example, but a smartphone worn by the user may also be used.

[0200] The wearable terminal 1800 may receive a touch detection signal from a higher-level device, similar to the above-described touch pen 1700. The wearable terminal 1800 may receive a touch detection signal via a network. In addition to the wearable terminal 1800, it is also possible to assist the touch operation using an information processing terminal such as a smartphone owned by the user.

[0201] According to this configuration, it is possible to make the user aware that he or she has touched an object via various terminals such as the wearable terminal 1800 owned by the user. <<Method to assist touch operation with vibration (3)>>

[0202] Next, another method of assisting a touch operation by vibration will be described. FIG. 41 is a diagram illustrating another example of a method of assisting a touch operation by vibration. In the example of FIG. 41, user 230 performs a touch operation while standing on diaphragm 1900. Diaphragm 1900 is installed at a predetermined position where user 230 performs a touch operation. In actual use, diaphragm 1900 is placed, for example, under a mat (not shown), and user 230 stands on diaphragm 1900 via the mat.

[0203] As shown in FIG. 41 , diaphragm 1900 is connected to, for example, communication unit 1132 of space floating image display device 1000 via cable 1910. When a touch on an object is detected, for example, control unit 1110 supplies AC voltage to diaphragm 1900 for a predetermined time via communication unit 1132. Diaphragm 1900 vibrates while the AC voltage is being supplied. In other words, the AC voltage is a control signal output from communication unit 1132 for vibrating diaphragm 1900. The vibration generated by diaphragm 1900 is transmitted from the feet to user 230, allowing user 230 to recognize that an object has been touched. In this way, the vibration of diaphragm 1900 assists the touch operation.

[0204] The frequency of the AC voltage is set to a value within a range in which the user 230 can sense vibrations. The frequencies of vibrations that people can sense are in the range of approximately 0.1 Hz to 500 Hz. Therefore, it is desirable to set the frequency of the AC voltage within this range.

[0205] Furthermore, it is desirable to change the frequency of the AC voltage as appropriate depending on the characteristics of diaphragm 1900. For example, when diaphragm 1900 vibrates vertically, humans are said to be most sensitive to vibrations of about 410 Hz. When diaphragm 1900 vibrates horizontally, humans are said to be most sensitive to vibrations of about 12 Hz. Furthermore, at frequencies of 34 Hz ​​or higher, humans are said to be more sensitive to vibrations in the vertical direction than in the horizontal direction.

[0206] Therefore, when diaphragm 1900 vibrates in the vertical direction, it is desirable to set the frequency of the AC voltage to a value within a range including, for example, 410 Hz. Also, when diaphragm 1900 vibrates in the horizontal direction, it is desirable to set the frequency of the AC voltage to a value within a range including, for example, 12 Hz. Note that the peak voltage and frequency of the AC voltage may be adjusted as appropriate depending on the performance of diaphragm 1900.

[0207] According to this configuration, it is possible to make the user 230 aware that an object has been touched by a vibration from the feet. Also, with this configuration, it is possible to set the display of the floating image 3 not to change when an object is touched, which reduces the possibility that the input contents will be discovered even if someone else peeks at the touch operation, making it possible to further improve security.

[0208] Various embodiments have been described above in detail. However, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments are detailed descriptions of the entire system to clearly explain the present invention, and the present invention is not necessarily limited to a system including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0209] 1...display device, 2...retroreflective member, 3...spatial image (space-floating image), 105...window glass, 100...transparent member, 101...polarized light separation member, 12...absorptive polarizer, 13...light source device, 54...light direction conversion panel, 151...retroreflective member, 102, 202...LED substrate, 203...light guide, 205, 271...reflective sheet, 206, 270...phase difference plate, 300...space-floating image, 301...ghost of space-floating image ghost image, 302...ghost image of floating image in space, 230...user, 1000...floating image display device, 1110...control unit, 1160...image control unit, 1180...imaging unit, 1102...image display unit, 1350...air operation detection unit, 1351...air operation detection sensor, 1500...virtual light source, 1510...virtual shadow, 1610...input content display area, 1700...touch pen, 1800...wearable terminal, 1900...diaphragm.

Claims

1. A space floating image display device, a display device for displaying an image; a retroreflective member that reflects image light from the display device and forms a floating image in the air using the reflected light; a sensor for detecting the position of a user's finger performing a touch operation on one or more objects displayed on the floating image; A control unit; Equipped with The control unit controls image processing for the image displayed on the display device based on the position of the user's finger detected using the sensor, so that the space-floating image display device displays a virtual shadow of the user's finger on the display surface of the space-floating image, which has no physical contact surface; a position of the virtual shadow displayed on the display surface of the space floating image is a position specified from a positional relationship between a position of a virtual light source and a position of the user's finger detected using the sensor, and the position of the virtual light source is set at a position at an infinite distance from the display surface of the space floating image; When the position of the tip of the user's finger approaches the display surface of the space floating image so that the distance in the normal direction to the display surface of the space floating image becomes small on the front side of the display surface of the space floating image as seen from the user, the display of the virtual shadow of the user's finger is continued until the distance in the normal direction between the position of the tip of the user's finger and the display surface of the space floating image becomes 0; A floating image display device.

2. 2. The space floating image display device according to claim 1, When the position of the tip of the user's finger changes in the normal direction on the front side of the display surface of the space floating image as seen from the user, the position of the tip of the virtual shadow displayed on the space floating image in the left-right direction on the display surface of the space floating image changes. A floating image display device.

3. 3. The space floating image display device according to claim 2, With respect to the change in the normal direction of the position of the tip of the user's finger, a left-right position of the tip of the virtual shadow displayed on the floating image in the space on the display surface of the floating image in the space changes linearly. A floating image display device.

4. 2. The space floating image display device according to claim 1, an imaging unit that captures an image of a user's hand or arm; the position of the virtual light source is changed depending on whether the finger of the user performing the touch operation is of the right hand or the left hand, so that when the finger of the user performing the touch operation on one or more objects displayed in the space floating image is of the right hand, the virtual shadow is displayed in the space floating image at a position to the left of the tip of the finger as seen from the user, and when the finger of the user performing the touch operation on one or more objects displayed in the space floating image is of the left hand, the virtual shadow is displayed in the space floating image at a position to the right of the tip of the finger as seen from the user; A floating image display device.

5. 2. The space floating image display device according to claim 1, The control unit detects the position of the tip of the finger on the display surface of the space floating image and the height position of the tip of the finger relative to the display surface using the sensor that detects the position of the user's finger. A floating image display device.

6. 2. The space floating image display device according to claim 1, The presence or absence of the user's finger touching the display surface of the floating image in space is detected by a sensor different from a sensor that detects the position of the user's finger. A floating image display device.

7. 2. The space floating image display device according to claim 1, The position of the virtual light source is A position where a virtual light source installation angle defined by the angle between a normal line extending from a center point of the display surface of the space floating image toward the user and a line connecting the virtual light source and the center point of the display surface of the space floating image is 20° or more. A floating image display device.

8. 2. The space floating image display device according to claim 1, The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image changes in conjunction with the angle of the user's finger imaged by the imaging unit of the space floating image display device. A floating image display device.

9. 2. The space floating image display device according to claim 1, The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image is a fixed angle, not linked to the angle of the user's finger imaged by the imaging unit of the space floating image display device, A floating image display device.

10. 2. The space floating image display device according to claim 1, When a distance in the normal direction between the position of the tip of the user's finger and the display surface of the space floating image becomes 0, the virtual shadow is displayed on the space floating image so that the tip of the virtual shadow and the tip of the user's finger come into contact with each other in the left-right direction on the display surface of the space floating image. A floating image display device.

Images