Space floating video display device

Abstract

To provide a more suitable space floating video display device.SOLUTION: A space floating video display device continuously displays a virtual shadow of a user's finger on a display surface of a space floating video until a distance in a normal direction between the position of a tip of the user's finger and the display surface of the space floating video becomes zero when the position of a tip of the user's finger approaches the display surface of the space floating video such that the distance in the normal direction to the display surface of the space floating video becomes small on a near side of the display surface of the space floating video viewed from the user.SELECTED DRAWING: Figure 27

Description

[Technical field]

[0001] The present invention relates to a space floating image display device. [Background technology]

[0002] As a space-floating information display system, a video display device that displays an image directly to the outside and a display method that displays the image as a spatial screen are already known. In addition, a detection system that reduces false detections of operations on the operation surface of the displayed spatial image is also disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-128722 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, a touch operation on a floating-in-space image is not performed on a physical button, touch panel, etc. For this reason, there are cases where the user cannot recognize whether or not a touch operation has been performed. Therefore, an object of the present invention is to provide a more suitable space floating image display device. [Means for solving the problem]

[0005] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problem, and as an example, a space-floating image display device includes a display device that displays an image, a retroreflective member that reflects image light from the display device and forms a space-floating image in the air with the reflected light, a sensor that detects the position of a user's finger that performs a touch operation on one or more objects displayed in the space-floating image, and a control unit, and the control unit controls image processing for the image displayed on the display device based on the position of the user's finger detected using the sensor, so that the space-floating image display device displays a virtual shadow of the user's finger on the display surface of the space-floating image that does not have a physical contact surface. The position of the virtual shadow displayed on the display surface of the floating-in-space image is a position specified from the positional relationship between the position of the virtual light source and the position of the user's finger detected using the sensor, and the position of the virtual light source is set at an infinite distance from the display surface of the floating-in-space image, and when the position of the tip of the user's finger approaches the display surface of the floating-in-space image such that the normal distance to the display surface of the floating-in-space image becomes small on the front side of the display surface of the floating-in-space image as seen from the user, the virtual shadow of the user's finger continues to be displayed on the display surface of the floating-in-space image until the normal distance between the position of the tip of the user's finger and the display surface of the floating-in-space image becomes zero. Effect of the Invention

[0006] According to the present invention, a more suitable space floating image display device can be realized. Other objects, configurations and effects will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram showing an example of a usage form of a space floating image display device according to an embodiment of the present invention; [Diagram 2] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 3A] 1A and 1B are diagrams illustrating an example of a method for installing a space floating image display device. [Figure 3B] 11A and 11B are diagrams showing another example of a method for installing the space floating image display device. [Figure 3C] FIG. 1 is a diagram showing a configuration example of a space floating image display device. [Figure 4] FIG. 11 is a diagram showing another example of the main part configuration of the space floating image display device according to an embodiment of the present invention. [Diagram 5] FIG. 11 is an explanatory diagram for explaining the function of a sensing device used in the space floating image display device. [Figure 6] FIG. 1 is an explanatory diagram of the principle of 3D image display used in a space floating image display device. [Figure 7] FIG. 1 is an explanatory diagram of a measurement system for evaluating the characteristics of a reflective polarizing plate. [Figure 8] FIG. 11 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the light incidence angle. [Figure 9] FIG. 4 is a characteristic diagram showing transmittance characteristics with respect to the light incidence angle of the reflection axis of a reflective polarizing plate. [Figure 10] FIG. 11 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the light incidence angle. [Figure 11] FIG. 4 is a characteristic diagram showing transmittance characteristics with respect to the light incidence angle of the reflection axis of a reflective polarizing plate. [Figure 12] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 13] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 14] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 15] 1 is a layout diagram showing a main part of a space floating image display device according to an embodiment of the present invention; [Figure 16] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 17] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 18] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 19] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 20] 1 is an explanatory diagram for explaining a light source diffusion characteristic of a video display device. [Figure 21] FIG. 1 is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Figure 22] FIG. 1 is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Diagram 23] FIG. 1 is a cross-sectional view showing a configuration of a video display device. [Figure 24] FIG. 1 is an explanatory diagram for explaining the principle of occurrence of a ghost image in the prior art. [Diagram 25] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 26] FIG. 2 is a diagram illustrating an example of a display on a display device according to an embodiment of the present invention. [Figure 27] 11A to 11C are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Figure 28] 11A to 11C are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Figure 29] 11A to 11C are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Diagram 30] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Diagram 31] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Diagram 32] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Diagram 33] FIG. 13 is a diagram illustrating a method for setting a virtual light source. [Diagram 34] FIG. 11 is a configuration diagram showing an example of a method for detecting the position of a finger. [Diagram 35] FIG. 11 is a configuration diagram showing another example of a method for detecting the position of a finger. [Diagram 36] FIG. 11 is a configuration diagram showing another example of a method for detecting the position of a finger. [Figure 37] 13A to 13C are diagrams illustrating a method of assisting a touch operation by displaying input contents. [Figure 38]11A and 11B are diagrams illustrating a method of assisting a touch operation by highlighting input contents. [Figure 39] 11A and 11B are diagrams illustrating an example of a method for assisting a touch operation by vibration. [Diagram 40] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation by vibration. [Diagram 41] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation by vibration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the description of the embodiments, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. In addition, in all the drawings for explaining the present invention, the same reference numerals are given to parts having the same functions, and repeated explanations thereof may be omitted.

[0009] The following embodiment relates to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member.

[0010] According to the following embodiment, an image display device suitable for use in, for example, bank ATMs, station ticket vending machines, digital signage, etc., can be realized. For example, currently, touch panels are usually used in bank ATMs, station ticket vending machines, etc., but by using a transparent glass surface or a light-transmitting plate material, high-resolution image information can be displayed in a state of floating in space on the glass surface or light-transmitting plate material. At this time, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflective member, so that the light utilization efficiency is high, and ghost images that occur in addition to the main space floating image, which was a problem in the conventional retroreflective method, can be suppressed, and a clear space floating image can be obtained. In addition, a device including the light source of this embodiment can provide a novel and highly usable space floating image display device (space floating image display system) that can significantly reduce power consumption. In addition, for example, a space floating image display device for vehicles that can display a so-called unidirectional space floating image that can be viewed inside and / or outside the vehicle can be provided.

[0011] On the other hand, in the conventional technology, an organic EL panel or a liquid crystal panel is combined with a retroreflective member 151 as a high-resolution color display image source 150. In the conventional technology, the image light is diffused at a wide angle, so in addition to the reflected light normally reflected by the retroreflective member 151, ghost images 301 and 302 are generated by the image light obliquely incident on the retroreflective member 2a as shown in FIG. 24, which impairs the image quality of the floating in space image. Also, as shown in FIG. 23, in addition to the normal floating in space image 300, multiple first ghost images 301 and second ghost images 302 are generated. For this reason, the same floating in space image, which is a ghost image, can be observed by people other than the observer, which poses a major security problem. <Space-floating image display device 1>

[0012] Fig. 1 is a diagram showing an example of a usage form of a space-floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using Fig. 2, etc., but light with a narrow-angle directional characteristic and specific polarization is emitted from the image display device 1 as an image light beam, once enters the retroreflective member 2, is retroreflected and passes through a transparent member 100 (glass, etc.), and forms a real aerial image (space-floating image 3) on the outside of the glass surface.

[0013] In addition, in a store or the like, a space is divided by a show window (also called "window glass") 105, which is a light-transmitting member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit the floating image through such a transparent member and display it in one direction to the outside and / or inside of the store (space).

[0014] 1A, the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing the window glass 105 with a means for reflecting a specific polarized wave, it is possible to form an aerial image at a desired position inside the store.

[0015] FIG. 1(B) is a schematic block diagram showing the configuration of the above-mentioned image display device 1. The image display device 1 includes an image display unit that displays an original image of an aerial image, an image control unit that converts the input image to match the resolution of the panel, and an image signal receiving unit that receives an image signal. The image signal receiving unit is compatible with wired input signals such as HDMI (High-Definition Multimedia Interface) input and wireless input signals such as Wi-Fi (Wireless Fidelity), and functions independently as an image receiving and display device, and can also display image information from a tablet, smartphone, etc. Furthermore, by connecting a stick PC, etc., it can be equipped with capabilities such as calculation processing and image analysis processing.

[0016] Fig. 2 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention. The configuration of the space floating image display device will be described in more detail with reference to Fig. 2. As shown in Fig. 2(A), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light having a narrow angle diffusion characteristic.

[0017] The image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 (in the figure, the polarization separation member 101 is formed into a sheet shape and adhered to the transparent member 100) having a film that selectively reflects the image light of a specific polarization provided on a transparent member 100, and enters the retroreflective member 2. A λ / 4 plate 21 is provided on the image light incidence surface of the retroreflective member. The image light is polarized and converted from the specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, when entering the retroreflective member and when exiting. Here, the polarization separation member 101 that selectively reflects the image light of a specific polarization has a property of transmitting the polarized light of the other polarization that has been polarized and converted, so the image light of the specific polarization after polarization conversion passes through the polarization separation member 101. The image light that has passed through the polarization separation member 101 forms a space floating image 3, which is a real image, outside the transparent member 100.

[0018] The light forming the floating image 3 is a collection of light rays that converge from the retroreflective member 2 to the optical image of the floating image 3, and these light rays continue to travel straight even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is an image with high directivity, unlike the diffuse image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2, when a user views the floating image 3 from the direction of the arrow A, the floating image 3 is viewed as a bright image. However, when another person views the floating image 3 from the direction of the arrow B, the floating image 3 cannot be viewed as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people facing the user.

[0019] Depending on the performance of the retroreflective member 2, the polarization axis of the reflected image light may become misaligned. In this case, a part of the image light with the misaligned polarization axis is reflected by the above-mentioned polarization separation member 101 and returns to the display device 1. This light may be reflected again on the image display surface of the liquid crystal display panel 11 constituting the display device 1, generating a ghost image and degrading the image quality of the space-floating image. Therefore, in this embodiment, an absorbing polarizing plate 12 is provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorbing polarizing plate 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorbing polarizing plate 12, thereby suppressing the above-mentioned re-reflection. This makes it possible to prevent degradation of image quality due to the ghost image of the space-floating image.

[0020] The above-mentioned polarization separation member 101 may be formed, for example, of a reflective polarizing plate or a metal multilayer film that reflects a specific polarized wave.

[0021] Next, FIG. 2(B) shows the surface shape of a typical retroreflective member 2 manufactured by Nippon Carbide Industries Co., Ltd., which was used in this study. A light ray incident on the interior of regularly arranged hexagonal prisms is reflected by the walls and bottom of the hexagonal prisms and is emitted as retroreflected light in a direction corresponding to the incident light, and a real image, a floating image in space, is displayed based on the image displayed on the display device 1. The resolution of this floating image in space is largely dependent on the outer shape D and pitch P of the retroreflecting part of the retroreflecting member 2 shown in FIG. 2(B) as well as the resolution of the liquid crystal display panel 11. For example, when a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel is used, even if one pixel (one triplet) is about 80 μm, if the diameter D of the retroreflecting part is 240 μm and the pitch is 300 μm, one pixel of the floating image in space is equivalent to 300 μm. For this reason, the effective resolution of the floating image in space is reduced to about 1 / 3. Therefore, in order to make the resolution of the spatial floating image equal to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective material and the pixels of the liquid crystal display panel, it is preferable to design the pitch ratio of each to be a different integer multiple of one pixel. In addition, it is preferable to arrange the shape so that none of the sides of the retroreflective portion overlaps with any of the sides of one pixel of the liquid crystal display panel.

[0022] On the other hand, in order to manufacture retroreflective members at low cost, it is recommended to use a roll press method. Specifically, this is a method in which the retroreflective parts are aligned and shaped on a film, in which the inverse shape of the shape to be shaped is formed on the roll surface, an ultraviolet curing resin is applied onto a base material for fixing, and the film is passed between the rolls to form the required shape, and then the resin is irradiated with ultraviolet light to cure it, thereby obtaining the retroreflective member 2 of the desired shape. <<How to install the space floating image display device>>

[0023] Next, a method of installing the space-floating image display device will be described. The installation method of the space-floating image display device can be freely changed depending on the usage form. FIG. 3A is a diagram showing an example of a method of installing the space-floating image display device. The space-floating image display device shown in FIG. 3A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 3A, the space-floating image display device is installed so that the transparent member 100 faces upward, and the space-floating image 3 is formed above the space-floating image display device.

[0024] Fig. 3B is a diagram showing another example of a method of installing the space-floating image display device. The space-floating image display device shown in Fig. 3B is installed vertically so that the surface on which the space-floating image 3 is formed faces the side (toward the user 230). That is, in Fig. 3B, the space-floating image display device is installed so that the transparent member 100 faces the side, and the space-floating image 3 is formed on the side of the space-floating image display device (toward the user 230). <<Configuration of the space floating image display device>>

[0025] Next, a description will be given of the configuration of the space-floating image display device 1000. Fig. 3C is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

[0026] The space floating image display device 1 includes a retroreflective unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power source 1106, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, a video signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc.

[0027] Each component of the space floating image display device 1000 is housed in a housing 1190. Note that the imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0028] The retroreflective portion 1101 in Fig. 3C corresponds to the retroreflective member 2 in Fig. 2. The retroreflective portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected by the retroreflective portion 1101, the light outputted to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0029] Image display unit 1102 in Fig. 3C corresponds to liquid crystal display panel 11 in Fig. 2. Light source 1105 in Fig. 3C corresponds to light source device 13 in Fig. 2. Moreover, image display unit 1102, light guide 1104, and light source 1105 in Fig. 3C correspond to display device 1 in Fig. 2.

[0030] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 described later. The video display unit 1102 corresponds to the liquid crystal display panel 11 in Fig. 2. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0031] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power source 1106 converts an AC current input from the outside into a DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each unit in the space floating image display device 1000.

[0032] The light guide 1104 guides the light generated by the light source 1105 and irradiates the light to the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can be called a backlight of the image display unit 1102. There are various types of combination of the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0033] The aerial operation detection sensor 1351 is a sensor that detects an operation of the floating-in-space image 3 by a finger of the user 230. The aerial operation detection sensor 1351 senses, for example, a range that overlaps with the entire display range of the floating-in-space image 3. Note that the aerial operation detection sensor 1351 may only sense a range that overlaps with at least a portion of the display range of the floating-in-space image 3.

[0034] Specific examples of the aerial operation detection sensor 1351 include a distance sensor using invisible light such as infrared light, an invisible light laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may be configured to detect coordinates on a two-dimensional plane by combining a plurality of sensors. The aerial operation detection sensor 1351 may be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) or an image sensor.

[0035] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect a touch operation or the like performed by a user with a finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0036] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and performs operations such as determining whether or not the finger of the user 230 has touched an object in the floating in space image 3 and calculating the position (contact position) where the finger of the user 230 has touched the object based on the sensing signal. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array). Some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0037] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally separately from the space-floating image display device 1000. When provided separately from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or image signal transmission path.

[0038] Also, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the air-floating image display device 1000 without the aerial operation detection function is used as the main body, and only the aerial operation detection function can be added as an option. Also, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the air-floating image display device 1000 may be used. In cases in which it is desired to more freely arrange the aerial operation detection sensor 1351 relative to the installation position of the air-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0039] The imaging unit 1180 is a camera having an image sensor, and captures the space near the floating-in-space image 3 and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180 or an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in the detection process of the touch operation of the floating-in-space image 3 by the user 230.

[0040] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that detects whether or not an object has intruded into a plane (intrusion detection plane) including the display surface of the floating image 3, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object (e.g., a user's finger) that has not intruded into the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0041] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the captured images of the multiple imaging units 1180 and object depth information by the depth sensor. These pieces of information and various pieces of information such as the distance between the object and the intrusion detection plane are used for various display controls for the floating in space image 3.

[0042] Furthermore, without using the aerial operation detection sensor 1351, the aerial operation detection section 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging section 1180.

[0043] Also, the imaging unit 1180 may capture an image of the face of the user 230 operating the floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not a third person is standing around or behind the user 230 operating the floating image 3 and peeking at the operation of the user 230 on the floating image 3, the imaging unit 1180 may capture an image of a range including the user 230 operating the floating image 3 and the surrounding area of ​​the user 230.

[0044] The operation input unit 1107 is, for example, an operation button or a light receiving unit of a remote controller, and inputs a signal for an operation different from the air operation (touch operation) by the user 230. In addition to the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may be used by, for example, an administrator to operate the space floating image display device 1000.

[0045] The video signal input unit 1131 connects to an external video output device and inputs video data. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio output unit 1140 can output audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may also output built-in operation sounds and error warning sounds.

[0046] The non-volatile memory 1108 stores various data used by the space-floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations to be displayed on the space-floating image 3, display icons, data of objects to be operated by user operations, layout information, etc. The memory 1109 stores image data to be displayed as the space-floating image 3, data for controlling the device, etc.

[0047] The control unit 1110 controls the operation of each connected unit. The control unit 1110 may also perform calculation processing based on information acquired from each unit in the space floating image display device 1000 in cooperation with a program stored in the memory 1109. The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless interface. Various data such as video data, image data, and audio data are transmitted and received by communication via the communication unit 1132.

[0048] The storage unit 1170 is a storage device that records various types of data & information such as video data, image data, audio data, etc. In the storage unit 1170, for example, various types of information such as video data, image data, audio data, etc. may be recorded in advance at the time of product shipment. In addition, the storage unit 1170 may record various types of information such as various types of data such as video data, image data, audio data, etc. acquired from an external device, an external server, etc. via the communication unit 1132.

[0049] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflective unit 1101. The video data, image data, etc. of the display icons and objects for the user to operate, which are displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0050] Layout information of display icons and objects displayed as the space floating image 3, various metadata information related to the objects, and the like are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0051] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. The video control unit 1160 performs control of video switching, such as determining which video signal to input to the video display unit 1102, between the video signal to be stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131, for example.

[0052] In addition, the image control unit 1160 may generate a superimposed image signal by superimposing the image signal to be stored in memory 1109 and the image signal input from the image signal input unit 1131, and input the superimposed image signal to the image display unit 1102, thereby performing control to form a composite image as a floating-in-space image 3.

[0053] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing for enlarging, reducing, transforming, etc. an image, brightness adjustment processing for changing the luminance, contrast adjustment processing for changing the contrast curve of the image, and Retinex processing for decomposing an image into light components and changing the weighting of each component.

[0054] Furthermore, the video control unit 1160 may perform special effect video processing or the like for assisting the aerial operation (touch operation) of the user 230 on the video signal input to the video display unit 1102. The special effect video processing is performed, for example, based on the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 or the captured image of the user 230 by the imaging unit 1180.

[0055] As explained above, the space-floating image display device 1000 is equipped with various functions. However, the space-floating image display device 1000 does not need to have all of these functions, and can have any configuration as long as it has the function of forming the space-floating image 3. <Space-floating image display device 2>

[0056] 4 is a diagram showing another example of the main part configuration of a space floating image display device according to an embodiment of the present invention. The image display element 11 constituting the display device 1 includes a light source device 13 that generates light of a specific polarized wave having a diffusion characteristic that forms a narrow angle with the liquid crystal display panel 11, and is composed of a liquid crystal display panel having a screen size ranging from a small one having a screen size of about 5 inches to a large one having a screen size of over 80 inches, and a polarization separation member 101 such as a reflective polarizing plate is provided on the surface of the folding mirror 22, and the image light from the liquid crystal display panel 11 is reflected toward the retroreflective member 2. The image light of the specific polarized wave from the display device 1 is reflected by a film (a sheet 101 is adhered in the figure) that selectively reflects the image light of the specific polarized wave provided on a transparent member 100, and enters the retroreflective member 2.

[0057] A λ / 4 plate 21 is provided on the light incidence surface of the retroreflective member, and the image light is polarized and converted by passing it twice to convert a specific polarized wave into the other polarized wave, and then the image light is transmitted through the polarization separation member 101, and a real image, a floating image 3, is displayed on the outside of the transparent member 100. An absorbing polarizing plate is provided on the external light incidence surface of the transparent member 100. In the above-mentioned polarization separation member 101, the polarization axis becomes uneven due to retroreflection, so a part of the image light is reflected and returns to the display device 1. This light is reflected again on the image display surface of the liquid crystal display panel 11 constituting the display device 1, generating a ghost image and significantly degrading the image quality of the floating image in space. Therefore, in this embodiment, an absorbing polarizing plate 12 is provided on the image display surface of the display device 1, and the image light is transmitted, and the reflected light is absorbed to prevent degradation of image quality due to the ghost image of the floating image in space. Furthermore, in order to reduce degradation of image quality due to sunlight or illumination outside the set, it is preferable to provide an absorbing polarizing plate 12 on the surface of the transparent member 100. The polarized light separating member 101 is formed from a reflective polarizing plate or a metal multilayer film that reflects a specific polarized wave.

[0058] Next, in order to sense the distance and position relationship between the object and the sensor 44 for the space floating image obtained by the space floating image display device described above, the sensor 44 having the TOF (Time of Fly) function is arranged in multiple layers as shown in FIG. 5, and it becomes possible to sense the coordinate in the depth direction and the moving direction and moving speed of the object in addition to the coordinate in the planar direction of the object. In order to read the two-dimensional distance and position, multiple combinations of ultraviolet light emitting units and light receiving units are arranged in a straight line, light from the light emitting point is irradiated on the object, and the reflected light is received by the light receiving unit. The distance to the object becomes clear by the product of the difference between the time of light emission and the time of light reception and the speed of light. In addition, the coordinates on the plane can be read from the coordinates at the part where the difference between the time of light emission and the time of light reception is the smallest with multiple light emitting units and light receiving units. As described above, it is also possible to obtain three-dimensional coordinate information by combining the coordinates of the object on the plane (two dimensions) and multiple sensors described above.

[0059] Furthermore, a method for obtaining a three-dimensional space-floating image as the above-mentioned space-floating image display device will be described with reference to FIG. 6. FIG. 6 is an explanatory diagram of the principle of three-dimensional image display used in the space-floating image display device. A horizontal lenticular lens is arranged in accordance with the pixels of the image display screen of the liquid crystal display panel 11 of the display device 1 shown in FIG. 4. As a result, in order to display motion parallax from three directions, P1, P2, and P3 in the horizontal direction of the screen as shown in FIG. 6, images from the three directions are treated as one block for every three pixels, and image information from the three directions is displayed for each pixel, and the light emission direction is controlled by the action of the corresponding lenticular lens (shown by vertical lines in FIG. 6) to separate and emit the light in three directions. As a result, a three-parallax stereoscopic image can be displayed. <Reflective polarizing plate>

[0060] In the space floating image display device of this embodiment, the polarization separation member 101 is used to improve the contrast performance, which determines the image quality, more than a general half mirror. The characteristics of a reflective polarizer will be described as an example of the polarization separation member 101 of this embodiment. FIG. 7 is an explanatory diagram of a measurement system for evaluating the characteristics of a reflective polarizer. The transmission characteristics and reflection characteristics of the reflective polarizer in FIG. 7 with respect to the incident angle of light from a direction perpendicular to the polarization axis of the reflective polarizer are shown in FIGS. 8 and 9 as V-AOI. Similarly, the transmission characteristics and reflection characteristics of the reflective polarizer with respect to the incident angle of light from a direction horizontal to the polarization axis of the reflective polarizer are shown in FIGS. 10 and 11 as H-AOI.

[0061] As shown in Figures 8 and 9, the reflective polarizing plate with a grid structure has poor characteristics for light coming from a direction perpendicular to the polarization axis. For this reason, specifications along the polarization axis are desirable, and the light source of this embodiment capable of emitting the image light from the liquid crystal display panel at a narrow angle is an ideal light source. Similarly, the horizontal characteristics also deteriorate for light coming from an oblique angle. Taking the above characteristics into consideration, a configuration example of this embodiment will be described below in which a light source capable of emitting the image light from the liquid crystal display panel at a narrower angle is used as the backlight for the liquid crystal display panel. This makes it possible to provide a high-contrast floating image. <Display device>

[0062] Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes its light source, and in Fig. 12, the light source device 13 is shown together with the liquid crystal display panel as an exploded perspective view.

[0063] As shown by the arrow 30 in Fig. 12, this liquid crystal display panel (image display element 11) obtains an illumination light beam with a narrow-angle diffusion characteristic, i.e., a strong directivity (straightness) and a characteristic similar to that of a laser beam with a polarization plane aligned in one direction, by using light from a light source device 13, which is a backlight device, and the image light modulated according to an input image signal is reflected by a retroreflective member 2 and transmitted through a transparent member 100 to form a real image, a floating image in space. (See Fig. 1). Also, in Fig. 12, the liquid crystal display panel 11 constituting the display device 1 is further configured to include a light direction conversion panel 54 for controlling the directivity characteristic of the light beam emitted from the light source device 13, and a narrow-angle diffusion plate (not shown) as necessary. That is, polarizing plates are provided on both sides of the liquid crystal display panel 11, and the image light of a specific polarization is output by modulating the light intensity according to the image signal (see the arrow 30 in Fig. 12). As a result, the desired image is projected as highly directional (straight-line) light of a specific polarization via the light direction changing panel 54 toward the retroreflective member 2, and after being reflected by the retroreflective member 2, it is transmitted toward the eyes of a monitor outside the store (space) to form the space floating image 3. Note that a protective cover 50 (see Figures 13 and 14) may be provided on the surface of the light direction changing panel 54 described above.

[0064] In this embodiment, in order to improve the utilization efficiency of the light flux 30 emitted from the light source device 13 and to significantly reduce power consumption, in the display device 1 including the light source device 13 and the liquid crystal display panel 11, the light from the light source device 13 (see the arrow 30 in FIG. 12) is projected toward the retroreflective member 2, and after being reflected by the retroreflective member 2, the directivity can be controlled so that the floating image is formed at a desired position by a transparent sheet (not shown) provided on the surface of the transparent member 100 (such as the window glass 105). Specifically, this transparent sheet controls the imaging position of the floating image while giving high directivity by optical components such as a Fresnel lens or a linear Fresnel lens. In this way, the image light from the display device 1 can efficiently reach the observer outside the show window 105 (for example, the sidewalk) with high directivity (straightness) like laser light, and as a result, it is possible to display a high-quality floating image with high resolution and significantly reduce the power consumption of the display device 1 including the LED element 201 of the light source device 13. <Display device example 1>

[0065] FIG. 13 shows an example of a specific configuration of the display device 1. In FIG. 13, a liquid crystal display panel 11 and a light direction conversion panel 54 are arranged on the light source device 13 of FIG. 12. This light source device 13 is formed, for example, from plastic on the case shown in FIG. 12, and is configured by storing an LED element 201 and a light guide 203 inside. As shown in FIG. 12, the end surface of the light guide 203 has a shape in which the cross-sectional area gradually increases toward the opposite side to the light receiving part in order to convert the divergent light from each LED element 201 into a substantially parallel light beam, and is provided with a lens shape that has an effect of gradually decreasing the divergence angle by multiple total reflections during propagation inside. The liquid crystal display panel 11 constituting the display device 1 is attached to the upper surface. In addition, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202, which implements their control circuits, are attached to one side of the case of the light source device 13 (the left end face in this example), and a heat sink, which is a member for cooling heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED board 202.

[0066] In addition, the liquid crystal display panel frame (not shown) attached to the upper surface of the case of the light source device 13 is configured by attaching the liquid crystal display panel 11 attached to the frame, and further by attaching an FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light based on a control signal from a control circuit (not shown) constituting an electronic device together with the LED element 201, which is a solid light source. At this time, the generated image light has a narrow diffusion angle and contains only specific polarization components, so that a new image display device that is similar to a surface-emitting laser image source driven by an image signal can be obtained. Note that, at present, it is technically and safety impossible to obtain a laser light beam of the same size as the image obtained by the above-mentioned display device 1 using a laser device. Therefore, in this embodiment, light similar to the above-mentioned surface-emitting laser image light is obtained from a light beam from a general light source equipped with an LED element, for example.

[0067] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 13 as well as FIG.

[0068] 13 and 14 are cross-sectional views, and only one of the LED elements 201 constituting the light source is shown, and this is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion of the end surface of the light guide and the LED element are attached while maintaining a predetermined positional relationship. Each of the light guides 203 is formed of a light-transmitting resin such as acrylic. The LED light-receiving surface at the end of the light guide has an outer peripheral surface of a cone convex shape obtained by rotating a parabolic cross section, and at the top of the LED light-receiving surface, a concave portion having a convex portion (i.e., a convex lens surface) formed in the center is formed, and the center of the flat portion has a convex lens surface protruding outward (or may be a concave lens surface recessed inward) (not shown). The outer shape of the light receiving part of the light guide to which the LED element 201 is attached is a parabolic shape forming a conical outer surface, and is set within an angle range within which the light emitted from the LED element in the peripheral direction can be totally reflected within it, or a reflective surface is formed.

[0069] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed such that the LED elements 201 on the surface are positioned in the center of the recessed portion described above with respect to the LED collimator (light-receiving end surface 203a).

[0070] According to this configuration, the shape of light-receiving end surface 203a of light guide 203 makes it possible to extract the light emitted from LED element 201 as substantially parallel light, thereby improving the efficiency of use of the generated light.

[0071] As described above, the light source device 13 is configured by mounting a light source unit in which a plurality of LED elements 201 serving as light sources are arranged on the light receiving end surface 203a serving as a light receiving section provided on the end surface of the light guide 203, and the divergent light beam from the LED elements is converted into substantially parallel light by the lens shape of the light receiving end surface 203a of the light guide end surface, and the light is guided inside the light guide 203 (in a direction parallel to the drawing) as shown by the arrow, and is emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11 arranged substantially parallel to the light guide (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of the light beam direction conversion means according to the shape inside or on the surface of the light guide. The above-mentioned light beam direction conversion means 204 emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 arranged approximately parallel to the light guide (in a direction perpendicular to the front of the drawing) by changing the shape of the light guide surface or providing a portion with a different refractive index inside the light guide. At this time, when the brightness of the screen center and the screen periphery are compared while facing the liquid crystal display panel 11 directly at the center of the screen and placing the viewpoint at the same position as the diagonal dimension of the screen, if the relative brightness ratio is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

[0072] 13 is a cross-sectional layout diagram for explaining the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including the above-mentioned light guide 203 and LED element 201. In Fig. 13, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface thereof is attached liquid crystal display panel 11 equipped with polarizing plates on the light source light entrance surface and image light exit surface.

[0073] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one side polarized wave (e.g., P wave) 212 of the natural light beam 210 emitted from the LED element 201, and the reflected light beam is reflected by a reflective sheet 205 provided on one surface (the lower surface in the figure) of the light guide 203, and directed again toward the liquid crystal display panel 52. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, thereby converting the reflected light beam from P polarized light to S polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam, the light intensity of which is modulated by the video signal in the liquid crystal display panel 11 (arrow 213 in Figure 13), enters the retroreflective member 2 and, after reflection, passes through the window glass 105, as shown in Figure 1, to obtain a real image, that is, a floating image in space, inside or outside the store (space).

[0074] Fig. 14, like Fig. 13, is a cross-sectional layout diagram for explaining the configuration and action of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED element 201. Light source device 13 is also similarly composed of light guide 203 formed of, for example, plastic and provided with light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface thereof is attached liquid crystal display panel 11 equipped with polarizing plates on the light source light entrance surface and image light exit surface as an image display element.

[0075] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one polarized wave (for example, S wave) 211 of the natural light beam 210 emitted from the LED light source 201, which is reflected by a reflective sheet 205 provided on one surface (the lower surface in the figure) of the light guide 203, and travels again toward the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, thereby converting the reflected light beam from S polarized light to P polarized light, thereby improving the efficiency of use of the light source light as image light. The image light beam, which has been intensity-modulated by the image signal in the liquid crystal display panel 11 (arrow 214 in Figure 14), enters the retroreflective member 2 and, after reflection, passes through the window glass 105 as shown in Figure 1, to obtain a real image, that is, a floating image in space, inside or outside the store (space).

[0076] In the light source device shown in Fig. 13 and Fig. 14, in addition to the action of the polarizing plate provided on the light incident surface of the corresponding liquid crystal display panel 11, the polarized component on one side is reflected by the reflective polarizing plate, so that the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizing plate multiplied by the reciprocal of the cross transmittance obtained by the two polarizing plates attached to the liquid crystal display panel. This results in high contrast performance. In fact, it was confirmed by experiments that the contrast performance of the displayed image was improved by more than 10 times. As a result, a high-quality image was obtained that was comparable to that of a self-luminous organic EL. <Display device example 2>

[0077] Fig. 15 shows another example of a specific configuration of the display device 1. The light source device 13 in Fig. 15 is similar to the light source device in Fig. 17 and the like. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, and the like in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. In addition, an LED board 102 on which LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and a control circuit for the LED elements are mounted are attached to one side of the case of the light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer surface of the LED board 102 (see also Figs. 17, 18, and the like).

[0078] The liquid crystal display panel frame attached to the top surface of the case is configured to have the liquid crystal display panel 11 attached to the frame, and further, an FPC (Flexible Printed Circuits) 403 (see FIG. 7) electrically connected to the liquid crystal display panel 11, etc. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light based on a control signal from a control circuit (not shown here) constituting the electronic device, together with the LED elements 14a and 14b, which are solid-state light sources. <Display device example 3>

[0079] Next, another example of the specific configuration of the display device 1 will be described with reference to Fig. 16. The light source device of this display device 1 converts a divergent light beam of natural light (a mixture of P-polarized and S-polarized waves) emitted from an LED into a substantially parallel light beam by an LED collimator 18, and reflects it toward the liquid crystal display panel 11 by a reflective light guide 304. The reflected light is incident on a wave plate and a reflective polarizing plate 49 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (e.g., S-polarized wave) is reflected by the reflective polarizing plate, the phase of which is converted by the wave plate, the light returns to the reflecting surface, and the light passes through the phase difference plate again to be converted into a polarized wave (e.g., P-polarized wave) that transmits through the reflective polarizing plate.

[0080] As a result, the natural light from the LEDs is aligned to a specific polarization (e.g., P polarization), enters the liquid crystal display panel 11, and is luminance-modulated according to the video signal to display an image on the panel surface. As in the above example, a plurality of LEDs constituting the light source are shown (however, since FIG. 16 shows a vertical section, only one LED is shown), and these are attached at a predetermined position relative to the LED collimator 18. Each of the LED collimators 18 is formed of, for example, a light-transmitting resin such as acrylic or glass. The LED collimator 18 has a cone-shaped outer peripheral surface obtained by rotating a parabolic section, and at the top of the LED collimator 18, a concave portion having a convex portion (i.e., a convex lens surface) formed in the center is formed. The center of the flat portion of the LED collimator 18 has a convex lens surface protruding outward (or may be a concave lens surface recessed inward). The parabolic surface forming the cone-shaped outer peripheral surface of the LED collimator 18 is set within an angle range in which the light emitted from the LEDs in the peripheral direction can be totally reflected inside the parabolic surface, or a reflective surface is formed.

[0081] The above configuration is the same as that of the light source device of the image display device shown in Fig. 17, Fig. 18, etc. Furthermore, the light converted into approximately parallel light by the LED collimator 15 shown in Fig. 16 is reflected by the reflective light guide 304, and a specific polarized light is transmitted by the action of the reflective polarizing plate 49, while the other polarized light is transmitted through the light guide 304 again and reflected by the reflector 271 provided on the other surface of the light guide that is not in contact with the liquid crystal display panel 11. At this time, the light is polarized and converted by passing twice through the retardation plate (lambda / 4 plate) 270 arranged between the reflector 271 and the liquid crystal display panel 11, and is transmitted through the light guide 304 again and the reflective polarizing plate 49 provided on the opposite surface, and is made incident on the liquid crystal display panel 11 with the polarization direction aligned. As a result, all the light from the light source can be used, and the light utilization efficiency is doubled.

[0082] In a conventional TV set, the light emitted from the liquid crystal display panel has similar diffusion characteristics in both the horizontal direction of the screen (shown by the X-axis in FIG. 22(a)) and the vertical direction of the screen (shown by the Y-axis in FIG. 22(b)). In contrast, the diffusion characteristics of the light flux emitted from the liquid crystal display panel of this embodiment are, for example, as shown in Example 1 of FIG. 22, 1 / 5 of the conventional 62 degrees when the viewing angle at which the luminance is 50% of that when viewed from the front (angle 0 degrees) is set to 13 degrees. Similarly, the vertical viewing angle is made uneven from top to bottom, and the reflection angle and the area of ​​the reflection surface of the reflective light guide are optimized to suppress the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light heading in the monitoring direction is significantly improved compared to conventional liquid crystal TVs, and the luminance is more than 50 times higher.

[0083] Furthermore, if the viewing angle characteristics shown in Example 2 of FIG. 22 are used, the viewing angle at which the luminance is 50% of that when viewed from the front (angle 0 degrees) is set to 5 degrees, which is 1 / 12 of the conventional 62 degrees. Similarly, the vertical viewing angle is equal from top to bottom, and the reflection angle and the area of ​​the reflection surface of the reflective light guide are optimized to suppress the viewing angle to about 1 / 12 of the conventional viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, and the luminance is more than 100 times higher. As described above, by setting the viewing angle to a narrow angle, the amount of light flux directed toward the monitoring direction can be concentrated, so the efficiency of light use is significantly improved. As a result, even if a conventional LCD display panel for TVs is used, a significant improvement in luminance can be achieved with similar power consumption by controlling the light diffusion characteristics of the light source device, and an image display device compatible with information display systems facing bright outdoors can be obtained.

[0084] When using a large LCD panel, the overall brightness of the screen can be improved by directing the light around the periphery of the screen inwards so that it faces the observer when he or she is facing directly at the center of the screen. Figure 20 shows the convergence angle of the long and short sides of the panel when the observer's distance from the panel L and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen vertically long, the convergence angle can be set to match the short side; for example, when a 22" panel is used vertically and the monitoring distance is 0.8m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed towards the observer.

[0085] Similarly, when monitoring with a 15" panel in portrait orientation and a monitoring distance of 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the observer. As described above, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the observer who is in the optimum position to monitor the center of the screen depending on the size of the LCD panel and whether it is used in portrait or landscape orientation.

[0086] The basic configuration is as shown in Figure 16, in which a light beam with a narrow-angle directional characteristic is incident on a liquid crystal display panel 11 from a light source device, and the brightness is modulated according to a video signal, so that the video information displayed on the screen of the liquid crystal display panel 11 is reflected by a retroreflective member, and the resulting spatial floating image is displayed indoors or outdoors via a transparent member 100. <Light source device example 1>

[0087] Next, the configuration of the optical system including the light source device housed in the case will be described in detail with reference to FIG. 17 as well as FIGS. 18(a) and (b).

[0088] 17 and 18 show LEDs 14a and 14b constituting the light source, which are attached at predetermined positions relative to the LED collimator 15. Each of the LED collimators 15 is formed of a light-transmitting resin such as acrylic. As shown in FIG. 18(b), the LED collimator 15 has a cone-shaped outer peripheral surface 156 obtained by rotating a parabolic cross section, and has a concave portion 153 at the top of the convex portion 157 formed in the center (i.e., a convex lens surface). The center of the flat portion has a convex lens surface 154 protruding outward (or a concave lens surface recessed inward). The parabolic surface 156 forming the cone-shaped outer peripheral surface of the LED collimator 15 is set within an angle range that allows the light emitted from the LEDs 14a and 14b in the peripheral direction to be totally reflected therein, or a reflective surface is formed thereon.

[0089] The LEDs 14a and 14b are disposed at predetermined positions on the surface of the LED substrate 102, which is the circuit substrate. The LED substrate 102 is disposed and fixed to the LED collimator 15 such that the LEDs 14a and 14b on the surface are located at the center of the recess 153.

[0090] According to this configuration, the light emitted from the LED 14a or 14b by the above-mentioned LED collimator 15, particularly the light emitted from the central portion toward the upper side (to the right in the figure), is collected and made parallel by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. The light emitted from the other portions toward the periphery is reflected by the parabolic surface that forms the conical outer circumferential surface of the LED collimator 15, and is similarly collected and made parallel. In other words, the LED collimator 15 that has a convex lens in its center and a parabolic surface formed on its periphery makes it possible to extract almost all of the light generated by the LED 14a or 14b as parallel light, and thus makes it possible to improve the utilization efficiency of the generated light.

[0091] A polarization conversion element 21 is provided on the light emission side of the LED collimator 15. As is clear from Fig. 18, this polarization conversion element 21 is configured by combining a columnar translucent member having a parallelogram cross section (hereinafter, parallelogram column) and a columnar translucent member having a triangular cross section (hereinafter, triangular column), and arranging them in an array in parallel to a plane perpendicular to the optical axis of the parallel light from the LED collimator 15. Furthermore, a polarizing beam splitter (hereinafter, abbreviated as "PBS film") 211 and a reflective film 212 are alternately provided at the interface between adjacent translucent members arranged in the array, and a λ / 2 phase plate 213 is provided on the emission surface from which the light incident on the polarization conversion element 21 and transmitted through the PBS film 211 is emitted.

[0092] 18(a) is provided on the exit surface of the polarization conversion element 21. That is, the light emitted from the LED 14a or 14b is converted into parallel light by the action of the LED collimator 15, enters the synthesizing diffusion block 16, is diffused by the texture 161 on the exit side, and then reaches the light guide 17.

[0093] The light guide 17 is a rod-shaped member made of a translucent resin such as acrylic and having a substantially triangular cross section (see Figure 18(b)), and, as is clear from Figure 17, is provided with a light guide light incident portion (surface) 171 facing the exit surface of the synthetic diffusion block 16 via a first diffuser plate 18a, a light guide light reflecting portion (surface) 172 forming an inclined surface, and a light guide light exit portion (surface) 173 facing the liquid crystal display panel 11, which is the liquid crystal display element, via a second diffuser plate 18b.

[0094] 17, a number of reflective surfaces 172a and connecting surfaces 172b are alternately formed in a sawtooth shape on the light guide light reflecting portion (surface) 172 of the light guide 17. The reflective surface 172a (a line segment sloping upward to the right in the figure) forms an angle αn (n: natural number, for example, 1 to 130 in this example) with the horizontal plane indicated by the dashed line in the figure, and as an example, αn is set to 43 degrees or less (but 0 degree or more) here.

[0095] The light guide light incident portion (surface) 171 is formed in a curved convex shape inclined toward the light source side. Accordingly, the parallel light from the exit surface of the synthetic diffusion block 16 is diffused and incident via the first diffusion plate 18a, and as is clear from the figure, the parallel light is bent (deflected) slightly upward by the light guide light incident portion (surface) 171 and reaches the light guide light reflecting portion (surface) 172, where it is reflected and reaches the liquid crystal display panel 11 provided on the exit surface at the top of the figure.

[0096] According to the display device 1 described above in detail, it is possible to improve the light utilization efficiency and the uniform illumination characteristics, and at the same time, to manufacture the modularized S-polarized light source device in a small size and at low cost. In the above description, the polarization conversion element 21 is described as being attached after the LED collimator 15, but the present invention is not limited to this, and the same action and effect can be obtained by providing it in the optical path leading to the liquid crystal display panel 11.

[0097] In addition, in the light guide light reflection portion (surface) 172, a large number of reflection surfaces 172a and connection surfaces 172b are alternately formed in a sawtooth shape, and the illumination light beam is totally reflected on each reflection surface 172a and directed upward, and furthermore, a narrow-angle diffusion plate is provided in the light guide light exit portion (surface) 173 to make the illumination light beam into a substantially parallel diffused light beam, which enters the light direction conversion panel 54 that controls the directional characteristics, and then enters the liquid crystal display panel 11 from an oblique direction. In this embodiment, the light direction conversion panel 54 is provided between the light guide exit portion (surface) 173 and the liquid crystal display panel 11, but the same effect can be obtained by providing the light direction conversion panel 54 on the exit surface of the liquid crystal display panel 11. <Light source device example 2>

[0098] Another example of the configuration of the optical system of the light source device 13 and the like is shown in FIG. 19. As in the example shown in FIG. 18, a plurality of (two in this example) LEDs 14a and 14b constituting the light source are shown, and these are attached at predetermined positions relative to the LED collimator 15. Each of the LED collimators 15 is formed of a light-transmitting resin such as acrylic. As in the example shown in FIG. 18, the LED collimator 15 has a cone-shaped outer peripheral surface 156 obtained by rotating a parabolic cross section, and at the top of the LED collimator 15, a concave portion 153 having a convex portion (i.e., a convex lens surface) 157 formed in the center thereof is formed. Also, at the center of the flat portion, a convex lens surface 154 protruding outward (or a concave lens surface recessed inward) is formed. The parabolic surface 156 forming the cone-shaped outer peripheral surface of the LED collimator 15 is set within an angle range in which the light emitted from the LED 14a in the peripheral direction can be totally reflected therein, or a reflective surface is formed thereon.

[0099] The LEDs 14a and 14b are disposed at predetermined positions on the surface of the LED substrate 102, which is the circuit substrate. The LED substrate 102 is disposed and fixed to the LED collimator 15 such that the LEDs 14a and 14b on the surface are located at the center of the recess 153.

[0100] According to this configuration, the light emitted from the LED 14a or 14b by the above-mentioned LED collimator 15, particularly the light emitted from the central portion toward the upper side (to the right in the figure), is collected and made parallel by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. The light emitted from the other portions toward the periphery is reflected by the parabolic surface that forms the conical outer circumferential surface of the LED collimator 15, and is similarly collected and made parallel. In other words, the LED collimator 15 that has a convex lens in its center and a parabolic surface formed on its periphery makes it possible to extract almost all of the light generated by the LED 14a or 14b as parallel light, and thus makes it possible to improve the utilization efficiency of the generated light.

[0101] A light guide 170 is provided on the light emission side of the LED collimator 15 via a first diffusion plate 18a. The light guide 170 is a rod-shaped member made of a light-transmitting resin such as acrylic and having a substantially triangular cross section (see FIG. 19(a)), and as is clear from FIG. 19(a), includes a light guide light entrance portion (surface) 171 facing the emission surface of the diffusion block 16 via the first diffusion plate 18a, a light guide light reflection portion (surface) 172 forming an inclined surface, and a light guide light emission portion (surface) 173 facing the liquid crystal display panel 11, which is a liquid crystal display element, via a reflective polarizing plate 200.

[0102] If this reflective polarizer 200 is selected to have, for example, the property of reflecting P-polarized light (and transmitting S-polarized light), it will reflect the P-polarized light in the natural light emitted from the LED light source, pass through the λ / 4 plate 202 provided in the light guide light reflecting section 172 shown in Figure 19(b), be reflected by the reflective surface 201, and pass through the λ / 4 plate 202 again to be converted into S-polarized light, and all the light beams entering the liquid crystal display panel 11 will be unified into S-polarized light.

[0103] Similarly, if a material having the property of reflecting S-polarized light (transmitting P-polarized light) is selected as reflective polarizer 200, it will reflect the S-polarized light in the natural light emitted from the LED light source, pass through λ / 4 plate 202 provided on light guide light reflector 172 shown in Fig. 19(b), be reflected by reflecting surface 201, and pass through λ / 4 plate 202 again to be converted into P-polarized light, and all the light beams entering liquid crystal display panel 52 will be unified into P-polarized light. Polarization conversion can also be achieved with the configuration described above. <Light source device example 3>

[0104] Another example of the configuration of the optical system of the light source device, etc., will be described with reference to Fig. 16. In the third example, as shown in Fig. 16, a divergent light beam of natural light (a mixture of P-polarized light and S-polarized light) from the LED 102 is converted into a substantially parallel light beam by the collimator lens 18, and is reflected toward the liquid crystal display panel 11 by the reflective light guide 304. The reflected light is incident on the reflective polarizing plate 206 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (e.g., S-polarized wave) is reflected by the reflective polarizing plate 206, passes through a surface connecting the reflecting surfaces of the light guide 304, is reflected by a reflector 271 arranged facing the opposite surface of the light guide 304, passes twice through a phase plate (λ / 4 wavelength plate) 270, and is polarized and converted. The light passes through the light guide and the reflective polarizing plate, enters the liquid crystal display panel 11, and is modulated into image light. At this time, by combining a specific polarized wave with the polarized converted polarization plane, the light utilization efficiency becomes twice as high as usual, and the degree of polarization (extinction ratio) of the reflective polarizing plate is also included in the extinction ratio of the entire system, so that the contrast ratio of the information display system is significantly improved by using the light source device of this embodiment.

[0105] As a result, the natural light from the LED is aligned to a specific polarization (e.g., P polarization). As in the above example, a plurality of LEDs constituting the light source are provided (however, since FIG. 16 shows only one LED because of the vertical cross section), and these are attached at a predetermined position relative to the LED collimator 18. Each of the LED collimators 18 is formed of, for example, a translucent resin such as acrylic or glass. The LED collimator 18 has a cone-shaped outer peripheral surface obtained by rotating a parabolic cross section, and at the top of the LED collimator 18, a concave portion having a convex portion (i.e., a convex lens surface) formed in the center is formed. The center of the flat portion of the LED collimator 18 has a convex lens surface protruding outward (or may be a concave lens surface recessed inward). The parabolic surface forming the cone-shaped outer peripheral surface of the LED collimator 18 is set within an angle range that allows the light emitted from the LED 18 in the peripheral direction to be totally reflected therein, or a reflective surface is formed thereon.

[0106] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 102. The LED substrate 102 is arranged and fixed to the LED collimator 18 such that the LEDs on the surface are located at the center of the recessed portion.

[0107] According to this configuration, the light emitted from the LEDs by the LED collimator 18, particularly the light emitted from the central part, is collected and made parallel by the two convex lens surfaces that form the outer shape of the LED collimator 18. The light emitted from the other parts toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the cone shape of the LED collimator 18, and is similarly collected and made parallel. In other words, the LED collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, makes it possible to extract almost all of the light generated by the LEDs as parallel light, and improves the utilization efficiency of the generated light. <Light source device example 4>

[0108] Further, another example of the configuration of the optical system of the light source device, etc., will be described with reference to FIG. 25. Two optical sheets 207 that convert the diffusion characteristics in the vertical and horizontal directions (front and rear directions in the figure, not shown) of the drawing are used on the light emission side of the LED collimator 18, and the light from the LED collimator 18 is made to enter between the two optical sheets 207 (diffusion sheets). When this optical sheet 207 is configured with one sheet, the vertical and horizontal diffusion characteristics are controlled by the fine shapes of the front and back surfaces. In addition, multiple diffusion sheets may be used to share the function. Depending on the front and back shapes of the optical sheet 207, the diffusion angle of the light from the LED collimator 18 in the vertical direction of the screen is matched to the width of the vertical surface of the reflection surface of the diffusion sheet, and in the horizontal direction, the number of LEDs and the divergence angle from the LED substrate (optical element) 102 are optimally designed as design parameters so that the surface density of the light flux emitted from the liquid crystal display panel 11 is uniform. In other words, the diffusion characteristics are controlled by the surface shapes of multiple diffusion sheets instead of the light guide. In this embodiment, the polarization conversion is performed in the same manner as in the above-mentioned example 3 of the light source device. In contrast to this, a polarization conversion element 21 may be provided between the LED collimator 18 and the diffusion film 207 so that the light source light is made incident on the diffusion sheet 207 after polarization conversion.

[0109] If the above-mentioned reflective polarizing plate 206 is selected to have the property of reflecting S-polarized light (transmitting P-polarized light), it will reflect the S-polarized light in the natural light emitted from the LED light source, pass through retardation plate 270 shown in Fig. 25, be reflected by reflective surface 271, and pass through retardation plate 270 again to be converted into P-polarized light and enter liquid crystal display panel 11. The optimum value for the thickness of this retardation plate must be selected depending on the angle of incidence of the light beam on the retardation plate, and the optimum value is in the range of λ / 16 to λ / 4. <Lenticular lens>

[0110] In order to control the diffusion distribution of the image light from the liquid crystal display panel 11, it is possible to control the emission characteristics in one direction by optimizing the lens shape by providing a lenticular lens between the light source device 13 and the liquid crystal display panel 11 or on the surface of the liquid crystal display panel 11. Furthermore, by arranging a microlens array in a matrix, it is possible to control the emission characteristics of the image light flux from the display device 1 in the X-axis and Y-axis directions, and as a result, it is possible to obtain an image display device with desired diffusion characteristics.

[0111] The action of the lenticular lens will be described. By optimizing the lens shape, the lenticular lens can efficiently obtain a floating image in space by transmitting or reflecting the light emitted from the above-mentioned display device 1 through the transparent member 100. That is, by combining two lenticular lenses or providing a sheet that controls the diffusion characteristics by arranging a microlens array in a matrix, the brightness (relative brightness) of the image light from the display device 1 can be controlled in the X-axis and Y-axis directions according to the reflection angle (vertical direction is 0 degrees). In this embodiment, such a lenticular lens makes the vertical brightness characteristic steeper as shown in Figure 22(b) compared to the conventional case, and further changes the balance of the directional characteristics in the up and down directions (positive and negative directions of the Y axis) to increase the brightness (relative brightness) of the light due to reflection and diffusion. This makes it possible to obtain image light with a narrow diffusion angle (high straightness) and containing only specific polarization components, like the image light from a surface-emitting laser image source, thereby suppressing ghost images that were generated by the retroreflective material when using an image display device using conventional technology, and making it possible to efficiently control the floating image in space due to retroreflection to reach the observer's eyes.

[0112] Furthermore, by using the light source device described above, it is possible to realize an image display device that emits light of a specific polarization that emits an image light beam that is nearly parallel to a specific direction by achieving a directional characteristic with a significantly narrower angle in both the X-axis and Y-axis directions compared to the diffusion characteristic of the emitted light from a typical liquid crystal display panel shown in Figure 22 (a) and (b) (labeled as "conventional" in the figure).

[0113] Fig. 21 shows an example of the characteristics of the lenticular lens employed in this embodiment. This example particularly shows the characteristics in the X direction (vertical direction), and characteristic O shows a luminance characteristic that is symmetrical up and down, with the peak of the light emission direction at an angle of about 30 degrees upward from the vertical direction (0 degrees). Characteristics A and B in Fig. 21 show examples of characteristics in which the image light above the peak luminance is further concentrated at about 30 degrees to increase the luminance (relative luminance). For this reason, in characteristics A and B, the luminance (relative luminance) of light is rapidly reduced at angles exceeding 30 degrees, compared to characteristic O.

[0114] That is, according to the optical system including the above-mentioned lenticular lens, when the image light beam from the display device 1 is incident on the retroreflective member 2, the emission angle and viewing angle of the image light aligned to a narrow angle by the light source device 13 can be controlled, and the degree of freedom of installation of the retroreflective sheet (retroreflective member 2) can be greatly improved. As a result, the degree of freedom of the relationship of the image formation position of the space floating image that is reflected or transmitted through the transparent member 100 and formed at a desired position can be greatly improved. As a result, it is possible to efficiently reach the eye of an observer outside or inside the room as light with a narrow diffusion angle (high straightness) and only a specific polarized wave component. As a result, even if the intensity (brightness) of the image light from the image display device is reduced, the observer can accurately recognize the image light and obtain information. In other words, by reducing the output of the image display device, it is possible to realize a space floating image display device with low power consumption. <Touch operation assistance function>

[0115] Next, the assist function of the touch operation for the user will be described. First, the touch operation without the assist function will be described. Note that, here, the case where the user selects and touches one of two buttons (objects) will be described as an example, but the following content can also be suitably applied to, for example, ATMs in banks, ticket vending machines in stations, digital signage, and the like.

[0116] FIG. 26 is a diagram for explaining a display example and touch operation of the space floating image display device 1000. The space floating image 3 shown in FIG. 26 includes a first button BUT1 displayed as "YES" and a second button BUT2 displayed as "NO". The user moves the finger 210 toward the space floating image 3 and touches the first button BUT1 or the second button BUT2 to select "YES" or "NO". In the examples of FIG. 26 and FIG. 27 to FIG. 29, the first button BUT1 and the second button BUT2 are displayed in different colors. Here, the area other than the first button BUT1 and the second button BUT2 in the space floating image 3 may be transparent without displaying an image, but in that case, the range of the effect of the virtual shadow described later will be only the area of ​​the displayed buttons (the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2). Therefore, in the following description, as a more suitable example, it is assumed that in the area of ​​the floating image 3 other than the first button BUT1 and the second button BUT2, an image of a different color or brightness than that of the first button BUT1 and the second button BUT2 is displayed in a wider area including the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2.

[0117] In a general touch panel image display device that is not a space-floating image display device, the button selected by the user is composed of an image button displayed on the touch panel surface. Therefore, the user can recognize the sense of distance between the object (e.g., a button) displayed on the touch panel surface and his / her finger by visually checking the touch panel surface. However, in a space-floating image display device, since the space-floating image 3 floats in the air, it may not be easy for the user to recognize the depth of the space-floating image 3. Therefore, in a touch operation on the space-floating image 3, it may not be easy for the user to recognize the sense of distance between the button displayed on the space-floating image 3 and his / her finger. Also, in a general touch panel image display device that is not a space-floating image display device, the user can easily determine whether or not the button has been touched by the feeling when touched. However, in a touch operation on the space-floating image 3, since there is no feeling when an object (e.g., a button) is touched, the user may not be able to determine whether or not the object has been touched. In consideration of the above situation, in this embodiment, a function to assist the user in touch operation is provided.

[0118] In the following description, processing based on the position of the user's finger will be described, and a specific method for detecting the position of the user's finger will be described later. <<Touch operation assistance using virtual shadows (1)>>

[0119] 27 to 29 are diagrams for explaining an example of a method for assisting a touch operation using a virtual shadow. In the examples of FIG. 27 to FIG. 29, the user touches the first button BUT1 to select "YES". The space-floating image display device 1000 of this embodiment assists the user's touch operation by displaying a virtual shadow on the display image of the space-floating image 3. Here, "displaying a virtual shadow on the display image of the space-floating image 3" refers to an image display process in which the brightness of the image signal is reduced for a part of the area of ​​the shape that resembles a finger for the image to be displayed as the space-floating image 3, so that the image appears as if a shadow is projected on the image. Specifically, the image control unit 1160 or the control unit 1110 may perform this process by calculation. In the display process of the virtual shadow, the brightness of the image signal may be completely set to 0 for a part of the area of ​​the shape that resembles a finger. However, it is preferable to display an image with reduced brightness in the part of the area of ​​the shape that resembles a finger, since the shadow is more naturally recognized as a shadow. In this case, in the process of displaying the virtual shadow, not only the luminance of the video signal may be reduced for a portion of the region having a shape resembling a finger, but also the saturation of the video signal may be reduced.

[0120] The floating image 3 exists in the air where there is no physical contact surface, and in a normal environment, the shadow of a finger is always projected. However, according to the display process of the virtual shadow of this embodiment, even in the air where the shadow of a finger is not projected, it is possible to improve the user's perception of the depth of the floating image 3 and the reality of the floating image 3 by making it appear as if a shadow exists in the floating image 3.

[0121] Fig. 27 shows a state at a first time point when the user attempts to touch the first button BUT1 on the display surface 3a of the floating-in-space image 3 with the finger 210, Fig. 28 shows a state at a second time point when the finger 210 is closer to the floating-in-space image 3 than in Fig. 27, and Fig. 29 shows a state at a third time point when the finger 210 touches the first button BUT1 on the display surface 3a of the floating-in-space image 3. In addition, (A) of Fig. 27 to Fig. 29 shows a state when the display surface 3a of the floating-in-space image 3 is viewed from the front (normal direction of the display surface 3a), and (B) of Fig. 27 to Fig. 29 shows a state when the display surface 3a of the floating-in-space image 3 is viewed from the side (direction parallel to the display surface 3a). In addition, in FIG. 27 to FIG. 29, the x direction is the horizontal direction on the display surface 3a of the space floating image 3, the y direction is the direction perpendicular to the x axis in the display surface 3a of the space floating image 3, and the z direction is the normal direction of the display surface 3a of the space floating image 3 (the height direction with respect to the display surface 3a). In addition, in the explanatory diagrams of FIG. 27 to 33, the space floating image 3 is illustrated as having a thickness in the depth direction for ease of explanation, but in reality, if the image display surface of the display device 1 is flat, the space floating image 3 is also flat and has no thickness in the depth direction. In this case, the space floating image 3 and the display surface 3a are on the same plane. In the explanation of this embodiment, the display surface 3a means a surface on which the space floating image 3 can be displayed, and the space floating image 3 means the part where the space floating image is actually displayed.

[0122] 27, 28, and 29, the detection process of the finger 210 is performed using, for example, an image generated by the imaging unit 1180 and a sensing signal of the aerial operation detection sensor 1351. In the detection process of the finger 210, for example, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the floating in space image 3, the height position (z coordinate) of the tip of the finger 210 relative to the display surface 3a, and the like are detected. Here, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the floating in space image 3 is the position coordinate on the display surface 3a of the intersection of the tip of the finger 210 to the display surface 3a of the floating in space image 3. Note that the height position of the tip of the finger 210 relative to the display surface 3a is also depth information that indicates the depth of the finger 210 relative to the display surface 3a. Note that the arrangement of the imaging unit 1180 and the aerial operation detection sensor 1351 that detect the finger 210 and the like will be described in detail later.

[0123] At the first time point shown in Fig. 27, the finger 210 is in a position farthest from the display surface 3a of the floating-in-space image 3 compared to the second time point shown in Fig. 28 and the third time point shown in Fig. 28. The distance (height position) between the tip of the finger 210 and the display surface 3a of the floating-in-space image 3 at this time is set to dz1. In other words, the distance dz1 indicates the height of the finger 210 relative to the display surface 3a of the floating-in-space image 3 in the z direction.

[0124] 27 and distance dz2 shown in Fig. 28 described later are positive on the user side with respect to the display surface 3a of the floating-in-space image 3, and negative on the opposite side of the display surface 3a from the user. That is, if the finger 210 is on the user side with respect to the display surface 3a, the distances dz1 and dz2 are positive values, and if the finger 210 is on the opposite side of the user with respect to the display surface 3a, the distances dz1 and dz2 are negative values.

[0125] In this embodiment, it is assumed that virtual light source 1500 is located on the user side with respect to display surface 3a of space-floating image 3. Here, the setting of the installation direction of virtual light source 1500 may be actually stored as information in non-volatile memory 1108 or memory 1109 of space-floating image display device 1000. Moreover, the setting of the installation direction of virtual light source 1500 may be a parameter that exists only in design. Even if the setting of the installation direction of virtual light source 1500 is a parameter that exists only in design, the installation direction of virtual light source 1500 in design is uniquely determined from the relationship between the position of the user's finger and the display position of the virtual shadow, which will be described later. Here, in the example of FIG. 27 to FIG. 29, virtual light source 1500 is located on the user side with respect to display surface 3a, and is provided on the right side of display surface 3a as seen from the user. Then, virtual shadow 1510 simulating the shadow of finger 210 formed by light irradiated from virtual light source 1500 is displayed on space-floating image 3. 27 to 29, a virtual shadow 1510 is displayed on the left side of the finger 210. This virtual shadow 1510 assists the user in performing a touch operation.

[0126] In the state of FIG. 27(B), the tip of the finger 210 is farthest in the normal direction from the display surface 3a of the floating-in-space image 3, compared to the states of FIG. 28(B) and FIG. 29(B). Therefore, in FIG. 27(A), the tip of the virtual shadow 1510 is formed at a position farthest in the horizontal direction from the first button BUT1 to be touched, compared to the states of FIG. 28(A) and FIG. 29(A). Therefore, in FIG. 27(A), the horizontal distance between the tip of the finger 210 and the tip of the virtual shadow 1510 when the display surface 3a of the floating-in-space image 3 is viewed from the front is the largest, compared to the states of FIG. 28(A) and FIG. 29(A). In FIG. 27(A), the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating-in-space image 3 is dx1.

[0127] In FIG. 28(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 28(B), distance dz2 in the normal direction between tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz1. At this time, in FIG. 28(A), virtual shadow 1510 is displayed at a position where distance between tip of finger 210 and tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx2, which is smaller than dx1. That is, in the example of FIG. 28, virtual light source 1500 is provided on the user side with respect to display surface 3a and on the right side of display surface 3a as seen from the user, so that the horizontal distance between tip of finger 210 and tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front changes in conjunction with the distance between tip of finger 210 and display surface 3a of space-floating image 3 in the normal direction.

[0128] Then, when the tip of the finger 210 and the tip of the virtual shadow 1510 come into contact, the distance in the normal direction between the tip of the finger 210 and the display surface 3a of the floating-in-space image 3 becomes 0, as shown in FIG. 29. At this time, the virtual shadow 1510 is displayed so that the distance between the finger 210 and the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating-in-space image 3 becomes zero. This allows the user to recognize that the finger 210 has touched the display surface 3a of the floating-in-space image 3. At this time, if the tip of the finger 210 is touching the area of ​​the first button BUT1, the user can recognize that the first button BUT1 has been touched. 29, since virtual light source 1500 is provided on the user side of display surface 3a and on the right side of display surface 3a as seen from the user, the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of floating in space image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of floating in space image 3. In other words, the display position of the tip of virtual shadow 1510 is a position specified by the positional relationship between the position of virtual light source 1500 and the position of the tip of user's finger 210, and changes in conjunction with the change in the position of the tip of user's finger 210.

[0129] According to the configuration and processing of "assistance of touch operation using virtual shadow (1)" described above, when performing a touch operation, the user can more appropriately recognize the normal direction distance (depth) between the finger 210 and the display surface 3a of the floating-in-space image 3 from the horizontal positional relationship between the finger 210 and the virtual shadow 1510 on the display surface 3a of the floating-in-space image 3. Also, when the finger 210 touches an object (e.g., a button) that is the floating-in-space image 3, the user can recognize that the object has been touched. This makes it possible to provide a more appropriate floating-in-space image display device. <<Touch operation assistance using virtual shadows (2)>>

[0130] Next, as another example of the method of assisting a touch operation using a virtual shadow, a case where virtual light source 1500 is provided on the left side of display surface 3a as seen from the user will be described. FIGS. 30 to 32 are diagrams for explaining another example of the method of assisting a touch operation using a virtual shadow. FIG. 30 corresponds to FIG. 27, and shows a state at a first time point when a user attempts to perform a touch operation on first button BUT1 on display surface 3a of floating-in-space image 3 with finger 210. FIG. 31 corresponds to FIG. 28, and shows a state at a second time point when finger 210 is closer to floating-in-space image 3 than in FIG. 30. FIG. 32 corresponds to FIG. 29, and shows a state when finger 210 touches floating-in-space image 3. Note that, for convenience of explanation, (B) of FIG. 30 to FIG. 32 shows a view from the opposite direction to (B) of FIG. 27 to FIG. 29.

[0131] 30 to 32, virtual light source 1500 is provided on the user side of display surface 3a, on the left side of display surface 3a as seen from the user. Then, a virtual shadow 1510 that imitates the shadow of finger 210 formed by light irradiated from virtual light source 1500 is displayed in space floating image 3. In FIGS. 30 to 32, virtual shadow 1510 is displayed to the right of finger 210. This virtual shadow 1510 assists the user in performing a touch operation.

[0132] In the state of Fig. 30(B), compared to the states of Fig. 31(B) and Fig. 32(B), the tip of finger 210 is the furthest away in the normal direction from display surface 3a of space floating image 3. In Fig. 30(B), the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 at this time is dz10. Also, in Fig. 30(A), the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space floating image 3 at this time is dx10.

[0133] In FIG. 31(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 31(B), distance dz20 in the normal direction between tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz10. At this time, in FIG. 31(A), virtual shadow 1510 is displayed at a position where distance between tip of finger 210 and tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx20, which is smaller than dx10. That is, in the example of FIG. 31, virtual light source 1500 is provided on the user side with respect to display surface 3a and on the left side of display surface 3a as seen from the user, so that the horizontal distance between tip of finger 210 and tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front changes in conjunction with the distance between tip of finger 210 and display surface 3a of space-floating image 3 in the normal direction.

[0134] Then, when the tip of the finger 210 and the tip of the virtual shadow 1510 come into contact, the distance in the normal direction between the tip of the finger 210 and the display surface 3a of the floating-in-space image 3 becomes 0, as shown in FIG. 32. At this time, the virtual shadow 1510 is displayed so that the distance between the finger 210 and the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating-in-space image 3 becomes zero. This allows the user to recognize that the finger 210 has touched the display surface 3a of the floating-in-space image 3. At this time, if the tip of the finger 210 is touching the area of ​​the first button BUT1, the user can recognize that the first button BUT1 has been touched. That is, in the example of FIG. 32, virtual light source 1500 is provided on the user's side of display surface 3a, to the left of display surface 3a as seen from the user, so that the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of floating-in-space image 3 is viewed from the front changes in conjunction with the normal distance between the tip of finger 210 and display surface 3a of floating-in-space image 3.

[0135] The above-described configuration and processing of "assistance of touch operation using virtual shadow (2)" also provides the same effects as those of the configurations of FIGS.

[0136] Here, when implementing the above-mentioned process of "assisting touch operation using a virtual shadow (1)" and / or the process of "assisting touch operation using a virtual shadow (2)" in the space floating image display device 1000, there are several possible implementation examples.

[0137] The first implementation example is a method of implementing only "assistance of touch operation using virtual shadow (1)" in the floating-in-space image display device 1000. In this case, since the virtual light source 1500 is provided on the user side of the display surface 3a and on the right side of the display surface 3a as seen from the user, the virtual shadow 1510 is displayed on the left side of the tip of the user's finger 210 as seen from the user. Therefore, if the user's finger 210 is a finger of the right hand, the visibility of the display of the virtual shadow 1510 is not blocked by the user's right hand or right arm, which is preferable. Therefore, considering the statistical tendency that there are more right-handed users, even if only "assistance of touch operation using virtual shadow (1)" is implemented in the floating-in-space image display device 1000, the probability that the display of the virtual shadow 1510 can be well viewed is sufficiently high and preferable.

[0138] As a first implementation example, both the process of "assisting touch operation using a virtual shadow (1)" and the process of "assisting touch operation using a virtual shadow (2)" may be implemented, and the process to be performed may be switched depending on whether the user performs the touch operation with the right hand or the left hand. In this case, it is possible to further increase the probability that the display of the virtual shadow 1510 can be clearly seen, improving the convenience for the user.

[0139] Specifically, when the user performs a touch operation with the right hand, the configurations of FIGS. 27 to 29 are used to display the virtual shadow 1510 on the left side of the finger 210. In this case, the visibility of the display of the virtual shadow 1510 is not blocked by the user's right hand or right arm, which is preferable. On the other hand, when the user performs a touch operation with the left hand, the configurations of FIGS. 30 to 32 are used to display the virtual shadow 1510 on the right side of the finger 210. In this case, the visibility of the display of the virtual shadow 1510 is not blocked by the user's left hand or left arm, which is preferable. As a result, whether the user performs a touch operation with the right hand or the left hand, the virtual shadow 1510 is displayed at a position that is easy for the user to view, improving user convenience.

[0140] Here, the determination of whether the touch operation is performed with the right hand or the left hand may be performed based on, for example, a captured image generated by the imaging unit 1180. For example, the control unit 1110 performs image processing on the captured image and detects the user's face, arms, hands, and fingers from the captured image. Then, the imaging unit 1180 may estimate the user's posture or movement from the arrangement of the detected (face, arms, hands, fingers) and determine whether the user is performing the touch operation with the right hand or the left hand. In this determination, if the vicinity of the center in the left-right direction of the user's body can be determined from other parts, it is not necessarily necessary to capture an image of the face. Also, the above determination may be performed only from the arrangement of the arms. The above determination may be performed only from the arrangement of the hands. The above determination may be performed from a combination of the arrangement of the arms and the arrangement of the hands. Also, when making these determinations, the determination may be performed in combination with the arrangement of the face.

[0141] 27 to 29 and 30 to 32 show virtual shadow 1510 extending at an angle corresponding to the extending direction of actual finger 210. The extending direction of actual finger 210 may be calculated by capturing an image of the finger using any of the imaging units already described. Here, virtual shadow 1510 may be displayed with the extending direction fixed at a predetermined angle without reflecting the angle corresponding to the extending direction of finger 210. This reduces the load on image control unit 1160 or control unit 1110 that controls the display of virtual shadow 1510.

[0142] For example, if the finger 210 is a finger of the right hand, it is natural for the user to extend his / her arm from the right side in front of the display surface 3a of the floating in space image 3, and try to touch the display surface 3a of the floating in space image 3 with the finger 210 pointing to the upper left toward the display surface 3a of the floating in space image 3. Therefore, if the finger 210 is a finger of the right hand, if the finger shadow indicated by the virtual shadow 1510 is configured to be displayed in a predetermined direction indicating the upper right direction toward the display surface 3a of the floating in space image 3, a natural display will be obtained without reflecting the angle corresponding to the finger 210.

[0143] Also, for example, if the finger 210 is a finger of the left hand, it is natural for the user to extend his / her arm from the left side in front of the display surface 3a of the floating in space image 3, and try to touch the display surface 3a of the floating in space image 3 with the finger 210 pointing to the upper right toward the display surface 3a of the floating in space image 3. Therefore, if the finger 210 is a finger of the left hand, if the finger shadow indicated by the virtual shadow 1510 is configured to be displayed in a predetermined direction indicating the upper left direction toward the display surface 3a of the floating in space image 3, a natural display will be obtained without reflecting the angle corresponding to the finger 210.

[0144] In addition, when the user's finger 210 is on the opposite side of the display surface 3a of the space floating image 3 from the user, a display may be made so that the user can recognize that the finger 210 is behind the space floating image 3 and cannot be touched. For example, a message may be displayed on the space floating image 3 to inform the user that the finger 210 is behind the space floating image 3 and cannot be touched. Alternatively, for example, the virtual shadow 1510 may be displayed in a color different from the normal color, such as red. This makes it possible to more appropriately prompt the user to return the finger 210 to the appropriate position. <<An example of virtual light source settings>>

[0145] Here, a method for setting virtual light source 1500 will be described. Fig. 33 is a diagram for explaining a method for setting a virtual light source. Fig. 33 shows a situation in which a user performs a touch operation with the left hand, but the contents described below are also preferably applicable to a case in which a user performs a touch operation with the right hand.

[0146] 33 shows normal line L1 of display surface 3a extending from point C at the center of display surface 3a of space floating image 3 toward the user, line L2 connecting virtual light source 1500 and point C where normal line L1 intersects with display surface 3a, and virtual light source installation angle α defined by the angle between normal line L1 and line L2. For ease of explanation, FIG. 33 shows the moment when the tip of user's finger 210 is on line L2.

[0147] Here, in order to simplify the explanation, in FIGS. 27 to 33, virtual light source 1500 is illustrated as being placed at a position not far from display surface 3a of space floating image 3 and user's finger 210. Although virtual light source 1500 may be set at such a position, the most preferable setting example is as follows. That is, it is desirable to set the distance between virtual light source 1500 and point C at the center of display surface 3a of space floating image 3 to infinity. The reason is as follows. If an object plane having a contact surface is in the same coordinate system as display surface 3a of space floating image 3 in FIGS. 27 to 32, and the light source is the sun rather than a virtual light source, the distance of the sun can be approximated as almost infinity, so that the horizontal direction (x direction) position of the tip of the shadow of the user's finger on the real object plane changes linearly with respect to the change in the distance (z direction) between the tip of the user's finger and the object plane. 27 to 33 of this embodiment, if the distance between the virtual light source 1500 and the central point C of the display surface 3a of the floating-in-space image 3 is set to infinity and the horizontal (x-direction) position of the tip of the virtual shadow 1510 in the floating-in-space image 3 changes linearly with respect to the change in the distance (z-direction) between the tip of the user's finger 210 and the display surface 3a of the floating-in-space image 3, a virtual shadow that can be recognized more naturally by the user can be expressed. If the virtual light source 1500 is set to be disposed at a position not far from the display surface 3a of the floating-in-space image 3 and the user's finger 210, the horizontal (x-direction) position of the tip of the virtual shadow 1510 in the floating-in-space image 3 changes nonlinearly with respect to the change in the distance (z-direction) between the tip of the user's finger 210 and the display surface 3a of the floating-in-space image 3, and the calculation for calculating the horizontal (x-direction) position of the tip of the virtual shadow 1510 becomes somewhat complicated. On the other hand, if the distance between virtual light source 1500 and center point C of display surface 3a of floating-in-space image 3 is set to infinity, the horizontal (x-direction) position of the tip of virtual shadow 1510 in floating-in-space image 3 changes linearly with the change in the distance (z-direction) between the tip of user's finger 210 and display surface 3a of floating-in-space image 3, which has the effect of simplifying the calculation for calculating the horizontal (x-direction) position of the tip of virtual shadow 1510.

[0148] When the virtual light source installation angle α is small, the angle between the line connecting virtual light source 1500 and finger 210 and normal line L1 cannot be increased as viewed from the user, so the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction (x direction) of display surface 3a of floating-in-space image 3 becomes short. This makes it difficult for the user to visually recognize the change in the position of virtual shadow 1510 when the tip of finger 210 performs a touch operation, and there is a risk that the effect of the user's depth perception in the touch operation will decrease. To avoid this, it is desirable to install virtual light source 1500 so that the angle between line L2 connecting virtual light source 1500 and point C and normal line L1 is, for example, 20° or more.

[0149] On the other hand, when the angle between the line connecting virtual light source 1500 and finger 210 and normal line L1 approaches 90°, the distance between the tip of finger 210 and the tip of virtual shadow 1510 becomes very long. This increases the probability that the display position of virtual shadow 1510 will be outside the range of floating-in-space image 3, and increases the probability that virtual shadow 1510 cannot be displayed in floating-in-space image 3. For this reason, it is desirable that installation angle α of virtual light source 1500 be 70° or less so that the angle between line L2 connecting virtual light source 1500 and point C and normal line L1 does not approach too close to 90°, for example.

[0150] In other words, it is desirable to place virtual light source 1500 in a position that is not too close to a plane that includes a normal line passing through finger 210 and is not too close to a plane that includes display surface 3a of space floating image 3.

[0151] The space-floating image display device 1000 of this embodiment can display a virtual shadow as described above. This is a video processing that produces a physically more natural effect than the case where a predetermined mark is superimposed on the image to assist the user's touch operation. Therefore, the touch operation assistance technology by displaying the above-mentioned virtual shadow of the space-floating image display device 1000 of this embodiment can provide a situation in which the user can recognize the depth of the touch operation more naturally. <<How to detect finger position>>

[0152] Next, a method for detecting the position of finger 210 will be described. A configuration for detecting the position of finger 210 of user 230 will be specifically described below. <<<How to detect finger position (1)>>>

[0153] Fig. 34 is a configuration diagram showing an example of a method for detecting the position of a finger. In the example shown in Fig. 34, the position of a finger 210 is detected using one imaging unit 1180 and one air operation sensor 1351. Note that each imaging unit in the embodiment of the present invention has an imaging sensor.

[0154] The first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the space floating image 3. The first imaging unit 1180a may be installed in the housing 1190 as shown in FIG. 34, or may be installed at a location away from the housing 1190.

[0155] The imaging area of ​​the first imaging unit 1180a is set to include, for example, the display area of ​​the space floating image 3, the fingers, hands, arms, face, etc. of the user 230. The first imaging unit 1180a images the user 230 performing a touch operation on the space floating image 3, and generates a first captured image. Note that even if the display area of ​​the space floating image 3 is captured by the first imaging unit 1180a, the image is captured from the opposite side of the traveling direction of the directional light flux of the space floating image 3, so the space floating image 3 itself cannot be visually recognized as an image. Here, in the example of the finger position detection method (1), the first imaging unit 1180a is not just an imaging unit, but has a built-in depth sensor in addition to an imaging sensor. The configuration and processing of the depth sensor may use existing technology. The depth sensor of the first imaging unit 1180a detects the depth of each part (for example, the user's fingers, hands, arms, face, etc.) in the captured image of the first imaging unit 1180a, and generates depth information.

[0156] The aerial operation sensor 1351 is installed at a position where it can sense the display surface 3a of the floating-in-space image 3 as the sensing target surface. In Fig. 34, the aerial operation sensor 1351 is installed below the display surface 3a of the floating-in-space image 3, but it may be installed to the side or above the display surface 3a. The aerial operation sensor 1351 may be installed in the housing 1190 as shown in Fig. 34, or may be installed at a location away from the housing 1190.

[0157] 34 is a sensor that detects the position where the finger 210 contacts or overlaps with the display surface 3a of the floating-in-space image 3. That is, when the tip of the finger 210 approaches the display surface 3a of the floating-in-space image 3 from the user's side of the display surface 3a of the floating-in-space image 3, the floating-in-space operation detection sensor 1351 can detect the contact of the finger 210 with the display surface 3a of the floating-in-space image 3.

[0158] For example, the control unit 1110 shown in Fig. 3C reads out a program for performing image processing and a program for displaying a virtual shadow 1510 from the non-volatile memory 1108. The control unit 1110 performs a first image processing on a first captured image generated by the imaging sensor of the first imaging unit 1180a, detects the finger 210, and calculates the position (x coordinate, y coordinate) of the finger 210. The control unit 1110 calculates the position (z coordinate) of the tip of the finger 210 with respect to the floating in space image 3 based on the first captured image generated by the imaging sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a.

[0159] 34, the image sensor and depth sensor of the first imaging unit 1180a, the aerial operation sensor 1351, the aerial operation detection unit 1350, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects a touch on an object in the floating in space image 3. This calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210. In addition, the touch detection result is calculated by combining the detection result of the aerial operation detection unit 1350 or the detection result of the aerial operation detection unit 1350 and information generated by the first imaging unit 1180a.

[0160] Then, control unit 1110 calculates the position (display position) at which virtual shadow 1510 is to be displayed, based on the position (x coordinate, y coordinate, z coordinate) of finger 210 and the position of virtual light source 1500, and generates video data of virtual shadow 1510 based on the calculated display position.

[0161] The control unit 1110 may calculate the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated. The control unit 1110 may not calculate the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated, but may store display position map data in which the display positions of the virtual shadow 1510 corresponding to each of the multiple positions of the finger 210 are calculated in advance in the non-volatile memory 1108, and generate video data of the virtual shadow 1150 based on the display position map data stored in the non-volatile memory 1108 after calculating the position of the finger 210. The control unit 1110 may also calculate the tip of the finger 210 and the extension direction of the finger 210 in the first image processing, calculate the display position of the tip of the finger 210 and the extension direction of the virtual shadow 1510 corresponding to the extension direction, and generate video data of the virtual shadow 1510 adjusted to a display angle corresponding to the actual orientation of the finger 210 based on these.

[0162] The control unit 1110 outputs the generated image data of the virtual shadow 1510 to the image control unit 1160. The image control unit 1160 generates image data (superimposed image data) in which the image data of the virtual shadow 1510 is superimposed on other image data such as an object, and outputs the superimposed image data including the image data of the virtual shadow 1510 to the image display unit 1102.

[0163] The image display unit 1102 displays an image based on the superimposed image data including image data of the virtual shadow 1510, whereby a space floating image 3 in which the virtual shadow 1510 and an object or the like are superimposed is displayed.

[0164] The detection of a touch on an object is performed, for example, as follows. The aerial operation detection unit 1350 and the aerial operation detection sensor 1351 are configured as described in FIG. 3, and when the finger 210 touches or overlaps a plane including the display surface 3a of the space floating image 3, the aerial operation detection unit 1350 detects the position and outputs touch position information indicating the position where the finger 210 touches or overlaps the display surface 3a to the control unit 1110. Then, when the touch position information is input, the control unit 1110 judges whether or not the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing is included in the display range of each object displayed on the display surface 3a of the space floating image 3. Then, when the position of the finger 210 is included in the display range of any one of the objects, the control unit 1110 judges that the object has been touched.

[0165] According to the detection method described above, it is possible to detect the position of the finger 210 and the touch operation with a simple configuration that combines one imaging unit 1180 (first imaging unit 1180a) having an image sensor and a depth sensor with one aerial operation detection sensor 1351.

[0166] As a modified example of the finger position detection method (1), the control unit 1110 may detect a touch operation by the finger 210 solely based on the first captured image generated by the imaging sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, without using the detection results by the aerial operation detection unit 1350 and the aerial operation detection sensor 1351. For example, during normal operation, the mode is configured to detect a touch operation by finger 210 by combining the captured image of the imaging sensor of first imaging unit 1180a, the detection result of the depth sensor, and the detection result of the aerial operation detection sensor 1351, and if there is any malfunction in the operation of the aerial operation detection sensor 1351 or the aerial operation detection unit 1350, the control unit 1110 may switch to a mode in which it detects a touch operation by finger 210 only based on the first captured image generated by the imaging sensor of first imaging unit 1180a and the depth information generated by the depth sensor of first imaging unit 1180a, without using the detection results of the aerial operation detection unit 1350 and the aerial operation detection sensor 1351. <<How to detect finger position (2)>>

[0167] Fig. 35 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in Fig. 35, the position of the finger 210 is detected using two imaging units. The second imaging unit 1180b (1180) and the third imaging unit 1180c (1180) are both provided on the opposite side of the user 230 with respect to the floating-in-space image 3.

[0168] The second imaging unit 1180b is installed, for example, on the right side as seen from the user 230. The imaging area of ​​the second imaging unit 1180b is set to include, for example, the floating-in-space image 3, and the fingers, hands, arms, face, etc. of the user 230. The second imaging unit 1180b images the user 230 performing a touch operation on the floating-in-space image 3 from the right side of the user 230, and generates a second captured image.

[0169] The third imaging unit 1180c is installed, for example, on the left side as seen from the user 230. The imaging area of ​​the third imaging unit 1180c is set to include, for example, the floating in space image 3, the fingers, hands, arms, face, etc. of the user 230. The third imaging unit 1180c images the user 230 performing a touch operation on the floating in space image 3 from the left side of the user 230, and generates a third captured image. Thus, in the example of FIG. 35, the second imaging unit 1180b and the third imaging unit 1180c configure a so-called stereo camera.

[0170] 35, the second imaging unit 1180b and the third imaging unit 1180c may be installed in the housing 1190, or may be installed at a location away from the housing 1190. Moreover, one imaging unit may be installed in the housing 1190, and the other imaging unit may be installed at a location away from the housing 1190.

[0171] The control unit 1110 performs second image processing on the second captured image and third image processing on the third captured image, respectively. Then, the control unit 1110 calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210 based on the result of the second image processing (second image processing result) and the result of the third image processing (third image processing result).

[0172] 35, the second imaging section 1180b, the third imaging section 1180c, and the control section 1110 constitute a touch detection section that detects the position of the user's finger and detects a touch on an object in the space floating image 3. Then, the position (x coordinate, y coordinate, z coordinate) of the finger 210 is calculated as a position detection result or a touch detection result.

[0173] 35, a virtual shadow 1510 is generated based on the position of finger 210 calculated based on the second image processing result and the third image processing result. Also, based on the position of finger 210 calculated based on the second image processing result and the third image processing result, a determination is made as to whether or not an object has been touched.

[0174] According to this configuration, it is not necessary to employ an imaging section having a depth sensor. Also, according to this configuration, by using the second imaging section 1180b and the third imaging section 1180c as a stereo camera, it is possible to improve the detection accuracy of the position of the finger 210. In particular, compared to the example of FIG. 34, it is possible to improve the detection accuracy of the x coordinate and the y coordinate. Therefore, it is possible to more accurately determine whether or not an object has been touched.

[0175] As a modified example of the finger position detection method (2), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the second captured image by the second imaging unit 1180b and the third captured image by the third imaging unit 1180c as described above, and the display of the virtual shadow 1510 is controlled based on this, and the presence or absence of a touch on the object of the floating in space image 3 may be detected by the mid-air operation detection unit 1350 or the control unit 1110 based on the detection result by the mid-air operation detection sensor 1351. According to this modified example, since the mid-air operation sensor 1351 that senses the display surface 3a of the floating in space image 3 as the sensing target surface is used, the touch of the user's finger 210 on the display surface 3a of the floating in space image 3 can be detected with higher accuracy than the detection accuracy in the depth direction by the stereo camera of the second imaging unit 1180b and the third imaging unit 1180c. <<<Finger position detection method (3)>>>

[0176] FIG. 36 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in FIG. 36, the position of the finger 210 is detected using two imaging units. The example in FIG. 36 is different from the example in FIG. 35 in that the fourth imaging unit 1180d (1180), which is one of the imaging units, is arranged at a position where it images the display surface 3a of the space floating image 3 from the side. Also, as in the example in FIG. 34, the first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the space floating image 3. In the example in FIG. 36, the first imaging unit 1180a (1180) only needs to be able to capture an image and does not need to be equipped with a depth sensor.

[0177] Therefore, the fourth imaging unit 1180d is installed around the display surface 3a of the space floating image 3. In Fig. 36, the fourth imaging unit 1180d is installed below the side of the display surface 3a of the space floating image 3, but it may be installed to the side or above the display surface 3a. The fourth imaging unit 1180d may be installed in the housing 1190 as shown in Fig. 36, or may be installed at a location away from the housing 1190.

[0178] The imaging area of ​​the fourth imaging unit 1180d is set to include, for example, the floating-in-space image 3 and the fingers, hands, arms, face, etc. of the user 230. The fourth imaging unit 1180d images the user 230 performing a touch operation on the floating-in-space image 3 from the periphery of the display surface 3a of the floating-in-space image 3, and generates a fourth captured image.

[0179] The control unit 1110 performs a fourth image processing on the fourth captured image, and calculates the distance (z coordinate) between the display surface 3a of the space floating image 3 and the tip of the finger 210. Then, the control unit 1110 performs processing related to the virtual shadow 1510 and determines whether or not an object has been touched, based on the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing on the first captured image by the first imaging unit 1180a described above and the position (z coordinate) of the finger 210 calculated by the fourth image processing.

[0180] 36, a touch detection unit that detects the position of a user's finger and detects a touch on an object is configured by the first imaging unit 1180a, the fourth imaging unit 1180d, and the control unit 1110. Then, the position (x coordinate, y coordinate, z coordinate) of the finger 210 is calculated as a position detection result or a touch detection result.

[0181] According to this configuration, it is possible to improve the detection accuracy of the distance between the display surface 3a of the floating in space image 3 and the tip of the finger 210, i.e., the depth of the finger 210 relative to the display surface 3a of the floating in space image 3, compared to the example of the stereo camera configuration in FIG.

[0182] As a modified example of the finger position detection method (3), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the first captured image by the first imaging unit 1180a and the fourth captured image by the fourth imaging unit 1180d as described above, and the display of the virtual shadow 1510 is controlled based on this, and the presence or absence of a touch on the object of the floating in space image 3 may be detected by the mid-air operation detection unit 1350 or the control unit 1110 based on the detection result by the mid-air operation detection sensor 1351. According to this modified example, since the mid-air operation sensor 1351 that senses the display surface 3a of the floating in space image 3 as the sensing target surface is used, the touch of the user's finger 210 on the display surface 3a of the floating in space image 3 can be detected with higher accuracy than the detection accuracy based on the fourth captured image by the fourth imaging unit 1180d. <<How to display input content and assist touch operation>>

[0183] An example of assisting the user's touch operation by another method will be described. For example, it is possible to assist the touch operation by displaying the input contents. Fig. 37 is a diagram for explaining a method of assisting the touch operation by displaying the input contents. Fig. 37 shows a case where numbers are input by touch operation.

[0184] The spatial floating image 3 in Figure 37 includes a key input UI (user interface) display area 1600 containing multiple objects including, for example, multiple objects for inputting numbers, etc., an object 1601 for erasing the input content, and an object 1603 for deciding the input content, and an input content display area 1610 for displaying the input content.

[0185] In the input content display area 1610, the contents (e.g., numbers) input by touch operation are displayed in sequence on the floating-in-space image 3 from the left end to the right. The user can check the contents input by touch operation while looking at the input content display area 1610. Then, when the user has input all the desired numbers, he / she touches the object 1603. This registers the input contents displayed in the input content display area 1610. The touch operation on the floating-in-space image 3 is different from physical contact on the surface of the display device, and the user cannot feel the touch. Therefore, by separately displaying the input contents in the input content display area 1610, the user can proceed with the operation while checking whether or not his / her touch operation has been performed effectively, which is preferable.

[0186] On the other hand, if the user inputs content different from what he or she intended, for example by touching the wrong object, the user can erase the last input content (here, "9") by touching object 1601. Then, the user continues to perform touch operations on objects for inputting numbers and the like. When the user has input all the desired numbers, he or she touches object 1603.

[0187] In this way, by displaying the input contents in the input contents display area 1610, the user can confirm the input contents, and convenience can be improved. Also, if the user touches the wrong object, the input contents can be corrected, and convenience can be improved. <<How to highlight input content to assist touch operation>>

[0188] Next, it is also possible to assist the touch operation by highlighting the input contents. Fig. 38 is a diagram for explaining a method of assisting the touch operation by highlighting the input contents.

[0189] An example in which a number entered by a touch operation is highlighted is shown in Fig. 38. In accordance with Fig. 38, when an object corresponding to the number "6" is touched, the touched object is erased, and the input number "6" is displayed in the area where this object was displayed.

[0190] In this way, the number corresponding to the touched object is displayed instead of the object, so that the user can recognize that the object has been touched, and convenience can be improved. The number corresponding to the touched object may be called a replacement object that replaces the touched object.

[0191] As another method for highlighting the input contents, for example, the object touched by the user may be brightly lit or the object touched by the user may be blinking. Although not shown here, by recognizing the distance between the finger 210 and the display surface 3a described in the embodiment of Figs. 27-28, the object to be touched may become brighter than the surrounding objects as the finger approaches the display surface, and finally, when the finger is moved to the display surface, the degree of highlighting may reach a maximum, or the object may be lit even brighter or blink. Even with such a configuration, it is possible to make the user recognize that the object has been touched, and convenience can be improved. <<Method of assisting touch operation with vibration (1)>>

[0192] Next, a method of assisting a touch operation by vibration will be described. Fig. 39 is a diagram for explaining an example of a method of assisting a touch operation by vibration. Fig. 39 shows a case where a touch operation is performed using a touch pen (touch input device) 1700 instead of a finger 210. The touch pen 1700 is equipped with a communication unit that transmits and receives various information such as signals and data between a device such as a space floating image display device, and a vibration mechanism that vibrates based on an input signal.

[0193] Assume that the user operates the touch pen 1700 and touches an object displayed in the key input UI display area 1600 of the floating-in-space image 3 with the touch pen 1700. At this time, for example, the control unit 1100 transmits a touch detection signal indicating that a touch on the object has been detected from the communication unit 1132. When the touch pen 1700 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the touch pen 1700 to vibrate. The vibration of the touch pen 1700 is then transmitted to the user, and the user recognizes that the object has been touched. In this way, the vibration of the touch pen 1700 assists the touch operation.

[0194] According to this configuration, it is possible to make the user aware that he or she has touched the object by means of vibration.

[0195] Here, the case where the touch detection signal transmitted from the space floating image device is received by the touch pen 1700 has been described, but other configurations are also possible. For example, when a touch on an object is detected, the space floating image display device notifies the host device that a touch on the object has been detected. Then, the host device transmits a touch detection signal to the touch pen 1700.

[0196] Alternatively, the space floating image display device and the higher-level device may transmit the touch detection signal via a network. In this manner, the touch pen 1700 may indirectly receive the touch detection signal from the space floating image display device. <<Method of assisting touch operation with vibration (2)>>

[0197] Next, another method of assisting a touch operation by vibration will be described. Here, the user is made aware that he or she has touched an object by vibrating the terminal owned by the user. FIG. 40 is a diagram for explaining another example of the method of assisting a touch operation by vibration. In the example of FIG. 40, a user 230 wearing a wristwatch-type wearable terminal 1800 performs a touch operation.

[0198] The wearable terminal 1800 is equipped with a communication unit that transmits and receives various information such as signals and data to and from a device such as a space floating image display device, and a vibration mechanism that vibrates based on an input signal.

[0199] Assume that the user performs a touch operation with the finger 210 and touches an object displayed in the key input UI display area 1600 of the floating-in-space image 3. At this time, for example, the control unit 1100 transmits a touch detection signal indicating that a touch on the object has been detected from the communication unit 1132. When the wearable terminal 1800 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the wearable terminal 1800 to vibrate. Then, the vibration of the wearable terminal 1800 is transmitted to the user, and the user recognizes that the object has been touched. In this way, the vibration of the wearable terminal 1800 assists the touch operation. Here, a wristwatch-type wearable terminal has been described as an example, but a smartphone worn by the user may also be used.

[0200] The wearable terminal 1800 may receive a touch detection signal from a higher-level device, similar to the above-described touch pen 1700. The wearable terminal 1800 may receive a touch detection signal via a network. In addition to the wearable terminal 1800, it is also possible to assist the touch operation using an information processing terminal such as a smartphone owned by the user.

[0201] According to this configuration, it is possible to make a user aware that he or she has touched an object via various terminals such as the wearable terminal 1800 owned by the user. <<Method of assisting touch operation with vibration (3)>>

[0202] Next, another method of assisting a touch operation by vibration will be described. FIG. 41 is a diagram for explaining another example of a method of assisting a touch operation by vibration. In the example of FIG. 41, user 230 performs a touch operation while standing on diaphragm 1900. Diaphragm 1900 is placed at a predetermined position where user 230 performs a touch operation. In an actual usage form, diaphragm 1900 is placed, for example, under a mat (not shown), and user 230 stands on diaphragm 1900 via the mat.

[0203] As shown in FIG. 41, the diaphragm 1900 is connected to, for example, the communication unit 1132 of the space floating image display device 1000 via a cable 1910. When a touch on an object is detected, for example, the control unit 1110 supplies an AC voltage to the diaphragm 1900 for a predetermined time via the communication unit 1132. The diaphragm 1900 vibrates while the AC voltage is being supplied. That is, the AC voltage is a control signal for vibrating the diaphragm 1900, which is output from the communication unit 1132. The vibration generated by the diaphragm 1900 is transmitted from the feet to the user 230, and the user 230 can recognize that the object has been touched. In this way, the vibration of the diaphragm 1900 assists the touch operation.

[0204] The frequency of the AC voltage is set to a value within a range in which the user 230 can feel the vibration. The frequency of vibration that a person can feel is within a range of approximately 0.1 Hz to 500 Hz. For this reason, it is desirable to set the frequency of the AC voltage within this range.

[0205] Also, it is desirable to change the frequency of the AC voltage as appropriate depending on the characteristics of the diaphragm 1900. For example, when the diaphragm 1900 vibrates in the vertical direction, it is said that people have the highest sensitivity to vibrations of about 410 Hz. Also, when the diaphragm 1900 vibrates in the horizontal direction, it is said that people have the highest sensitivity to vibrations of about 12 Hz. Furthermore, at frequencies of 34 Hz ​​or higher, it is said that people have a higher sensitivity to vibrations in the vertical direction than in the horizontal direction.

[0206] Therefore, when the diaphragm 1900 vibrates in the vertical direction, it is desirable to set the frequency of the AC voltage to a value within a range including, for example, 410 Hz. When the diaphragm 1900 vibrates in the horizontal direction, it is desirable to set the frequency of the AC voltage to a value within a range including, for example, 12 Hz. Note that the peak voltage and frequency of the AC voltage may be appropriately adjusted depending on the performance of the diaphragm 1900.

[0207] According to this configuration, it is possible for the user 230 to recognize that an object has been touched by vibrations from the feet. In addition, in the case of this configuration, it is also possible to set the display of the floating image 3 not to change when an object is touched, so that even if a third party peeks at the touch operation, the possibility of the input contents being known is reduced, making it possible to further improve security.

[0208] Although various embodiments have been described above in detail, the present invention is not limited to the above-described embodiments, and various modified examples are included. For example, the above-described embodiments are detailed descriptions of the entire system in order to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0209] 1...display device, 2...retroreflective member, 3...spatial image (space-floating image), 105...window glass, 100...transparent member, 101...polarized light separation member, 12...absorptive polarizing plate, 13...light source device, 54...light direction conversion panel, 151...retroreflective member, 102, 202...LED substrate, 203...light guide, 205, 271...reflective sheet, 206, 270...phase difference plate, 300...space-floating image, 301...ghost of space-floating image image, 302...ghost image of floating in space image, 230...user, 1000...space floating image display device, 1110...control unit, 1160...image control unit, 1180...imaging unit, 1102...image display unit, 1350...air operation detection unit, 1351...air operation detection sensor, 1500...virtual light source, 1510...virtual shadow, 1610...input content display area, 1700...touch pen, 1800...wearable terminal, 1900...diaphragm.

Claims

1. A space floating image display device, A display device for displaying an image; A retroreflective member that reflects image light from the display device and forms a floating image in the air by the reflected light; A sensor for detecting a position of a finger of a user performing a touch operation on one or more objects displayed on the space floating image; A control unit; Equipped with The control unit controls image processing for the image displayed on the display device based on the position of the user's finger detected using the sensor, so that the space-floating image display device displays a virtual shadow of the user's finger on the display surface of the space-floating image on which there is no physical contact surface, a position of the virtual shadow displayed on a display surface of the floating-in-space image is a position specified from a positional relationship between a position of a virtual light source and a position of the user's finger detected using the sensor, and the position of the virtual light source is set at a position at an infinite distance from the display surface of the floating-in-space image; When the position of the tip of the user's finger approaches the display surface of the space floating image so that the distance in the normal direction to the display surface of the space floating image becomes small on the front side of the display surface of the space floating image as seen from the user, the display of the virtual shadow of the user's finger is continued until the distance in the normal direction between the position of the tip of the user's finger and the display surface of the space floating image becomes 0; A floating image display device.

2. 2. The space floating image display device according to claim 1, When the position of the tip of the user's finger changes in the normal direction on the front side of the display surface of the floating image in space as seen from the user, the left-right position of the tip of the virtual shadow displayed in the floating image in space changes on the display surface of the floating image in space. A floating image display device.

3. 3. The space floating image display device according to claim 2, With respect to the change in the normal direction of the position of the tip of the user's finger, the left-right position of the tip of the virtual shadow displayed in the space floating image on the display surface of the space floating image changes linearly. A floating image display device.

4. 2. The space floating image display device according to claim 1, An imaging unit that captures an image of a user's hand or arm, a position of the virtual light source is changed depending on whether the finger of the user performing the touch operation is of the right hand or the left hand, so that when the finger of the user performing the touch operation on one or more objects displayed in the space floating image is of the right hand, the virtual shadow is displayed in the space floating image at a position to the left of the tip of the finger as seen from the user, and when the finger of the user performing the touch operation on one or more objects displayed in the space floating image is of the left hand, the virtual shadow is displayed in the space floating image at a position to the right of the tip of the finger as seen from the user, A floating image display device.

5. 2. The space floating image display device according to claim 1, The control unit detects the position of the tip of the finger on the display surface of the space floating image and the height position of the tip of the finger relative to the display surface using the sensor that detects the position of the user's finger. A floating image display device.

6. 2. The space floating image display device according to claim 1, The presence or absence of the user's finger touching the display surface of the floating image in space is detected by a sensor different from a sensor that detects the position of the user's finger. A floating image display device.

7. 2. The space floating image display device according to claim 1, The position of the virtual light source is A position where a virtual light source installation angle defined by the angle between a normal line extending from a central point of the display surface of the space floating image toward the user side and a line connecting the virtual light source and the central point of the display surface of the space floating image is 20° or more. A floating image display device.

8. 2. The space floating image display device according to claim 1, The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image changes in conjunction with the angle of the user's finger imaged by the imaging unit of the space floating image display device. A floating image display device.

9. 2. The space floating image display device according to claim 1, The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image is a fixed angle without being linked to the angle of the user's finger captured by the imaging unit of the space floating image display device. A floating image display device.

10. 2. The space floating image display device according to claim 1, When a distance in the normal direction between the position of the tip of the user's finger and the display surface of the floating-in-space image becomes 0, the virtual shadow is displayed in the floating-in-space image such that the tip of the virtual shadow and the tip of the user's finger come into contact with each other in the left-right direction on the display surface of the floating-in-space image. A floating image display device.

Images