Spatial-suspended image display device

Abstract

To provide a more suitable spatial-suspended image display device which contributes to "3. Good Health and Well-Being", "9. Industry, Innovation and Infrastructure", and "11. Sustainable Cities and Communities" of Sustainable Development Goals.SOLUTION: A spatial-suspended image display device includes a display part, and a retroreflection plate which reflects image light from the display part, and forms a spatial-suspended image in the air by the reflected light, wherein there is a region where an object is displayed in a display range of the spatial-suspended image, a black display region surrounding the region is arranged, a physical frame is arranged so as to surround the spatial-suspended image from its periphery, the physical frame forms an opening window having such a cover structure as to cover a storage part storing the display part and the retroreflection plate, and a light-shielding plate is arranged between the opening window and the storage part storing the display part and the retroreflection plate inside the cover structure, and extends toward the storage part storing the display part and the retroreflection plate from at least both the upper end and the lower end of the opening window.SELECTED DRAWING: Figure 46

Description

[Technical field]

[0001] The present invention relates to a space floating image display device. [Background technology]

[0002] As a space-floating information display system, a video display device that displays an image directly to the outside and a display method that displays the image as a spatial screen are already known. In addition, a detection system that reduces false detections of operations on the operation surface of the displayed spatial image is also disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-128722 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, a touch operation on a floating-in-space image is not performed on a physical button, touch panel, etc. For this reason, there are cases where the user cannot recognize whether or not a touch operation has been performed.

[0005] An object of the present invention is to provide a more suitable space floating image display device. [Means for solving the problem]

[0006] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problem, but to give one example, a space floating image display device includes a display unit that displays an image, and a retroreflector that reflects image light from the display unit and forms a space floating image in the air with the reflected light, and in the display range of the space floating image, there is an area where an object is displayed, and a black display area that surrounds the area where the object is displayed is arranged, and the device has a physical frame arranged to surround the space floating image from the periphery, and the physical frame forms an opening window of a cover structure that covers the display unit and a storage unit that stores the retroreflector, and inside the cover structure, a light shielding plate is arranged between the opening window and the storage unit that stores the display unit and the retroreflector, and extends from at least both the upper and lower ends of the opening window toward the storage unit that stores the display unit and the retroreflector. Effect of the Invention

[0007] According to the present invention, a more suitable space floating image display device can be realized. Other objects, configurations and effects will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of a usage form of a space floating image display device according to an embodiment of the present invention; [Diagram 2] 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; [Figure 3A] 1A and 1B are diagrams illustrating an example of a method for installing a space floating image display device. [Figure 3B] 11A and 11B are diagrams showing another example of a method for installing the space floating image display device. [Figure 3C] FIG. 1 is a diagram showing a configuration example of a space floating image display device. [Figure 4] FIG. 11 is a diagram showing another example of the main part configuration of the space floating image display device according to an embodiment of the present invention. [Diagram 5] FIG. 11 is an explanatory diagram for explaining the function of a sensing device used in the space floating image display device. [Figure 6] FIG. 1 is an explanatory diagram of the principle of 3D image display used in a space floating image display device. [Figure 7] FIG. 1 is an explanatory diagram of a measurement system for evaluating the characteristics of a reflective polarizing plate. [Figure 8] FIG. 11 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the light incidence angle. [Figure 9] FIG. 4 is a characteristic diagram showing transmittance characteristics with respect to the light incidence angle of the reflection axis of a reflective polarizing plate. [Figure 10] FIG. 11 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the light incidence angle. [Figure 11] FIG. 4 is a characteristic diagram showing transmittance characteristics with respect to the light incidence angle of the reflection axis of a reflective polarizing plate. [Figure 12] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 13] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 14] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 15] 1 is a layout diagram showing a main part of a space floating image display device according to an embodiment of the present invention; [Figure 16] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 17] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 18] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 19] FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. [Figure 20] 1 is an explanatory diagram for explaining a light source diffusion characteristic of a video display device. [Figure 21] FIG. 1 is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Figure 22] FIG. 1 is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Figure 23]FIG. 1 is a cross-sectional view showing a configuration of a video display device. [Figure 24] FIG. 1 is an explanatory diagram for explaining the principle of occurrence of a ghost image in the prior art. [Diagram 25] 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. [Figure 26] FIG. 2 is a diagram illustrating an example of a display on a display device according to an embodiment of the present invention. [Figure 27] 11A to 11C are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Figure 28] 11A to 11C are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Figure 29] 11A to 11C are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. [Diagram 30] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Diagram 31] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Diagram 32] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. [Diagram 33] FIG. 13 is a diagram illustrating a method for setting a virtual light source. [Diagram 34] FIG. 11 is a configuration diagram showing an example of a method for detecting the position of a finger. [Diagram 35] FIG. 11 is a configuration diagram showing another example of a method for detecting the position of a finger. [Diagram 36] FIG. 11 is a configuration diagram showing another example of a method for detecting the position of a finger. [Figure 37] 13A to 13C are diagrams illustrating a method of assisting a touch operation by displaying input contents. [Figure 38] 11A and 11B are diagrams illustrating a method of assisting a touch operation by highlighting input contents. [Figure 39] 11A and 11B are diagrams illustrating an example of a method for assisting a touch operation by vibration. [Diagram 40] 13A to 13C are diagrams illustrating another example of a method for assisting a touch operation by vibration. [Diagram 41]13A to 13C are diagrams illustrating another example of a method for assisting a touch operation by vibration. [Figure 42A] FIG. 1 is a diagram illustrating an example of a display of a space floating image according to an embodiment of the present invention. [Figure 42B] FIG. 1 is a diagram illustrating an example of a display of a space floating image according to an embodiment of the present invention. [Diagram 43] FIG. 1 is a diagram illustrating an example of a configuration of a space floating image display device according to an embodiment of the present invention. [Diagram 44] FIG. 1 is a diagram illustrating an example of a configuration example of a part of a space floating image display device according to an embodiment of the present invention. [Diagram 45] FIG. 1 is a diagram illustrating an example of a display of a space floating image according to an embodiment of the present invention. [Figure 46] FIG. 1 is a diagram illustrating an example of a configuration of a space floating image display device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiment of the present invention will be described in detail with reference to the drawings. The present invention is not limited to the description of the embodiment, and various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. In addition, in all the drawings for explaining the present invention, the same reference numerals are given to the parts having the same functions, and the repeated description may be omitted. In the following description of the embodiment, the image floating in space is expressed by the term "space floating image". Instead of this term, it may be expressed as "aerial image", "space image", "floating image", "floating optical image of a displayed image", "floating optical image of a displayed image", etc. The term "floating image of a space" mainly used in the description of the embodiment is used as a representative example of these terms.

[0010] The following embodiment relates to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member.

[0011] According to the following embodiment, an image display device suitable for use in, for example, bank ATMs, station ticket vending machines, digital signage, etc., can be realized. For example, currently, touch panels are usually used in bank ATMs, station ticket vending machines, etc., but by using a transparent glass surface or a light-transmitting plate material, high-resolution image information can be displayed in a state of floating in space on the glass surface or light-transmitting plate material. At this time, by making the divergence angle of the emitted image light small, i.e., an acute angle, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflective member, so that the light utilization efficiency is high, and ghost images that occur in addition to the main space floating image, which was a problem in the conventional retroreflective method, can be suppressed, and a clear space floating image can be obtained. In addition, a device including the light source of this embodiment can provide a novel and highly usable space floating image display device (space floating image display system) that can significantly reduce power consumption. In addition, for example, a space floating image display device for vehicles that can display a so-called unidirectional space floating image that can be viewed inside and / or outside the vehicle can be provided. In the following examples, the retroreflective member may be a plate-like member, and may be referred to as a retroreflective plate.

[0012] On the other hand, in the conventional technology, an organic EL panel or a liquid crystal panel is combined with a retroreflective member 151 as a high-resolution color display image source 150. In the conventional technology, the image light is diffused at a wide angle, so in addition to the reflected light normally reflected by the retroreflective member 151, ghost images 301 and 302 are generated by the image light obliquely incident on the retroreflective member 2a as shown in FIG. 24, which impairs the image quality of the floating in space image. Also, as shown in FIG. 23, in addition to the normal floating in space image 300, a first ghost image 301, a second ghost image 302, and the like are generated multiple times. For this reason, the same floating in space image, which is a ghost image, can be observed by people other than the observer, which poses a major security problem.

[0013] <Space-floating image display device> Fig. 1 is a diagram showing an example of a usage form of a space-floating image display device according to an embodiment of the present invention, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using Fig. 2, etc., but light with a narrow-angle directional characteristic and specific polarization is emitted from the image display device 1 as an image light beam, once enters the retroreflective member 2, is retroreflected and passes through a transparent member 100 (glass, etc.), and forms a real aerial image (space-floating image 3) on the outside of the glass surface.

[0014] In addition, in a store or the like, a space is divided by a show window (also called "window glass") 105, which is a light-transmitting member such as glass. According to the space floating image display device of this embodiment, it is possible to transmit the floating image through such a transparent member and display it in one direction to the outside and / or inside of the store (space).

[0015] 1A, the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing the window glass 105 with a means for reflecting a specific polarized wave, it is possible to form an aerial image at a desired position inside the store.

[0016] FIG. 1(B) is a schematic block diagram showing the configuration of the above-mentioned display device 1. The display device 1 includes a video display unit that displays an original image of an aerial image, a video control unit that converts the input video to match the resolution of the panel, and a video signal receiving unit that receives a video signal. The video signal receiving unit supports wired input signals such as HDMI (High-Definition Multimedia Interface) input and wireless input signals such as Wi-Fi (Wireless Fidelity), and functions independently as a video receiving and display device, and can also display video information from a tablet, smartphone, etc. Furthermore, by connecting a stick PC or the like, it can be given the ability to perform calculation processing, video analysis processing, and the like.

[0017] Fig. 2 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention. The configuration of the space floating image display device will be described in more detail with reference to Fig. 2. As shown in Fig. 2(A), a display device 1 that diverges specific polarized image light at a narrow angle is provided in an oblique direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light having a narrow angle diffusion characteristic.

[0018] The image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 (in the figure, the polarization separation member 101 is formed into a sheet shape and adhered to the transparent member 100) having a film that selectively reflects the image light of a specific polarization provided on a transparent member 100, and enters the retroreflective member 2. A λ / 4 plate 21 is provided on the image light incidence surface of the retroreflective member 2. The image light is made to pass through the λ / 4 plate 21 twice, when it enters the retroreflective member 2 and when it leaves, and is polarized and converted from the specific polarization to the other polarization. Here, the polarization separation member 101 that selectively reflects the image light of a specific polarization has a property of transmitting the polarized light of the other polarization that has been polarized and converted, so that the image light of the specific polarization after the polarization conversion passes through the polarization separation member 101. The image light that has passed through the polarization separation member 101 forms a space floating image 3, which is a real image, outside the transparent member 100.

[0019] The light forming the floating image 3 is a collection of light rays that converge from the retroreflective member 2 to the optical image of the floating image 3, and these light rays continue to travel straight even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is an image with high directivity, unlike the diffuse image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2, when a user views the floating image 3 from the direction of the arrow A, the floating image 3 is viewed as a bright image. However, when another person views the floating image 3 from the direction of the arrow B, the floating image 3 cannot be viewed as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should be kept secret from people facing the user.

[0020] Depending on the performance of the retroreflective member 2, the polarization axis of the reflected image light may become misaligned. In this case, a part of the image light with the misaligned polarization axis is reflected by the above-mentioned polarization separation member 101 and returns to the display device 1. This light may be reflected again by the image display surface of the liquid crystal display panel 11 constituting the display device 1, generating a ghost image and degrading the image quality of the spatial floating image.

[0021] Therefore, in this embodiment, an absorptive polarizing plate 12 is provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorptive polarizing plate 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorptive polarizing plate 12, thereby suppressing the re-reflection. This makes it possible to prevent degradation of image quality due to ghost images of spatially floating images.

[0022] The above-mentioned polarization separation member 101 may be formed, for example, of a reflective polarizing plate or a metal multilayer film that reflects a specific polarized wave.

[0023] Next, Fig. 2(B) shows the surface shape of a retroreflective member manufactured by Nippon Carbide Industries Co., Ltd., which was used in this study as a representative retroreflective member 2. Light rays incident inside the regularly arranged hexagonal prisms are reflected by the walls and bottoms of the hexagonal prisms and emitted as retroreflected light in a direction corresponding to the incident light, and a real image floating in space is displayed based on the image displayed on the display device 1.

[0024] The resolution of this floating image in space depends not only on the resolution of the liquid crystal display panel 11, but also on the outer shape D and pitch P of the retroreflective portion of the retroreflective member 2 shown in Fig. 2(B). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is about 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, one pixel of the floating image in space will be equivalent to 300 μm. For this reason, the effective resolution of the floating image in space will be reduced to about 1 / 3.

[0025] Therefore, in order to make the resolution of the spatial floating image equal to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective material and the pixels of the liquid crystal display panel, it is preferable to design the pitch ratio of each to be a different integer multiple of one pixel. In addition, it is preferable to arrange the shape so that none of the sides of the retroreflective portion overlaps with any of the sides of one pixel of the liquid crystal display panel.

[0026] On the other hand, in order to manufacture retroreflective members at low cost, it is recommended to use a roll press method for molding. Specifically, this is a method in which the retroreflective parts are aligned and shaped on a film, in which the inverse shape of the shape to be shaped is formed on the roll surface, an ultraviolet curing resin is applied onto a base material for fixing, and the film is passed between the rolls to form the required shape, and then the resin is irradiated with ultraviolet light to cure it, thereby obtaining the retroreflective member 2 of the desired shape.

[0027] <<How to install the space floating image display device>> Next, a method of installing the space-floating image display device will be described. The installation method of the space-floating image display device can be freely changed depending on the usage form. FIG. 3A is a diagram showing an example of a method of installing the space-floating image display device. The space-floating image display device shown in FIG. 3A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in FIG. 3A, the space-floating image display device is installed so that the transparent member 100 faces upward, and the space-floating image 3 is formed above the space-floating image display device.

[0028] Fig. 3B is a diagram showing another example of a method of installing the space-floating image display device. The space-floating image display device shown in Fig. 3B is installed vertically so that the surface on which the space-floating image 3 is formed faces the side (toward the user 230). That is, in Fig. 3B, the space-floating image display device is installed so that the transparent member 100 faces the side, and the space-floating image 3 is formed on the side of the space-floating image display device (toward the user 230).

[0029] <<Configuration of the space floating image display device>> Next, a description will be given of the configuration of the space-floating image display device 1000. Fig. 3C is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

[0030] The space floating image display device 1000 includes a retroreflective section 1101, an image display section 1102, a light guide 1104, a light source 1105, a power source 1106, an operation input section 1107, a non-volatile memory 1108, a memory 1109, a control section 1110, an image signal input section 1131, an audio signal input section 1133, a communication section 1132, an aerial operation detection sensor 1351, an aerial operation detection section 1350, an audio output section 1140, an image control section 1160, a storage section 1170, an imaging section 1180, etc.

[0031] Each component of the space floating image display device 1000 is disposed in a housing 1190. Note that the imaging unit 1180 and the mid-air operation detection sensor 1351 shown in FIG.

[0032] The retroreflection portion 1101 in Fig. 3C corresponds to the retroreflection member 2 in Fig. 2. The retroreflection portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected by the retroreflection portion 1101, the light outputted to the outside of the space-floating image display device 1000 forms the space-floating image 3.

[0033] Image display unit 1102 in Fig. 3C corresponds to liquid crystal display panel 11 in Fig. 2. Light source 1105 in Fig. 3C corresponds to light source device 13 in Fig. 2. Moreover, image display unit 1102, light guide 1104, and light source 1105 in Fig. 3C correspond to display device 1 in Fig. 2.

[0034] The video display unit 1102 is a display unit that generates a video by modulating transmitted light based on a video signal input under the control of a video control unit 1160 described later. The video display unit 1102 corresponds to the liquid crystal display panel 11 in Fig. 2. For example, a transmissive liquid crystal panel is used as the video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the video display unit 1102.

[0035] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED light source or a laser light source. The power source 1106 converts an AC current input from the outside into a DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each unit in the space floating image display device 1000.

[0036] The light guide 1104 guides the light generated by the light source 1105 and irradiates the light to the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can be called a backlight of the image display unit 1102. There are various types of combination of the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described in detail later.

[0037] The aerial operation detection sensor 1351 is a sensor that detects an operation of the floating-in-space image 3 by a finger of the user 230. The aerial operation detection sensor 1351 senses, for example, a range that overlaps with the entire display range of the floating-in-space image 3. Note that the aerial operation detection sensor 1351 may only sense a range that overlaps with at least a portion of the display range of the floating-in-space image 3.

[0038] Specific examples of the aerial operation detection sensor 1351 include a distance sensor using invisible light such as infrared light, an invisible light laser, ultrasonic waves, etc. The aerial operation detection sensor 1351 may be configured to detect coordinates on a two-dimensional plane by combining a plurality of sensors. The aerial operation detection sensor 1351 may be configured with a ToF (Time of Flight) LiDAR (Light Detection and Ranging) or an image sensor.

[0039] The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect a touch operation or the like performed by a user with a finger on an object displayed as the floating-in-space image 3. Such sensing can be performed using existing technology.

[0040] The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and performs operations such as determining whether or not the finger of the user 230 has touched an object in the floating in space image 3 and calculating the position (contact position) where the finger of the user 230 has touched the object based on the sensing signal. The aerial operation detection unit 1350 is configured with a circuit such as an FPGA (Field Programmable Gate Array). Some of the functions of the aerial operation detection unit 1350 may be realized by software using a spatial operation detection program executed by the control unit 1110, for example.

[0041] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally separately from the space-floating image display device 1000. When provided separately from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or image signal transmission path.

[0042] Also, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the air-floating image display device 1000 without the aerial operation detection function is used as the main body, and only the aerial operation detection function can be added as an option. Also, a configuration in which only the aerial operation detection sensor 1351 is provided separately, and the aerial operation detection unit 1350 is built into the air-floating image display device 1000 may be used. In cases in which it is desired to more freely arrange the aerial operation detection sensor 1351 relative to the installation position of the air-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

[0043] The imaging unit 1180 is a camera having an image sensor, and captures the space near the floating-in-space image 3 and / or the face, arms, fingers, etc. of the user 230. A plurality of imaging units 1180 may be provided. By using a plurality of imaging units 1180 or an imaging unit with a depth sensor, the mid-air operation detection unit 1350 can be assisted in the detection process of the touch operation of the floating-in-space image 3 by the user 230. The imaging unit 1180 may be provided separately from the floating-in-space image display device 1000. When the imaging unit 1180 is provided separately from the floating-in-space image display device 1000, it is sufficient to configure it so that an imaging signal can be transmitted to the floating-in-space image display device 1000 via a wired or wireless communication connection path or the like.

[0044] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that detects whether or not an object has intruded into a plane (intrusion detection plane) including the display surface of the floating image 3, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object (e.g., a user's finger) that has not intruded into the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

[0045] In such a case, the distance between the object and the intrusion detection plane can be calculated by using information such as object depth calculation information based on the captured images of the multiple imaging units 1180 and object depth information by the depth sensor. These pieces of information and various pieces of information such as the distance between the object and the intrusion detection plane are used for various display controls for the floating in space image 3.

[0046] Furthermore, without using the aerial operation detection sensor 1351, the aerial operation detection section 1350 may detect a touch operation on the floating-in-space image 3 by the user 230 based on the captured image by the imaging section 1180.

[0047] Also, the imaging unit 1180 may capture an image of the face of the user 230 operating the floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not a third person is standing around or behind the user 230 operating the floating image 3 and peeking at the operation of the user 230 on the floating image 3, the imaging unit 1180 may capture an image of a range including the user 230 operating the floating image 3 and the surrounding area of ​​the user 230.

[0048] The operation input unit 1107 is, for example, an operation button or a light receiving unit of a remote controller, and inputs a signal for an operation different from the air operation (touch operation) by the user 230. In addition to the above-mentioned user 230 who touches the space floating image 3, the operation input unit 1107 may be used by, for example, an administrator to operate the space floating image display device 1000.

[0049] The video signal input unit 1131 connects to an external video output device and inputs video data. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio output unit 1140 can output audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may also output built-in operation sounds and error warning sounds.

[0050] The non-volatile memory 1108 stores various data used by the space-floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations to be displayed on the space-floating image 3, display icons, data of objects to be operated by user operations, layout information, etc. The memory 1109 stores image data to be displayed as the space-floating image 3, data for controlling the device, etc.

[0051] The control unit 1110 controls the operation of each connected unit. The control unit 1110 may also perform calculation processing based on information acquired from each unit in the space floating image display device 1000 in cooperation with a program stored in the memory 1109. The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless interface. Various data such as video data, image data, and audio data are transmitted and received by communication via the communication unit 1132.

[0052] The storage unit 1170 is a storage device that records various types of data & information such as video data, image data, audio data, etc. In the storage unit 1170, for example, various types of information such as video data, image data, audio data, etc. may be recorded in advance at the time of product shipment. In addition, the storage unit 1170 may record various types of information such as various types of data such as video data, image data, audio data, etc. acquired from an external device, an external server, etc. via the communication unit 1132.

[0053] The video data, image data, etc. recorded in the storage unit 1170 are output as the space floating image 3 via the video display unit 1102 and the retroreflection unit 1101. The video data, image data, etc. of the display icons and objects for the user to operate, which are displayed as the space floating image 3, are also recorded in the storage unit 1170.

[0054] Layout information of display icons and objects displayed as the space floating image 3, various metadata information related to the objects, and the like are also recorded in the storage unit 1170. The audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

[0055] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. The video control unit 1160 performs control of video switching, such as determining which video signal to input to the video display unit 1102, between the video signal to be stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131, for example.

[0056] In addition, the image control unit 1160 may generate a superimposed image signal by superimposing the image signal to be stored in memory 1109 and the image signal input from the image signal input unit 1131, and input the superimposed image signal to the image display unit 1102, thereby performing control to form a composite image as a floating-in-space image 3.

[0057] Furthermore, the video control unit 1160 may control image processing of the video signal input from the video signal input unit 1131, the video signal to be stored in the memory 1109, etc. Examples of image processing include scaling processing for enlarging, reducing, transforming, etc. an image, brightness adjustment processing for changing the luminance, contrast adjustment processing for changing the contrast curve of the image, and Retinex processing for decomposing an image into light components and changing the weighting of each component.

[0058] Furthermore, the video control unit 1160 may perform special effect video processing or the like for assisting the aerial operation (touch operation) of the user 230 on the video signal input to the video display unit 1102. The special effect video processing is performed, for example, based on the detection result of the touch operation of the user 230 by the aerial operation detection unit 1350 or the captured image of the user 230 by the imaging unit 1180.

[0059] As explained above, the space-floating image display device 1000 is equipped with various functions. However, the space-floating image display device 1000 does not need to have all of these functions, and can have any configuration as long as it has the function of forming the space-floating image 3.

[0060] <Space-floating image display device 2> FIG. 4 is a diagram showing another example of the main part configuration of a space floating image display device according to an embodiment of the present invention. The display device 1 includes a liquid crystal display panel 11, which is an image display element, and a light source device 13 that generates light of a specific polarized wave having a narrow-angle diffusion characteristic. The display device 1 is composed of a liquid crystal display panel having a screen size ranging from a small one having a screen size of about 5 inches to a large one having a screen size of more than 80 inches. The folding mirror 22 has a transparent member 100 as a substrate. A polarization separation member 101 that selectively reflects image light of a specific polarized wave, such as a reflective polarizing plate, is provided on the surface of the transparent member 100 on the display device 1 side, and reflects the image light from the liquid crystal display panel 11 toward the retroreflector 2. This allows the folding mirror 22 to function as a mirror. The image light of a specific polarized wave from the display device 1 is reflected by the polarization separation member 101 (a sheet-shaped polarization separation member 101 is adhered in the figure) provided on the transparent member 100, and enters the retroreflector 2. Instead of the polarization separation member 101, an optical film having polarization separation properties may be vapor-deposited on the surface of the transparent member 100.

[0061] A λ / 4 plate 21 is provided on the light incident surface of the retroreflector, and the image light is polarized and converted by passing through it twice, converting a specific polarized wave into another polarized wave with a phase difference of 90°. As a result, the image light after retroreflection is transmitted through the polarization separation member 101, and a real image, a floating image 3, is displayed on the outside of the transparent member 100.

[0062] Here, since the polarization axis becomes misaligned due to retroreflection in the above-mentioned polarization separation member 101, a part of the image light is reflected and returns to the display device 1. This light is reflected again by the image display surface of the liquid crystal display panel 11 constituting the display device 1, generating a ghost image and significantly degrading the image quality of the spatial floating image.

[0063] Therefore, in this embodiment, an absorptive polarizing plate 12 may be provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted, and the reflected light from the above-mentioned polarization separation member 101 is absorbed, thereby preventing degradation of image quality due to ghost images of the spatially floating images. In addition, in order to reduce degradation of image quality due to sunlight or illumination light outside the set, it is preferable to provide an absorptive polarizing plate 102 on the surface of the transparent member 100 on the image light transmission output side.

[0064] Next, in order to sense the distance and position relationship between the object and the sensor 44 for the space floating image obtained by the space floating image display device described above, the sensor 44 having a TOF (Time of Fly) function is arranged in multiple layers as shown in FIG. 5, and it becomes possible to sense the coordinate in the depth direction and the moving direction and moving speed of the object in addition to the coordinate in the planar direction of the object. In order to read the two-dimensional distance and position, multiple combinations of infrared light emitting units and light receiving units are arranged in a straight line, light from the light emitting point is irradiated on the object, and the reflected light is received by the light receiving unit. The distance to the object becomes clear by the product of the difference between the time of light emission and the time of light reception and the speed of light. In addition, the coordinate on the plane can be read from the coordinate at the part where the difference between the time of light emission and the time of light reception is the smallest with multiple light emitting units and light receiving units. As described above, by combining the coordinate of the object on the plane (two dimensions) with multiple sensors described above, three-dimensional coordinate information can also be obtained.

[0065] Furthermore, a method for obtaining a three-dimensional space-floating image as the above-mentioned space-floating image display device will be described with reference to FIG. 6. FIG. 6 is an explanatory diagram of the principle of three-dimensional image display used in the space-floating image display device. A horizontal lenticular lens is arranged in accordance with the pixels of the image display screen of the liquid crystal display panel 11 of the display device 1 shown in FIG. 4. As a result, in order to display motion parallax from three directions, P1, P2, and P3 in the horizontal direction of the screen as shown in FIG. 6, images from the three directions are treated as one block for every three pixels, image information from the three directions is displayed for each pixel, and the light emission direction is controlled by the action of the corresponding lenticular lens (shown by vertical lines in FIG. 6) to separate and emit the light in three directions. As a result, a three-parallax stereoscopic image can be displayed.

[0066] <Reflective polarizing plate> In the space floating image display device of this embodiment, the polarization separation member 101 is used to improve the contrast performance, which determines the image quality, more than a general half mirror. As an example of the polarization separation member 101 of this embodiment, the characteristics of a reflective polarizing plate will be described. FIG. 7 is an explanatory diagram of a measurement system for evaluating the characteristics of a reflective polarizing plate. The transmission characteristics and reflection characteristics of the reflective polarizing plate in FIG. 7 with respect to the incident angle of a light beam from a direction perpendicular to the polarization axis are shown in FIG. 8 and FIG. 9, respectively, as V-AOI. Similarly, the transmission characteristics and reflection characteristics of the reflective polarizing plate with respect to the incident angle of a light beam from a direction horizontal to the polarization axis are shown in FIG. 10 and FIG. 11, respectively, as H-AOI.

[0067] In the characteristic graphs of Figs. 8 to 11 (each shown in color), the angle (deg) values ​​shown in the margins on the right side are shown from the top in ascending order of the vertical axis, i.e., the transmittance (%). For example, in Fig. 8, in the range of the horizontal axis showing light with a wavelength of approximately 400 nm to 800 nm, the transmittance is highest when the vertical (V) angle is 0 degrees (deg), and the transmittance decreases in the order of 10 degrees, 20 degrees, 30 degrees, and 40 degrees. In Fig. 9, in the range of the horizontal axis showing light with a wavelength of approximately 400 nm to 800 nm, the transmittance is highest when the vertical (V) angle is 0 degrees (deg), and the transmittance decreases in the order of 10 degrees, 20 degrees, 30 degrees, and 40 degrees. In Fig. 10, in the range of the horizontal axis showing light with a wavelength of approximately 400 nm to 800 nm, the transmittance is highest when the horizontal (H) angle is 0 degrees (deg), and the transmittance decreases in the order of 10 degrees and 20 degrees. Also, in FIG. 11, in the range of light with wavelengths of approximately 400 nm to 800 nm on the horizontal axis, the transmittance is highest when the angle in the horizontal (H) direction is 0 degrees (deg), and the transmittance decreases in the order of 10 degrees and 20 degrees.

[0068] As shown in Figures 8 and 9, the reflective polarizing plate with a grid structure has poor characteristics for light coming from a direction perpendicular to the polarization axis. For this reason, specifications along the polarization axis are desirable, and the light source of this embodiment capable of emitting the image light from the liquid crystal display panel at a narrow angle is an ideal light source. Similarly, the characteristics in the horizontal direction also deteriorate for light coming from an oblique angle. Taking the above characteristics into consideration, the following describes an example of the configuration of this embodiment in which a light source capable of emitting the image light from the liquid crystal display panel at a narrower angle is used as the backlight for the liquid crystal display panel. This makes it possible to provide a high-contrast floating image.

[0069] <Display device> Next, the display device 1 of this embodiment will be described with reference to the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes the light source thereof, and Fig. 12 shows the light source device 13 together with the liquid crystal display panel as an exploded perspective view.

[0070] As shown by arrow 30 in Fig. 12, this liquid crystal display panel (image display element 11) receives an illumination light beam from light source device 13, which is a backlight device, that has a narrow-angle diffusion characteristic, i.e., has strong directivity (straightness) and characteristics similar to laser light with a polarization plane aligned in one direction. The liquid crystal display panel (image display element 11) modulates the received illumination light beam speed according to the input video signal. The modulated image light is reflected by retroreflective member 2 and passes through transparent member 100 to form a real image, which is a floating image in space (see Fig. 1).

[0071] In addition, in FIG. 12, the display device 1 is configured to include a liquid crystal display panel 11, a light direction conversion panel 54 that controls the directional characteristics of the light beam emitted from the light source device 13, and a narrow-angle diffusion plate (not shown) as necessary. That is, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarized wave is modulated in intensity by a video signal and emitted (see arrow 30 in FIG. 12). As a result, a desired image is projected as light of a specific polarized wave with high directivity (linearity) through the light direction conversion panel 54 toward the retroreflective member 2, and after being reflected by the retroreflective member 2, it is transmitted toward the eyes of a monitor outside the store (space) to form a space-floating image 3. Note that a protective cover 50 (see FIG. 13 and FIG. 14) may be provided on the surface of the light direction conversion panel 54 described above.

[0072] In this embodiment, in order to improve the utilization efficiency of the light flux 30 emitted from the light source device 13 and to significantly reduce power consumption, in the display device 1 including the light source device 13 and the liquid crystal display panel 11, the light from the light source device 13 (see the arrow 30 in FIG. 12) is projected toward the retroreflective member 2, and after being reflected by the retroreflective member 2, the directivity can be controlled so that the floating image is formed at a desired position by a transparent sheet (not shown) provided on the surface of the transparent member 100 (such as the window glass 105). Specifically, this transparent sheet controls the imaging position of the floating image while providing high directivity by optical components such as a Fresnel lens or a linear Fresnel lens. According to this configuration, the image light from the display device 1 efficiently reaches the observer outside the show window 105 (for example, the sidewalk) with high directivity (straightness) like laser light. As a result, it is possible to display high-quality floating images with high resolution and significantly reduce the power consumption of the display device 1 including the LED elements 201 of the light source device 13.

[0073] <Display device example 1> FIG. 13 shows an example of a specific configuration of the display device 1. In FIG. 13, a liquid crystal display panel 11 and a light direction conversion panel 54 are arranged on the light source device 13 of FIG. 12. This light source device 13 is formed, for example, of plastic on a case shown in FIG. 12, and is configured by storing an LED element 201 and a light guide 203 inside. As shown in FIG. 12, the end surface of the light guide 203 has a shape in which the cross-sectional area gradually increases toward the opposite side to the light receiving part in order to convert the divergent light from each LED element 201 into a substantially parallel light beam, and is provided with a lens shape that has an effect of gradually decreasing the divergence angle by multiple total reflections during propagation inside. The liquid crystal display panel 11 constituting the display device 1 is attached to the upper surface of the display device 1. In addition, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202, which implements their control circuits, are attached to one side of the case of the light source device 13 (the left end face in this example), and a heat sink, which is a member for cooling heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED board 202.

[0074] In addition, the frame (not shown) of the liquid crystal display panel attached to the upper surface of the case of the light source device 13 is configured by attaching the liquid crystal display panel 11 attached to the frame, and further by attaching an FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is an image display element, generates a display image by modulating the intensity of transmitted light based on a control signal from a control circuit (not shown) constituting an electronic device together with the LED element 201, which is a solid light source. At this time, the generated image light has a narrow diffusion angle and contains only a specific polarization component, so that a new image display device that is similar to a surface-emitting laser image source driven by a video signal can be obtained. Note that, at present, it is technically and safety impossible to obtain a laser light beam of the same size as the image obtained by the above-mentioned display device 1 using a laser device. Therefore, in this embodiment, light similar to the above-mentioned surface-emitting laser image light is obtained from a light beam from a general light source equipped with an LED element, for example.

[0075] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 13 as well as FIG.

[0076] 13 and 14 are cross-sectional views, and only one of the LED elements 201 constituting the light source is shown, and this is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving part of the end surface of the light guide and the LED element are attached while maintaining a predetermined positional relationship.

[0077] Each of the light guides 203 is formed of a light-transmitting resin such as acrylic. Although not shown in Figs. 13 and 14, the light receiving surface of the LED light at one end of the light guide 203 has a cone-shaped outer circumferential surface obtained by rotating a parabolic cross section, and a concave portion having a convex portion (i.e., a convex lens surface) is formed in the central area of ​​the top side of the outer circumferential surface. Also, the central area of ​​the flat portion at the other end of the light guide 203 has a convex lens surface protruding outward (or may be a concave lens surface recessed inward). These configurations will be described later in the explanation of Fig. 16 and the like. The outer shape of the light receiving portion of the light guide to which the LED element 201 is attached is a parabolic shape forming a cone-shaped outer circumferential surface, and is set within an angle range in which the light emitted from the LED element in the peripheral direction can be totally reflected inside, or a reflective surface is formed.

[0078] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed such that the LED elements 201 on the surface are positioned in the center of the recessed portion described above with respect to the LED collimator (light-receiving end surface 203a).

[0079] According to this configuration, the shape of light-receiving end surface 203a of light guide 203 makes it possible to extract the light emitted from LED element 201 as substantially parallel light, thereby improving the efficiency of use of the generated light.

[0080] As described above, the light source device 13 is configured by mounting a light source unit in which a plurality of LED elements 201 serving as light sources are arranged on the light receiving end surface 203a serving as a light receiving section provided on the end surface of the light guide 203. The light source device 13 converts the divergent light beam from the LED elements 201 into substantially parallel light by the lens shape of the light receiving end surface 203a of the end surface of the light guide, guides the light inside the light guide 203 (in a direction parallel to the drawing) as shown by the arrow, and emits the light toward the liquid crystal display panel 11 arranged substantially parallel to the light guide 203 by the light beam direction conversion means 204 (in a direction perpendicular to the front side of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of the light beam direction conversion means 204 depending on the shape of the inside or surface of the light guide.

[0081] The light beam direction conversion means 204 described above emits the light beam propagated inside the light guide toward the liquid crystal display panel 11 arranged substantially parallel to the light guide 203 (in a direction perpendicular to the front of the drawing) by changing the shape of the light guide surface or by providing, for example, a portion with a different refractive index inside the light guide. At this time, when the brightness of the screen center and the screen periphery are compared while facing the liquid crystal display panel 11 directly at the center of the screen and placing the viewpoint at the same position as the diagonal dimension of the screen, if the relative brightness ratio is 20% or more, there is no practical problem, and if it exceeds 30%, it is an even better characteristic.

[0082] 13 is a cross-sectional layout diagram for explaining the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including the above-mentioned light guide 203 and LED element 201. In Fig. 13, light source device 13 is composed of light guide 203 formed of, for example, plastic or the like and having light beam direction conversion means 204 on its surface or inside, LED element 201 as a light source, reflective sheet 205, retardation plate 206, lenticular lens, etc., and on the upper surface thereof is attached liquid crystal display panel 11 equipped with polarizing plates on the light source light entrance surface and image light exit surface.

[0083] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one polarized wave (e.g., P wave) 212 of the natural light beam 210 emitted from the LED element 201, and the reflected light beam is reflected by a reflective sheet 205 provided on one surface (lower surface in the figure) of the light guide 203, and directed again toward the liquid crystal display panel 11. Therefore, a retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, thereby converting the reflected light beam from P polarized light to S polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam, the light intensity of which is modulated by the image signal in the liquid crystal display panel 11 (arrow 213 in Figure 13), enters the retroreflective member 2 and, after reflection, passes through the window glass 105, as shown in Figure 1, to obtain a real image, that is, a floating image in space, inside or outside the store (space).

[0084] Fig. 14 is a cross-sectional layout diagram for explaining the configuration and action of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED element 201, similarly to Fig. 13. Light source device 13 is also similarly composed of light guide 203 formed of, for example, plastic, on the surface or inside of which light flux direction conversion means 204 is provided, LED element 201 as a light source, reflection sheet 205, retardation plate 206, lenticular lens, etc. On the upper surface of light source device 13, liquid crystal display panel 11 having polarizing plates on the light source light entrance surface and image light exit surface is attached as an image display element.

[0085] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (the lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, and selectively reflects one polarized wave (for example, S wave) 211 of the natural light beam 210 emitted from the LED light source 201, which is reflected by a reflective sheet 205 provided on one surface (the lower surface in the figure) of the light guide 203, and travels again toward the liquid crystal display panel 11. A retardation plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, thereby converting the reflected light beam from S polarized light to P polarized light, thereby improving the efficiency of use of the light source light as image light. The image light beam, which has been intensity-modulated by the image signal in the liquid crystal display panel 11 (arrow 214 in Figure 14), enters the retroreflective member 2 and, after reflection, passes through the window glass 105 as shown in Figure 1, to obtain a real image, that is, a floating image in space, inside or outside the store (space).

[0086] In the light source device shown in Fig. 13 and Fig. 14, in addition to the action of the polarizing plate provided on the light incident surface of the corresponding liquid crystal display panel 11, the polarized component on one side is reflected by the reflective polarizing plate, so that the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizing plate multiplied by the reciprocal of the cross transmittance obtained by the two polarizing plates attached to the liquid crystal display panel. This results in high contrast performance. In fact, it was confirmed by experiments that the contrast performance of the displayed image was improved by more than 10 times. As a result, a high-quality image was obtained that was comparable to that of a self-luminous organic EL.

[0087] <Display device example 2> Fig. 15 shows another example of a specific configuration of the display device 1. The light source device 13 in Fig. 15 is similar to the light source device in Fig. 17 and the like. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, and the like in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. In addition, LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and an LED board on which a control circuit is mounted are attached to one side of the case of the light source device 13, and a heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer side of the LED board (see also Figs. 17, 18, and the like).

[0088] The liquid crystal display panel frame attached to the top surface of the case is configured to have the liquid crystal display panel 11 attached to the frame, and further, an FPC (Flexible Printed Circuits) 403 (see FIG. 7) electrically connected to the liquid crystal display panel 11, etc. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light based on a control signal from a control circuit (not shown here) constituting the electronic device, together with the LED elements 14a and 14b, which are solid-state light sources.

[0089] <Example 1 of light source device for example 2 of display device> Next, the configuration of the optical system including the light source device housed in the case will be described in detail with reference to FIG. 17 as well as FIGS. 18(a) and (b).

[0090] 17 and 18 show LEDs 14a and 14b constituting the light source, which are attached at predetermined positions relative to the LED collimator 15. Each of the LED collimators 15 is formed of a light-transmitting resin such as acrylic. As shown in FIG. 18(b), the LED collimator 15 has a cone-shaped outer peripheral surface 156 obtained by rotating a parabolic cross section. The LED collimator 15 has a concave portion 153 in which a convex portion (i.e., a convex lens surface) 157 is formed at the center of the top (the side facing the LED substrate 102). The LED collimator 15 has a convex lens surface 154 protruding outward (or may have a concave lens surface recessed inward) at the center of the flat surface (the side opposite to the top). The parabolic surface 156 that forms the conical outer peripheral surface of the LED collimator 15 is set within an angle range that allows the light emitted in the peripheral direction from the LEDs 14a and 14b to be totally reflected therein, or a reflective surface is formed.

[0091] The LEDs 14a and 14b are disposed at predetermined positions on the surface of the LED substrate 102, which is the circuit substrate. The LED substrate 102 is disposed and fixed to the LED collimator 15 such that the LEDs 14a and 14b on the surface are located at the center of the recess 153.

[0092] According to this configuration, the light emitted from the LED 14a or 14b by the above-mentioned LED collimator 15, particularly the light emitted from the central portion toward the upper side (to the right in the figure), is collected and made parallel by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. The light emitted from the other portions toward the periphery is reflected by the parabolic surface that forms the conical outer circumferential surface of the LED collimator 15, and is similarly collected and made parallel. In other words, the LED collimator 15 that has a convex lens in its center and a parabolic surface formed on its periphery makes it possible to extract almost all of the light generated by the LED 14a or 14b as parallel light, and thus makes it possible to improve the utilization efficiency of the generated light.

[0093] A polarization conversion element 21 is provided on the light emission side of the LED collimator 15. As is clear from FIG. 18, the polarization conversion element 21 is configured by combining a columnar translucent member having a parallelogram cross section (hereinafter, parallelogram column) and a columnar translucent member having a triangular cross section (hereinafter, triangular column), and arranging them in an array in parallel to a plane perpendicular to the optical axis of the parallel light from the LED collimator 15. Furthermore, a polarization beam splitter (hereinafter, abbreviated as "PBS film") 211 and a reflection film 212 are alternately provided at the interface between the adjacent translucent members arranged in the array. In addition, a λ / 2 phase plate 213 is provided on the emission surface from which the light incident on the polarization conversion element 21 and transmitted through the PBS film 211 is emitted.

[0094] 18(a) is provided on the exit surface of the polarization conversion element 21. That is, the light emitted from the LED 14a or 14b is converted into parallel light by the action of the LED collimator 15, enters the synthesizing diffusion block 16, is diffused by the texture 161 on the exit side, and then reaches the light guide 17.

[0095] Light guide 17 is a rod-shaped member made of a light-transmitting resin such as acrylic and having a substantially triangular cross section (see FIG. 18(b)). As is clear from FIG. 17, light guide 17 includes a light guide light incident portion (surface) 171 that faces the exit surface of synthetic diffusion block 16 via first diffusion plate 18a, a light guide light reflecting portion (surface) 172 that forms an inclined surface, and a light guide light exit portion (surface) 173 that faces liquid crystal display panel 11, which is a liquid crystal display element, via second diffusion plate 18b.

[0096] 17, a number of reflective surfaces 172a and connecting surfaces 172b are alternately formed in a sawtooth shape on the light guide light reflecting portion (surface) 172 of the light guide 17. The reflective surface 172a (a line segment sloping upward to the right in the figure) forms an angle αn (n: natural number, for example, 1 to 130 in this example) with the horizontal plane indicated by the dashed line in the figure, and as an example, αn is set to 43 degrees or less (but 0 degree or more) here.

[0097] The light guide light incident portion (surface) 171 is formed in a curved convex shape inclined toward the light source side. Accordingly, the parallel light from the exit surface of the synthetic diffusion block 16 is diffused and incident via the first diffusion plate 18a, and as is clear from the figure, the parallel light is bent (deflected) slightly upward by the light guide light incident portion (surface) 171 and reaches the light guide light reflecting portion (surface) 172, where it is reflected and reaches the liquid crystal display panel 11 provided on the exit surface at the top of the figure.

[0098] According to the display device 1 described above in detail, it is possible to improve the light utilization efficiency and the uniform illumination characteristics, and at the same time, to manufacture the modularized S-polarized light source device in a small size and at low cost. In the above description, the polarization conversion element 21 is described as being attached after the LED collimator 15, but the present invention is not limited to this, and the same action and effect can be obtained by providing it in the optical path leading to the liquid crystal display panel 11.

[0099] In addition, in the light guide light reflection portion (surface) 172, a large number of reflection surfaces 172a and connection surfaces 172b are alternately formed in a sawtooth shape, and the illumination light beam is totally reflected on each reflection surface 172a and directed upward, and further enters the light direction conversion panel 54, which is provided with a narrow-angle diffusion plate in the light guide light exit portion (surface) 173 and controls the directional characteristics as a substantially parallel diffused light beam, and enters the liquid crystal display panel 11 from an oblique direction. In this embodiment, the light direction conversion panel 54 is provided between the light guide exit portion (surface) 173 and the liquid crystal display panel 11, but the same effect can be obtained by providing the light direction conversion panel 54 on the exit surface of the liquid crystal display panel 11.

[0100] <Example 2 of light source device for example 2 of display device> Another example of the configuration of the optical system of the light source device 13 and the like is shown in Fig. 19. In the example shown in Fig. 19, similar to the example shown in Fig. 18, a plurality of (two in this example) LEDs 14a, 14b constituting the light source are shown, which are attached at predetermined positions relative to the LED collimator 15. Each of the LED collimators 15 is formed of a light-transmitting resin such as acrylic.

[0101] 18, the LED collimator 15 shown in Fig. 19 has a cone-shaped outer circumferential surface 156 obtained by rotating a parabolic cross section. Also, the LED collimator 15 has a concave portion 153 (see Fig. 18(b)) in which a convex portion (i.e., a convex lens surface) 157 is formed at the center of the top (top side) of the LED collimator 15.

[0102] The center of the flat surface of the LED collimator 15 has a convex lens surface 154 (see FIG. 18(b)) protruding outward (or may have a concave lens surface recessed inward). The parabolic surface 156 forming the outer peripheral surface of the cone shape of the LED collimator 15 is set within an angle range that allows the light emitted from the LED 14a in the peripheral direction to be totally reflected therein, or a reflective surface is formed.

[0103] The LEDs 14a and 14b are disposed at predetermined positions on the surface of the LED substrate 102, which is the circuit substrate. The LED substrate 102 is disposed and fixed to the LED collimator 15 such that the LEDs 14a and 14b on the surface are located at the center of the recess 153.

[0104] According to this configuration, the light emitted from the LED 14a or 14b by the above-mentioned LED collimator 15, particularly the light emitted from the central portion toward the upper side (to the right in the figure), is collected and made parallel by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. The light emitted from the other portions toward the periphery is reflected by the parabolic surface that forms the conical outer circumferential surface of the LED collimator 15, and is similarly collected and made parallel. In other words, the LED collimator 15 that has a convex lens in its center and a parabolic surface formed on its periphery makes it possible to extract almost all of the light generated by the LED 14a or 14b as parallel light, and thus makes it possible to improve the utilization efficiency of the generated light.

[0105] A light guide 170 is provided on the light emission side of the LED collimator 15 via a first diffusion plate 18a. The light guide 170 is a rod-shaped member made of a light-transmitting resin such as acrylic and having a substantially triangular cross section (see FIG. 19(a)). As is clear from FIG. 19(a), the light guide 170 includes a light guide light entrance portion (surface) 171 that faces the emission surface of the diffusion block 16 via the first diffusion plate 18a, a light guide light reflection portion (surface) 172 that forms an inclined surface, and a light guide light emission portion (surface) 173 that faces the liquid crystal display panel 11, which is a liquid crystal display element, via a reflective polarizing plate 200.

[0106] If this reflective polarizer 200 is selected to have, for example, the property of reflecting P-polarized light (and transmitting S-polarized light), it will reflect the P-polarized light in the natural light emitted from the LED light source, pass through the λ / 4 plate 202 provided in the light guide light reflecting section 172 shown in Figure 19(b), be reflected by the reflective surface 201, and pass through the λ / 4 plate 202 again to be converted into S-polarized light, and all the light beams entering the liquid crystal display panel 11 will be unified into S-polarized light.

[0107] Similarly, if a material having the property of reflecting S-polarized light (transmitting P-polarized light) is selected as reflective polarizer 200, it will reflect the S-polarized light in the natural light emitted from the LED light source, pass through λ / 4 plate 202 provided on light guide light reflector 172 shown in Fig. 19(b), be reflected by reflecting surface 201, and pass through λ / 4 plate 202 again to be converted into P-polarized light, and all the light beams entering liquid crystal display panel 52 will be unified into P-polarized light. Polarization conversion can also be achieved with the configuration described above.

[0108] <Display device example 3> Next, another example of the specific configuration of the display device 1 (Example 3 of the display device) will be described with reference to Fig. 16. The light source device of this display device 1 converts a divergent light flux of light (a mixture of P-polarized light and S-polarized light) from an LED into a substantially parallel light flux by a collimator 18, and reflects the converted light flux toward the liquid crystal display panel 11 by the reflecting surface of the reflective light guide 304. Such reflected light is incident on a reflective polarizing plate 49 disposed between the liquid crystal display panel 11 and the reflective light guide 304. The reflective polarizing plate 49 transmits light of a specific polarized wave (e.g., P-polarized light) and causes the transmitted polarized light to be incident on the liquid crystal display panel 11. Here, polarized waves other than the specific polarized wave (e.g., S-polarized light) are reflected by the reflective polarizing plate 49 and head toward the reflective light guide 304 again.

[0109] The reflective polarizing plate 49 is disposed at an angle with respect to the liquid crystal display panel 11 so as not to be perpendicular to the chief ray of the light from the reflecting surface of the reflective light guide 304. The chief ray of the light reflected by the reflective polarizing plate 49 is incident on the transmitting surface of the reflective light guide 304. The light incident on the transmitting surface of the reflective light guide 304 is transmitted through the rear surface of the reflective light guide 304, transmitted through the λ / 4 plate 270 which is a retardation plate, and reflected by the reflecting plate 271. The light reflected by the reflecting plate 271 is transmitted through the λ / 4 plate 270 again, and transmitted through the transmitting surface of the reflective light guide 304. The light transmitted through the transmitting surface of the reflective light guide 304 is incident on the reflective polarizing plate 49 again.

[0110] At this time, the light re-entering the reflective polarizing plate 49 has passed through the λ / 4 plate 270 twice, and therefore its polarization has been converted to a polarized wave (for example, P-polarized light) that is transmitted through the reflective polarizing plate 49. Therefore, the light whose polarization has been converted is transmitted through the reflective polarizing plate 49 and enters the liquid crystal display panel 11. Note that, with regard to the polarization design related to the polarization conversion, the polarization may be configured inversely from the above explanation (S-polarized light and P-polarized light may be reversed).

[0111] As a result, the light from the LED is aligned to a specific polarization (e.g., P polarization), enters the liquid crystal display panel 11, and is brightness-modulated according to the video signal to display an image on the panel surface. As in the above example, multiple LEDs that make up the light source are shown (however, because this is a vertical cross section, only one is shown in FIG. 16), and these are attached at predetermined positions relative to the collimator 18.

[0112] Each of the collimators 18 is formed of a light-transmitting resin such as acrylic or glass. The collimator 18 may have a cone-shaped outer circumferential surface obtained by rotating a parabolic cross section. The top of the collimator 18 may have a concave portion with a convex portion (i.e., a convex lens surface) formed in the center. The center of the flat portion has a convex lens surface protruding outward (or a concave lens surface recessed inward). The parabolic surface forming the cone-shaped outer circumferential surface of the collimator 18 is set within an angle range that allows the light emitted from the LED in the peripheral direction to be totally reflected therein, or a reflective surface is formed thereon.

[0113] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED board 102. The LED board 102 is arranged and fixed to the collimator 18 so that the LEDs on the surface are located at the center of the apex of the convex cone shape (or in the concave portion if the apex has a concave portion).

[0114] According to this configuration, among the light emitted from the LED by the collimator 18, the light emitted from the central part is condensed into parallel light by the convex lens surface forming the outer shape of the collimator 18. The light emitted from the other parts toward the periphery is reflected by the parabolic surface forming the outer peripheral surface of the cone shape of the collimator 18, and is condensed into parallel light in the same manner. In other words, according to the collimator 18 having a convex lens in its center and a parabolic surface formed on its periphery, it is possible to extract almost all of the light generated by the LED as parallel light, and it is possible to improve the utilization efficiency of the generated light.

[0115] The above configuration is the same as that of the light source device of the image display device shown in FIG. 17, FIG. 18, etc. Furthermore, the light converted into approximately parallel light by the collimator 18 shown in FIG. 16 is reflected by the reflective light guide 304. Of the light, light of a specific polarized wave is transmitted through the reflective polarizing plate 49 by the action of the reflective polarizing plate 49, and light of the other polarized wave reflected by the action of the reflective polarizing plate 49 is transmitted through the light guide 304 again. The light is reflected by the reflector 271 located opposite to the liquid crystal display panel 11 with respect to the reflective light guide 304. At this time, the light is polarized and converted by passing twice through the λ / 4 plate 270, which is a retardation plate. The light reflected by the reflector 271 is transmitted through the light guide 304 again and enters the reflective polarizing plate 49 provided on the opposite surface. Since the incident light has been polarized and converted, it is transmitted through the reflective polarizing plate 49 and enters the liquid crystal display panel 11 with the same polarization direction. As a result, the light from the light source can be fully utilized, and the geometrical optical efficiency of the light is doubled. In addition, the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, so that the use of the light source device of this embodiment significantly improves the contrast ratio of the entire display device. By adjusting the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271, the reflection diffusion angle of the light at each reflective surface can be adjusted. In order to improve the uniformity of the light incident on the liquid crystal display panel 11, the surface roughness of the reflective surface of the reflective light guide 304 and the surface roughness of the reflector 271 can be adjusted for each design.

[0116] It should be noted that the λ / 4 plate 270, which is the retardation plate in Fig. 16, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light that is perpendicularly incident on the λ / 4 plate 270. In the configuration of Fig. 16, any retardation plate may be used as long as the phase changes by 90° (λ / 2) when polarized light passes through it twice. The thickness of the retardation plate may be adjusted according to the incidence angle distribution of the polarized light.

[0117] <Display device example 4> Further, another example (Example 4 of the display device) of the configuration of the optical system such as the light source device of the display device will be described with reference to FIG. 25. This is an example of the configuration in which a diffusion sheet is used instead of the reflective light guide 304 in the light source device of Example 3 of the display device. Specifically, two optical sheets (optical sheet 207A and optical sheet 207B) that convert the diffusion characteristics in the vertical direction and horizontal direction (front and back directions in the figure, not shown) of the drawing are used on the light emission side of the collimator 18, and the light from the collimator 18 is made to enter between the two optical sheets (diffusion sheets). This optical sheet may be one sheet instead of two sheets. In the case of a single sheet configuration, the vertical and horizontal diffusion characteristics are adjusted by the fine shapes on the front and back surfaces of the single optical sheet. In addition, a plurality of diffusion sheets may be used to share the functions. Here, in the example of FIG. 25, the reflection diffusion characteristics due to the front and rear shapes of the optical sheets 207A and 207B may be optimally designed with the number of LEDs, the divergence angle from the LED substrate (optical element) 102, and the optical specifications of the collimator 18 as design parameters so that the surface density of the light flux emitted from the liquid crystal display panel 11 is uniform. In other words, the diffusion characteristics are adjusted by the surface shapes of multiple diffusion sheets instead of the light guide. In the example of FIG. 25, the polarization conversion is performed in the same manner as in the above-mentioned display device example 3. That is, in the example of FIG. 25, the reflective polarizing plate 49 may be configured to have a characteristic of reflecting S-polarized light (transmitting P-polarized light). In that case, the reflective polarizing plate 49 transmits P-polarized light out of the light emitted from the LED, which is the light source, and the transmitted light enters the liquid crystal display panel 11. The reflective polarizing plate 49 reflects S-polarized light out of the light emitted from the LED, which is the light source, and the reflected light passes through the retardation plate 270 shown in FIG. 25. The light that passes through the retardation plate 270 is reflected by the reflection surface 271. The light reflected by the reflecting surface 271 is converted into P-polarized light by passing through the phase difference plate 270 again. The polarized light is transmitted through the reflective polarization plate 49 and enters the liquid crystal display panel 11.

[0118] It should be noted that the λ / 4 plate 270, which is the retardation plate in Fig. 25, does not necessarily have to have a phase difference of λ / 4 with respect to polarized light perpendicularly incident on the λ / 4 plate 270. In the configuration of Fig. 25, any retardation plate may be used in which the phase changes by 90° (λ / 2) when polarized light passes through it twice. The thickness of the retardation plate may be adjusted according to the distribution of the incident angles of the polarized light. It should be noted that, in Fig. 25 as well, the polarization design related to the polarization conversion may be configured in reverse (reversing the S-polarized light and the P-polarized light) based on the above explanation.

[0119] In a typical TV device, the light emitted from the liquid crystal display panel 11 has similar diffusion characteristics in both the horizontal direction of the screen (shown by the X-axis in FIG. 22(a)) and the vertical direction of the screen (shown by the Y-axis in FIG. 22(b)). In contrast, the diffusion characteristics of the light flux emitted from the liquid crystal display panel of this embodiment are, for example, as shown in Example 1 of FIG. 22, 1 / 5 of the conventional 62 degrees when the viewing angle at which the luminance is 50% of that when viewed from the front (angle 0 degrees) is set to 13 degrees. Similarly, the vertical viewing angle is made uneven between the top and bottom, and the reflection angle and the area of ​​the reflection surface of the reflective light guide are optimized to suppress the upper viewing angle to about 1 / 3 of the lower viewing angle. As a result, the amount of image light heading in the monitoring direction is significantly improved compared to conventional LCD TVs, and the luminance is 50 times or more.

[0120] Furthermore, if the viewing angle characteristics shown in Example 2 of FIG. 22 are used, the viewing angle at which the luminance is 50% of that when viewed from the front (angle 0 degrees) is set to 5 degrees, which is 1 / 12 of the conventional 62 degrees. Similarly, the vertical viewing angle is equal from top to bottom, and the reflection angle and the area of ​​the reflection surface of the reflective light guide are optimized to suppress the viewing angle to about 1 / 12 of the conventional viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, and the luminance is more than 100 times higher. As described above, by setting the viewing angle to a narrow angle, the amount of light flux directed toward the monitoring direction can be concentrated, so the efficiency of light use is significantly improved. As a result, even if a conventional LCD display panel for TVs is used, a significant improvement in luminance can be achieved with similar power consumption by controlling the light diffusion characteristics of the light source device, and an image display device compatible with information display systems facing bright outdoors can be obtained.

[0121] When using a large LCD panel, the overall brightness of the screen can be improved by directing the light around the periphery of the screen inwards so that it faces the observer when he or she is facing directly at the center of the screen. Figure 20 shows the convergence angle of the long and short sides of the panel when the observer's distance from the panel L and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen vertically long, the convergence angle can be set to match the short side; for example, when a 22" panel is used vertically and the monitoring distance is 0.8m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed towards the observer.

[0122] Similarly, when monitoring with a 15" panel in portrait orientation and a monitoring distance of 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the observer. As described above, the overall brightness of the screen can be improved by directing the image light from the periphery of the screen towards the observer who is in the optimum position to monitor the center of the screen depending on the size of the LCD panel and whether it is used in portrait or landscape orientation.

[0123] The basic configuration is as shown in Figure 16 above, in which a light beam with a narrow angle directional characteristic is incident on the liquid crystal display panel 11 from a light source device, and the brightness is modulated according to a video signal, so that the video information displayed on the screen of the liquid crystal display panel 11 is reflected by a retroreflective member, and the resulting spatial floating image is displayed outdoors or indoors via a transparent member 100.

[0124] <Lenticular lens> In order to control the diffusion distribution of the image light from the liquid crystal display panel 11, it is possible to control the emission characteristics in one direction by optimizing the lens shape by providing a lenticular lens between the light source device 13 and the liquid crystal display panel 11 or on the surface of the liquid crystal display panel 11. Furthermore, by arranging a microlens array in a matrix, it is possible to control the emission characteristics of the image light flux from the display device 1 in the X-axis and Y-axis directions, and as a result, it is possible to obtain an image display device with desired diffusion characteristics.

[0125] The action of the lenticular lens will be described. By optimizing the lens shape, the lenticular lens can efficiently obtain a floating image in space by transmitting or reflecting the light emitted from the above-mentioned display device 1 through the transparent member 100. That is, by combining two lenticular lenses or providing a sheet that controls the diffusion characteristics by arranging a microlens array in a matrix, the brightness (relative brightness) of the image light from the display device 1 can be controlled in the X-axis and Y-axis directions according to the reflection angle (vertical direction is 0 degrees). In this embodiment, such a lenticular lens makes the vertical brightness characteristic steeper as shown in Figure 22(b) compared to the conventional case, and further changes the balance of the directional characteristics in the up and down directions (positive and negative directions of the Y axis) to increase the brightness (relative brightness) of the light due to reflection and diffusion. This makes it possible to obtain image light with a narrow diffusion angle (high straightness) and containing only specific polarization components, like the image light from a surface-emitting laser image source, thereby suppressing ghost images that were generated by the retroreflective material when using an image display device using conventional technology, and making it possible to efficiently control the floating image in space due to retroreflection to reach the observer's eyes.

[0126] Furthermore, by using the light source device described above, it is possible to realize an image display device that emits light of a specific polarization that emits an image light beam that is nearly parallel to a specific direction by achieving a directional characteristic with a significantly narrower angle in both the X-axis and Y-axis directions compared to the diffusion characteristic of the emitted light from a typical liquid crystal display panel shown in Figure 22 (a) and (b) (labeled as "conventional" in the figure).

[0127] Fig. 21 shows an example of the characteristics of the lenticular lens employed in this embodiment. This example particularly shows the characteristics in the X direction (vertical direction), and characteristic O shows a luminance characteristic that is symmetrical up and down, with the peak of the light emission direction at an angle of about 30 degrees upward from the vertical direction (0 degrees). Characteristics A and B in Fig. 21 show examples of characteristics in which the image light above the peak luminance is further concentrated at about 30 degrees to increase the luminance (relative luminance). For this reason, in characteristics A and B, the luminance (relative luminance) of light is rapidly reduced at angles exceeding 30 degrees, compared to characteristic O.

[0128] That is, according to the optical system including the above-mentioned lenticular lens, when the image light beam from the display device 1 is incident on the retroreflective member 2, the emission angle and viewing angle of the image light aligned to a narrow angle by the light source device 13 can be controlled, and the degree of freedom of installation of the retroreflective sheet (retroreflective member 2) can be greatly improved. As a result, the degree of freedom of the relationship of the image formation position of the space floating image that is reflected or transmitted through the transparent member 100 and formed at a desired position can be greatly improved. As a result, it is possible to efficiently reach the eye of an observer outside or inside the room as light with a narrow diffusion angle (high straightness) and only a specific polarized wave component. As a result, even if the intensity (brightness) of the image light from the image display device is reduced, the observer can accurately recognize the image light and obtain information. In other words, by reducing the output of the image display device, it is possible to realize a space floating image display device with low power consumption.

[0129] <Touch operation assistance function> Next, the assist function of the touch operation for the user will be described. First, the touch operation without the assist function will be described. Note that, here, the case where the user selects and touches one of two buttons (objects) will be described as an example, but the following content can also be suitably applied to, for example, ATMs in banks, ticket vending machines in stations, digital signage, and the like.

[0130] FIG. 26 is a diagram for explaining a display example and touch operation of the space floating image display device 1000. The space floating image 3 shown in FIG. 26 includes a first button BUT1 displayed as "YES" and a second button BUT2 displayed as "NO". The user moves the finger 210 toward the space floating image 3 and touches the first button BUT1 or the second button BUT2 to select "YES" or "NO". In the examples of FIG. 26 and FIG. 27 to FIG. 29, the first button BUT1 and the second button BUT2 are displayed in different colors. Here, the area other than the first button BUT1 and the second button BUT2 in the space floating image 3 may be transparent without displaying an image, but in that case, the range of the effect of the virtual shadow described later will be only the area of ​​the displayed buttons (the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2). Therefore, in the following description, as a more suitable example, it is assumed that in the area of ​​the floating image 3 other than the first button BUT1 and the second button BUT2, an image of a different color or brightness than that of the first button BUT1 and the second button BUT2 is displayed in a wider area including the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2.

[0131] In a general touch panel image display device that is not a space-floating image display device, the button selected by the user is composed of an image button displayed on the touch panel surface. Therefore, the user can recognize the sense of distance between the object (e.g., a button) displayed on the touch panel surface and his / her finger by visually checking the touch panel surface. However, in a space-floating image display device, since the space-floating image 3 floats in the air, it may not be easy for the user to recognize the depth of the space-floating image 3. Therefore, in a touch operation on the space-floating image 3, it may not be easy for the user to recognize the sense of distance between the button displayed on the space-floating image 3 and his / her finger. Also, in a general touch panel image display device that is not a space-floating image display device, the user can easily determine whether or not the button has been touched by the feeling when touched. However, in a touch operation on the space-floating image 3, since there is no feeling when an object (e.g., a button) is touched, the user may not be able to determine whether or not the object has been touched. In consideration of the above situation, in this embodiment, a function to assist the user in touch operation is provided.

[0132] In the following description, processing based on the position of the user's finger will be described, and a specific method for detecting the position of the user's finger will be described later.

[0133] <<Touch operation assistance using virtual shadows (1)>> 27 to 29 are diagrams for explaining an example of a method for assisting a touch operation using a virtual shadow. In the examples of FIG. 27 to FIG. 29, the user touches the first button BUT1 to select "YES". The space-floating image display device 1000 of this embodiment assists the user's touch operation by displaying a virtual shadow on the display image of the space-floating image 3. Here, "displaying a virtual shadow on the display image of the space-floating image 3" refers to an image display process in which the brightness of the image signal is reduced for a part of the area of ​​the shape that resembles a finger for the image to be displayed as the space-floating image 3, so that the image appears as if a shadow is projected on the image. Specifically, the image control unit 1160 or the control unit 1110 may perform this process by calculation. In the display process of the virtual shadow, the brightness of the image signal may be completely set to 0 for a part of the area of ​​the shape that resembles a finger. However, it is preferable to display an image with reduced brightness in the part of the area of ​​the shape that resembles a finger, since the shadow is more naturally recognized as a shadow. In this case, in the process of displaying the virtual shadow, not only the luminance of the video signal for a part of the region of the shape resembling a finger may be reduced, but also the saturation of the video signal may be reduced.

[0134] The floating image 3 exists in the air where there is no physical contact surface, and in a normal environment, the shadow of a finger is always projected. However, according to the display process of the virtual shadow of this embodiment, even in the air where the shadow of a finger is not projected, it is possible to improve the user's perception of the depth of the floating image 3 and the reality of the floating image 3 by making it appear as if a shadow exists in the floating image 3.

[0135] Fig. 27 shows a state at a first time point when the user attempts to touch the first button BUT1 on the display surface 3a of the floating-in-space image 3 with the finger 210, Fig. 28 shows a state at a second time point when the finger 210 is closer to the floating-in-space image 3 than in Fig. 27, and Fig. 29 shows a state at a third time point when the finger 210 touches the first button BUT1 on the display surface 3a of the floating-in-space image 3. In addition, (A) of Fig. 27 to Fig. 29 shows a state when the display surface 3a of the floating-in-space image 3 is viewed from the front (normal direction of the display surface 3a), and (B) of Fig. 27 to Fig. 29 shows a state when the display surface 3a of the floating-in-space image 3 is viewed from the side (direction parallel to the display surface 3a). In addition, in FIG. 27 to FIG. 29, the x direction is the horizontal direction on the display surface 3a of the space floating image 3, the y direction is the direction perpendicular to the x axis in the display surface 3a of the space floating image 3, and the z direction is the normal direction of the display surface 3a of the space floating image 3 (the height direction with respect to the display surface 3a). In addition, in the explanatory diagrams of FIG. 27 to 33, the space floating image 3 is illustrated as having a thickness in the depth direction for ease of explanation, but in reality, if the image display surface of the display device 1 is flat, the space floating image 3 is also flat and has no thickness in the depth direction. In this case, the space floating image 3 and the display surface 3a are on the same plane. In the explanation of this embodiment, the display surface 3a means a surface on which the space floating image 3 can be displayed, and the space floating image 3 means the part where the space floating image is actually displayed.

[0136] 27, 28, and 29, the detection process of the finger 210 is performed using, for example, an image generated by the imaging unit 1180 and a sensing signal of the aerial operation detection sensor 1351. In the detection process of the finger 210, for example, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the floating in space image 3, the height position (z coordinate) of the tip of the finger 210 relative to the display surface 3a, and the like are detected. Here, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the floating in space image 3 is the position coordinate on the display surface 3a of the intersection of the tip of the finger 210 to the display surface 3a of the floating in space image 3. Note that the height position of the tip of the finger 210 relative to the display surface 3a is also depth information that indicates the depth of the finger 210 relative to the display surface 3a. Note that the arrangement of the imaging unit 1180 and the aerial operation detection sensor 1351 that detect the finger 210 and the like will be described in detail later.

[0137] At the first time point shown in Fig. 27, the finger 210 is in a position farthest from the display surface 3a of the floating-in-space image 3 compared to the second time point shown in Fig. 28 and the third time point shown in Fig. 28. The distance (height position) between the tip of the finger 210 and the display surface 3a of the floating-in-space image 3 at this time is set to dz1. In other words, the distance dz1 indicates the height of the finger 210 relative to the display surface 3a of the floating-in-space image 3 in the z direction.

[0138] 27 and distance dz2 shown in Fig. 28 described later are positive on the user side with respect to the display surface 3a of the floating-in-space image 3, and negative on the opposite side of the display surface 3a from the user. That is, if the finger 210 is on the user side with respect to the display surface 3a, the distances dz1 and dz2 are positive values, and if the finger 210 is on the opposite side of the user with respect to the display surface 3a, the distances dz1 and dz2 are negative values.

[0139] In this embodiment, it is assumed that virtual light source 1500 is located on the user side with respect to display surface 3a of space-floating image 3. Here, the setting of the installation direction of virtual light source 1500 may be actually stored as information in non-volatile memory 1108 or memory 1109 of space-floating image display device 1000. Moreover, the setting of the installation direction of virtual light source 1500 may be a parameter that exists only in design. Even if the setting of the installation direction of virtual light source 1500 is a parameter that exists only in design, the installation direction of virtual light source 1500 in design is uniquely determined from the relationship between the position of the user's finger and the display position of the virtual shadow, which will be described later. Here, in the example of FIG. 27 to FIG. 29, virtual light source 1500 is located on the user side with respect to display surface 3a, and is provided on the right side of display surface 3a as seen from the user. Then, virtual shadow 1510 simulating the shadow of finger 210 formed by light irradiated from virtual light source 1500 is displayed on space-floating image 3. 27 to 29, a virtual shadow 1510 is displayed on the left side of the finger 210. This virtual shadow 1510 assists the user in performing a touch operation.

[0140] In the state of FIG. 27(B), the tip of the finger 210 is farthest in the normal direction from the display surface 3a of the floating-in-space image 3, compared to the states of FIG. 28(B) and FIG. 29(B). Therefore, in FIG. 27(A), the tip of the virtual shadow 1510 is formed at a position farthest in the horizontal direction from the first button BUT1 to be touched, compared to the states of FIG. 28(A) and FIG. 29(A). Therefore, in FIG. 27(A), the horizontal distance between the tip of the finger 210 and the tip of the virtual shadow 1510 when the display surface 3a of the floating-in-space image 3 is viewed from the front is the largest, compared to the states of FIG. 28(A) and FIG. 29(A). In FIG. 27(A), the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating-in-space image 3 is dx1.

[0141] In FIG. 28(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 28(B), distance dz2 in the normal direction between tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz1. At this time, in FIG. 28(A), virtual shadow 1510 is displayed at a position where distance between tip of finger 210 and tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx2, which is smaller than dx1. That is, in the example of FIG. 28, virtual light source 1500 is provided on the user side with respect to display surface 3a and on the right side of display surface 3a as seen from the user, so that the horizontal distance between tip of finger 210 and tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front changes in conjunction with the distance between tip of finger 210 and display surface 3a of space-floating image 3 in the normal direction.

[0142] Then, when the tip of the finger 210 and the tip of the virtual shadow 1510 come into contact, the distance in the normal direction between the tip of the finger 210 and the display surface 3a of the floating-in-space image 3 becomes 0, as shown in FIG. 29. At this time, the virtual shadow 1510 is displayed so that the distance between the finger 210 and the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating-in-space image 3 becomes zero. This allows the user to recognize that the finger 210 has touched the display surface 3a of the floating-in-space image 3. At this time, if the tip of the finger 210 is touching the area of ​​the first button BUT1, the user can recognize that the first button BUT1 has been touched. 29, since virtual light source 1500 is provided on the user side of display surface 3a and on the right side of display surface 3a as seen from the user, the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of floating in space image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of floating in space image 3. In other words, the display position of the tip of virtual shadow 1510 is a position specified by the positional relationship between the position of virtual light source 1500 and the position of the tip of user's finger 210, and changes in conjunction with the change in the position of the tip of user's finger 210.

[0143] According to the configuration and processing of "assistance of touch operation using virtual shadow (1)" described above, when performing a touch operation, the user can more appropriately recognize the normal direction distance (depth) between the finger 210 and the display surface 3a of the floating-in-space image 3 from the horizontal positional relationship between the finger 210 and the virtual shadow 1510 on the display surface 3a of the floating-in-space image 3. Also, when the finger 210 touches an object (e.g., a button) that is the floating-in-space image 3, the user can recognize that the object has been touched. This makes it possible to provide a more appropriate floating-in-space image display device.

[0144] <<Touch operation assistance using virtual shadows (2)>> Next, as another example of the method of assisting a touch operation using a virtual shadow, a case where virtual light source 1500 is provided on the left side of display surface 3a as seen from the user will be described. FIGS. 30 to 32 are diagrams for explaining another example of the method of assisting a touch operation using a virtual shadow. FIG. 30 corresponds to FIG. 27, and shows a state at a first time point when a user attempts to perform a touch operation on first button BUT1 on display surface 3a of floating-in-space image 3 with finger 210. FIG. 31 corresponds to FIG. 28, and shows a state at a second time point when finger 210 is closer to floating-in-space image 3 than in FIG. 30. FIG. 32 corresponds to FIG. 29, and shows a state when finger 210 touches floating-in-space image 3. Note that, for convenience of explanation, (B) of FIG. 30 to FIG. 32 shows a view from the opposite direction to (B) of FIG. 27 to FIG. 29.

[0145] 30 to 32, virtual light source 1500 is provided on the user side of display surface 3a, on the left side of display surface 3a as seen from the user. Then, a virtual shadow 1510 that imitates the shadow of finger 210 formed by light irradiated from virtual light source 1500 is displayed in space floating image 3. In FIGS. 30 to 32, virtual shadow 1510 is displayed to the right of finger 210. This virtual shadow 1510 assists the user in performing a touch operation.

[0146] In the state of Fig. 30(B), compared to the states of Fig. 31(B) and Fig. 32(B), the tip of finger 210 is the furthest away in the normal direction from display surface 3a of space floating image 3. In Fig. 30(B), the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 at this time is dz10. Also, in Fig. 30(A), the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space floating image 3 at this time is dx10.

[0147] In FIG. 31(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 31(B), distance dz20 in the normal direction between tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz10. At this time, in FIG. 31(A), virtual shadow 1510 is displayed at a position where distance between tip of finger 210 and tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx20, which is smaller than dx10. That is, in the example of FIG. 31, virtual light source 1500 is provided on the user side with respect to display surface 3a and on the left side of display surface 3a as seen from the user, so that the horizontal distance between tip of finger 210 and tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front changes in conjunction with the distance between tip of finger 210 and display surface 3a of space-floating image 3 in the normal direction.

[0148] Then, when the tip of the finger 210 and the tip of the virtual shadow 1510 come into contact, the distance in the normal direction between the tip of the finger 210 and the display surface 3a of the floating-in-space image 3 becomes 0, as shown in FIG. 32. At this time, the virtual shadow 1510 is displayed so that the distance between the finger 210 and the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating-in-space image 3 becomes zero. This allows the user to recognize that the finger 210 has touched the display surface 3a of the floating-in-space image 3. At this time, if the tip of the finger 210 is touching the area of ​​the first button BUT1, the user can recognize that the first button BUT1 has been touched. That is, in the example of FIG. 32, virtual light source 1500 is provided on the user's side of display surface 3a, to the left of display surface 3a as seen from the user, so that the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of floating-in-space image 3 is viewed from the front changes in conjunction with the normal distance between the tip of finger 210 and display surface 3a of floating-in-space image 3.

[0149] The above-described configuration and processing of "assistance of touch operation using virtual shadow (2)" also provides the same effects as those of the configurations of FIGS.

[0150] Here, when implementing the above-mentioned process of "assisting touch operation using a virtual shadow (1)" and / or the process of "assisting touch operation using a virtual shadow (2)" in the space floating image display device 1000, there are several possible implementation examples.

[0151] The first implementation example is a method of implementing only "assistance of touch operation using virtual shadow (1)" in the floating-in-space image display device 1000. In this case, since the virtual light source 1500 is provided on the user side of the display surface 3a and on the right side of the display surface 3a as seen from the user, the virtual shadow 1510 is displayed on the left side of the tip of the user's finger 210 as seen from the user. Therefore, if the user's finger 210 is a finger of the right hand, the visibility of the display of the virtual shadow 1510 is not blocked by the user's right hand or right arm, which is preferable. Therefore, considering the statistical tendency that there are more right-handed users, even if only "assistance of touch operation using virtual shadow (1)" is implemented in the floating-in-space image display device 1000, the probability that the display of the virtual shadow 1510 can be well viewed is sufficiently high and preferable.

[0152] As a first implementation example, both the process of "assisting touch operation using a virtual shadow (1)" and the process of "assisting touch operation using a virtual shadow (2)" may be implemented, and the process to be performed may be switched depending on whether the user performs the touch operation with the right hand or the left hand. In this case, it is possible to further increase the probability that the display of the virtual shadow 1510 can be clearly seen, improving the convenience for the user.

[0153] Specifically, when the user performs a touch operation with the right hand, the configurations of FIGS. 27 to 29 are used to display the virtual shadow 1510 on the left side of the finger 210. In this case, the visibility of the display of the virtual shadow 1510 is not blocked by the user's right hand or right arm, which is preferable. On the other hand, when the user performs a touch operation with the left hand, the configurations of FIGS. 30 to 32 are used to display the virtual shadow 1510 on the right side of the finger 210. In this case, the visibility of the display of the virtual shadow 1510 is not blocked by the user's left hand or left arm, which is preferable. As a result, whether the user performs a touch operation with the right hand or the left hand, the virtual shadow 1510 is displayed at a position that is easy for the user to view, improving user convenience.

[0154] Here, the determination of whether the touch operation is performed with the right hand or the left hand may be performed based on, for example, a captured image generated by the imaging unit 1180. For example, the control unit 1110 performs image processing on the captured image and detects the user's face, arms, hands, and fingers from the captured image. Then, the imaging unit 1180 may estimate the user's posture or movement from the arrangement of the detected (face, arms, hands, fingers) and determine whether the user is performing the touch operation with the right hand or the left hand. In this determination, if the vicinity of the center in the left-right direction of the user's body can be determined from other parts, it is not necessarily necessary to capture an image of the face. Also, the above determination may be performed only from the arrangement of the arms. The above determination may be performed only from the arrangement of the hands. The above determination may be performed from a combination of the arrangement of the arms and the arrangement of the hands. Also, when making these determinations, the determination may be performed in combination with the arrangement of the face.

[0155] 27 to 29 and 30 to 32 show virtual shadow 1510 extending at an angle corresponding to the extending direction of actual finger 210. The extending direction of actual finger 210 may be calculated by capturing an image of the finger using any of the imaging units already described. Here, virtual shadow 1510 may be displayed with the extending direction fixed at a predetermined angle without reflecting the angle corresponding to the extending direction of finger 210. This reduces the load on image control unit 1160 or control unit 1110 that controls the display of virtual shadow 1510.

[0156] For example, if the finger 210 is a finger of the right hand, it is natural for the user to extend his / her arm from the right side in front of the display surface 3a of the floating in space image 3, and try to touch the display surface 3a of the floating in space image 3 with the finger 210 pointing to the upper left toward the display surface 3a of the floating in space image 3. Therefore, if the finger 210 is a finger of the right hand, if the finger shadow indicated by the virtual shadow 1510 is configured to be displayed in a predetermined direction indicating the upper right direction toward the display surface 3a of the floating in space image 3, a natural display will be obtained without reflecting the angle corresponding to the finger 210.

[0157] Also, for example, if the finger 210 is a finger of the left hand, it is natural for the user to extend his / her arm from the left side in front of the display surface 3a of the floating in space image 3, and try to touch the display surface 3a of the floating in space image 3 with the finger 210 pointing to the upper right toward the display surface 3a of the floating in space image 3. Therefore, if the finger 210 is a finger of the left hand, if the finger shadow indicated by the virtual shadow 1510 is configured to be displayed in a predetermined direction indicating the upper left direction toward the display surface 3a of the floating in space image 3, a natural display will be obtained without reflecting the angle corresponding to the finger 210.

[0158] In addition, when the user's finger 210 is on the opposite side of the display surface 3a of the space floating image 3 from the user, a display may be made so that the user can recognize that the finger 210 is behind the space floating image 3 and cannot be touched. For example, a message may be displayed on the space floating image 3 to inform the user that the finger 210 is behind the space floating image 3 and cannot be touched. Alternatively, for example, the virtual shadow 1510 may be displayed in a color different from the normal color, such as red. This makes it possible to more appropriately prompt the user to return the finger 210 to the appropriate position.

[0159] <<An example of virtual light source settings>> Here, a method for setting virtual light source 1500 will be described. Fig. 33 is a diagram for explaining a method for setting a virtual light source. Fig. 33 shows a situation in which a user performs a touch operation with the left hand, but the contents described below are also preferably applicable to a case in which a user performs a touch operation with the right hand.

[0160] 33 shows normal line L1 of display surface 3a extending from point C at the center of display surface 3a of space floating image 3 toward the user, line L2 connecting virtual light source 1500 and point C where normal line L1 intersects with display surface 3a, and virtual light source installation angle α defined by the angle between normal line L1 and line L2. For ease of explanation, FIG. 33 shows the moment when the tip of user's finger 210 is on line L2.

[0161] Here, in order to simplify the explanation, in FIGS. 27 to 33, virtual light source 1500 is illustrated as being placed at a position not far from display surface 3a of space floating image 3 and user's finger 210. Although virtual light source 1500 may be set at such a position, the most preferable setting example is as follows. That is, it is desirable to set the distance between virtual light source 1500 and point C at the center of display surface 3a of space floating image 3 to infinity. The reason is as follows. If an object plane having a contact surface is in the same coordinate system as display surface 3a of space floating image 3 in FIGS. 27 to 32, and the light source is the sun rather than a virtual light source, the distance of the sun can be approximated as almost infinity, so that the horizontal direction (x direction) position of the tip of the shadow of the user's finger on the real object plane changes linearly with respect to the change in the distance (z direction) between the tip of the user's finger and the object plane. Therefore, even in the setting of virtual light source 1500 shown in FIGS. 27 to 33 of this embodiment, if the distance between virtual light source 1500 and center point C of display surface 3a of floating-in-space image 3 is set to infinity, and the horizontal (x-direction) position of the tip of virtual shadow 1510 in floating-in-space image 3 changes linearly with the change in the distance (z-direction) between the tip of user's finger 210 and display surface 3a of floating-in-space image 3, it is possible to express a virtual shadow that can be perceived more naturally by the user.

[0162] If the virtual light source 1500 is set to be disposed at a position not far from the display surface 3a of the space floating image 3 or the user's finger 210, the horizontal (x-direction) position of the tip of the virtual shadow 1510 in the space floating image 3 changes nonlinearly with respect to the change in the distance (z-direction) between the tip of the user's finger 210 and the display surface 3a of the space floating image 3, and the calculation for calculating the horizontal (x-direction) position of the tip of the virtual shadow 1510 becomes somewhat complicated. On the other hand, if the distance between the virtual light source 1500 and the center point C of the display surface 3a of the space floating image 3 is set to infinity, the horizontal (x-direction) position of the tip of the virtual shadow 1510 in the space floating image 3 changes linearly with respect to the change in the distance (z-direction) between the tip of the user's finger 210 and the display surface 3a of the space floating image 3, so that the calculation for calculating the horizontal (x-direction) position of the tip of the virtual shadow 1510 can be simplified.

[0163] When the virtual light source installation angle α is small, the angle between the line connecting virtual light source 1500 and finger 210 and normal line L1 cannot be increased as viewed from the user, so the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction (x direction) of display surface 3a of floating-in-space image 3 becomes short. This makes it difficult for the user to visually recognize the change in the position of virtual shadow 1510 when the tip of finger 210 performs a touch operation, and there is a risk that the effect of the user's depth perception in the touch operation will decrease. To avoid this, it is desirable to install virtual light source 1500 so that the angle between line L2 connecting virtual light source 1500 and point C and normal line L1 is, for example, 20° or more.

[0164] On the other hand, when the angle between the line connecting virtual light source 1500 and finger 210 and normal line L1 approaches 90°, the distance between the tip of finger 210 and the tip of virtual shadow 1510 becomes very long. This increases the probability that the display position of virtual shadow 1510 will be outside the range of floating-in-space image 3, and increases the probability that virtual shadow 1510 cannot be displayed in floating-in-space image 3. For this reason, it is desirable that installation angle α of virtual light source 1500 be 70° or less so that the angle between line L2 connecting virtual light source 1500 and point C and normal line L1 does not approach too close to 90°, for example.

[0165] In other words, it is desirable to place virtual light source 1500 in a position that is not too close to a plane that includes a normal line passing through finger 210 and is not too close to a plane that includes display surface 3a of space floating image 3.

[0166] The space-floating image display device 1000 of this embodiment can display a virtual shadow as described above. This is a video processing that produces a physically more natural effect than the case where a predetermined mark is superimposed on the image to assist the user's touch operation. Therefore, the touch operation assistance technology by displaying the above-mentioned virtual shadow of the space-floating image display device 1000 of this embodiment can provide a situation in which the user can recognize the depth of the touch operation more naturally.

[0167] <<How to detect finger position>> Next, a method for detecting the position of finger 210 will be described. A configuration for detecting the position of finger 210 of user 230 will be specifically described below.

[0168] <<<How to detect finger position (1)>>> Fig. 34 is a configuration diagram showing an example of a method for detecting the position of a finger. In the example shown in Fig. 34, the position of a finger 210 is detected using one imaging unit 1180 and one air operation sensor 1351. Note that each imaging unit in the embodiment of the present invention has an imaging sensor.

[0169] The first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the space floating image 3. The first imaging unit 1180a may be installed in the housing 1190 as shown in FIG. 34, or may be installed at a location away from the housing 1190.

[0170] The imaging area of ​​the first imaging unit 1180a is set to include, for example, the display area of ​​the space floating image 3, the fingers, hands, arms, face, etc. of the user 230. The first imaging unit 1180a images the user 230 performing a touch operation on the space floating image 3, and generates a first captured image. Note that even if the display area of ​​the space floating image 3 is captured by the first imaging unit 1180a, the image is captured from the opposite side of the traveling direction of the directional light flux of the space floating image 3, so the space floating image 3 itself cannot be visually recognized as an image. Here, in the example of the finger position detection method (1), the first imaging unit 1180a is not just an imaging unit, but has a built-in depth sensor in addition to an imaging sensor. The configuration and processing of the depth sensor may use existing technology. The depth sensor of the first imaging unit 1180a detects the depth of each part (for example, the user's fingers, hands, arms, face, etc.) in the captured image of the first imaging unit 1180a, and generates depth information.

[0171] The aerial operation sensor 1351 is installed at a position where it can sense the display surface 3a of the floating-in-space image 3 as the sensing target surface. In Fig. 34, the aerial operation sensor 1351 is installed below the display surface 3a of the floating-in-space image 3, but it may be installed to the side or above the display surface 3a. The aerial operation sensor 1351 may be installed in the housing 1190 as shown in Fig. 34, or may be installed at a location away from the housing 1190.

[0172] 34 is a sensor that detects the position where the finger 210 contacts or overlaps with the display surface 3a of the floating-in-space image 3. That is, when the tip of the finger 210 approaches the display surface 3a of the floating-in-space image 3 from the user's side of the display surface 3a of the floating-in-space image 3, the floating-in-space operation detection sensor 1351 can detect the contact of the finger 210 with the display surface 3a of the floating-in-space image 3.

[0173] For example, the control unit 1110 shown in Fig. 3C reads out a program for performing image processing and a program for displaying a virtual shadow 1510 from the non-volatile memory 1108. The control unit 1110 performs a first image processing on a first captured image generated by the imaging sensor of the first imaging unit 1180a, detects the finger 210, and calculates the position (x coordinate, y coordinate) of the finger 210. The control unit 1110 calculates the position (z coordinate) of the tip of the finger 210 with respect to the floating in space image 3 based on the first captured image generated by the imaging sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a.

[0174] 34, the image sensor and depth sensor of the first imaging unit 1180a, the aerial operation sensor 1351, the aerial operation detection unit 1350, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects a touch on an object in the floating in space image 3. This calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210. In addition, the touch detection result is calculated by combining the detection result of the aerial operation detection unit 1350 or the detection result of the aerial operation detection unit 1350 and information generated by the first imaging unit 1180a.

[0175] Then, control unit 1110 calculates the position (display position) at which virtual shadow 1510 is to be displayed, based on the position (x coordinate, y coordinate, z coordinate) of finger 210 and the position of virtual light source 1500, and generates video data of virtual shadow 1510 based on the calculated display position.

[0176] The control unit 1110 may calculate the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated. The control unit 1110 may not calculate the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated, but may store display position map data in which the display positions of the virtual shadow 1510 corresponding to each of the multiple positions of the finger 210 are calculated in advance in the non-volatile memory 1108, and generate video data of the virtual shadow 1150 based on the display position map data stored in the non-volatile memory 1108 after calculating the position of the finger 210. The control unit 1110 may also calculate the tip of the finger 210 and the extension direction of the finger 210 in the first image processing, calculate the display position of the tip of the finger 210 and the extension direction of the virtual shadow 1510 corresponding to the extension direction, and generate video data of the virtual shadow 1510 adjusted to a display angle corresponding to the actual orientation of the finger 210 based on these.

[0177] The control unit 1110 outputs the generated image data of the virtual shadow 1510 to the image control unit 1160. The image control unit 1160 generates image data (superimposed image data) in which the image data of the virtual shadow 1510 is superimposed on other image data such as an object, and outputs the superimposed image data including the image data of the virtual shadow 1510 to the image display unit 1102.

[0178] The image display unit 1102 displays an image based on the superimposed image data including image data of the virtual shadow 1510, whereby a space floating image 3 in which the virtual shadow 1510 and an object or the like are superimposed is displayed.

[0179] The detection of a touch on an object is performed, for example, as follows. The aerial operation detection unit 1350 and the aerial operation detection sensor 1351 are configured as described in FIG. 3, and when the finger 210 touches or overlaps a plane including the display surface 3a of the space floating image 3, they detect the position and output touch position information indicating the position where the finger 210 touches or overlaps the display surface 3a to the control unit 1110. Then, when the touch position information is input, the control unit 1110 judges whether or not the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing is included in the display range of each object displayed on the display surface 3a of the space floating image 3. Then, when the position of the finger 210 is included in the display range of any one of the objects, the control unit 1110 judges that the object has been touched.

[0180] According to the detection method described above, it is possible to detect the position of the finger 210 and the touch operation with a simple configuration that combines one imaging unit 1180 (first imaging unit 1180a) having an image sensor and a depth sensor with one aerial operation detection sensor 1351.

[0181] As a modified example of the finger position detection method (1), the control unit 1110 may detect a touch operation by the finger 210 solely based on the first captured image generated by the imaging sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, without using the detection results by the aerial operation detection unit 1350 and the aerial operation detection sensor 1351. For example, during normal operation, the mode is configured to detect a touch operation by finger 210 by combining the captured image of the imaging sensor of first imaging unit 1180a, the detection result of the depth sensor, and the detection result of the aerial operation detection sensor 1351, and if there is any malfunction in the operation of the aerial operation detection sensor 1351 or the aerial operation detection unit 1350, the control unit 1110 may switch to a mode in which it detects a touch operation by finger 210 only based on the first captured image generated by the imaging sensor of first imaging unit 1180a and the depth information generated by the depth sensor of first imaging unit 1180a, without using the detection results of the aerial operation detection unit 1350 and the aerial operation detection sensor 1351.

[0182] <<How to detect finger position (2)>> Fig. 35 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in Fig. 35, the position of the finger 210 is detected using two imaging units. The second imaging unit 1180b (1180) and the third imaging unit 1180c (1180) are both provided on the opposite side of the user 230 with respect to the floating-in-space image 3.

[0183] The second imaging unit 1180b is installed, for example, on the right side as seen from the user 230. The imaging area of ​​the second imaging unit 1180b is set to include, for example, the floating-in-space image 3, and the fingers, hands, arms, face, etc. of the user 230. The second imaging unit 1180b images the user 230 performing a touch operation on the floating-in-space image 3 from the right side of the user 230, and generates a second captured image.

[0184] The third imaging unit 1180c is installed, for example, on the left side as seen from the user 230. The imaging area of ​​the third imaging unit 1180c is set to include, for example, the floating in space image 3, the fingers, hands, arms, face, etc. of the user 230. The third imaging unit 1180c images the user 230 performing a touch operation on the floating in space image 3 from the left side of the user 230, and generates a third captured image. Thus, in the example of FIG. 35, the second imaging unit 1180b and the third imaging unit 1180c configure a so-called stereo camera.

[0185] 35, the second imaging unit 1180b and the third imaging unit 1180c may be installed in the housing 1190, or may be installed at a location away from the housing 1190. Moreover, one imaging unit may be installed in the housing 1190, and the other imaging unit may be installed at a location away from the housing 1190.

[0186] The control unit 1110 performs second image processing on the second captured image and third image processing on the third captured image, respectively. Then, the control unit 1110 calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210 based on the result of the second image processing (second image processing result) and the result of the third image processing (third image processing result).

[0187] 35, the second imaging section 1180b, the third imaging section 1180c, and the control section 1110 constitute a touch detection section that detects the position of the user's finger and detects a touch on an object in the space floating image 3. Then, the position (x coordinate, y coordinate, z coordinate) of the finger 210 is calculated as a position detection result or a touch detection result.

[0188] 35, a virtual shadow 1510 is generated based on the position of finger 210 calculated based on the second image processing result and the third image processing result. Also, based on the position of finger 210 calculated based on the second image processing result and the third image processing result, a determination is made as to whether or not an object has been touched.

[0189] According to this configuration, it is not necessary to employ an imaging section having a depth sensor. Also, according to this configuration, by using the second imaging section 1180b and the third imaging section 1180c as a stereo camera, it is possible to improve the detection accuracy of the position of the finger 210. In particular, compared to the example of FIG. 34, it is possible to improve the detection accuracy of the x coordinate and the y coordinate. Therefore, it is possible to more accurately determine whether or not an object has been touched.

[0190] As a modified example of the finger position detection method (2), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the second captured image by the second imaging unit 1180b and the third captured image by the third imaging unit 1180c as described above, and the display of the virtual shadow 1510 is controlled based on this, and the presence or absence of a touch on the object of the floating in space image 3 may be detected by the mid-air operation detection unit 1350 or the control unit 1110 based on the detection result by the mid-air operation detection sensor 1351. According to this modified example, since the mid-air operation sensor 1351 that senses the display surface 3a of the floating in space image 3 as the sensing target surface is used, the touch of the user's finger 210 on the display surface 3a of the floating in space image 3 can be detected with higher accuracy than the detection accuracy in the depth direction by the stereo camera of the second imaging unit 1180b and the third imaging unit 1180c.

[0191] <<<Method of detecting finger position (3)>>> FIG. 36 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in FIG. 36, the position of the finger 210 is detected using two imaging units. The example in FIG. 36 is different from the example in FIG. 35 in that the fourth imaging unit 1180d (1180), which is one of the imaging units, is arranged at a position where it images the display surface 3a of the space floating image 3 from the side. Also, as in the example in FIG. 34, the first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the space floating image 3. In the example in FIG. 36, the first imaging unit 1180a (1180) only needs to be able to capture an image and does not need to be equipped with a depth sensor.

[0192] Therefore, the fourth imaging unit 1180d is installed around the display surface 3a of the space floating image 3. In Fig. 36, the fourth imaging unit 1180d is installed below the side of the display surface 3a of the space floating image 3, but it may be installed to the side or above the display surface 3a. The fourth imaging unit 1180d may be installed in the housing 1190 as shown in Fig. 36, or may be installed at a location away from the housing 1190.

[0193] The imaging area of ​​the fourth imaging unit 1180d is set to include, for example, the floating-in-space image 3 and the fingers, hands, arms, face, etc. of the user 230. The fourth imaging unit 1180d images the user 230 performing a touch operation on the floating-in-space image 3 from the periphery of the display surface 3a of the floating-in-space image 3, and generates a fourth captured image.

[0194] The control unit 1110 performs a fourth image processing on the fourth captured image, and calculates the distance (z coordinate) between the display surface 3a of the space floating image 3 and the tip of the finger 210. Then, the control unit 1110 performs processing related to the virtual shadow 1510 and determines whether or not an object has been touched, based on the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing on the first captured image by the first imaging unit 1180a described above and the position (z coordinate) of the finger 210 calculated by the fourth image processing.

[0195] 36, a touch detection unit that detects the position of a user's finger and detects a touch on an object is configured by the first imaging unit 1180a, the fourth imaging unit 1180d, and the control unit 1110. Then, the position (x coordinate, y coordinate, z coordinate) of the finger 210 is calculated as a position detection result or a touch detection result.

[0196] According to this configuration, it is possible to improve the detection accuracy of the distance between the display surface 3a of the floating in space image 3 and the tip of the finger 210, i.e., the depth of the finger 210 relative to the display surface 3a of the floating in space image 3, compared to the example of the stereo camera configuration in FIG.

[0197] As a modified example of the finger position detection method (3), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the first captured image by the first imaging unit 1180a and the fourth captured image by the fourth imaging unit 1180d as described above, and the display of the virtual shadow 1510 is controlled based on this, and the presence or absence of a touch on the object of the floating in space image 3 may be detected by the mid-air operation detection unit 1350 or the control unit 1110 based on the detection result by the mid-air operation detection sensor 1351. According to this modified example, since the mid-air operation sensor 1351 that senses the display surface 3a of the floating in space image 3 as the sensing target surface is used, the touch of the user's finger 210 on the display surface 3a of the floating in space image 3 can be detected with higher accuracy than the detection accuracy based on the fourth captured image by the fourth imaging unit 1180d.

[0198] <<How to display input content and assist touch operation>> An example of assisting the user's touch operation by another method will be described. For example, it is possible to assist the touch operation by displaying the input contents. Fig. 37 is a diagram for explaining a method of assisting the touch operation by displaying the input contents. Fig. 37 shows a case where numbers are input by touch operation.

[0199] The spatial floating image 3 in Figure 37 includes a key input UI (user interface) display area 1600 containing multiple objects including, for example, multiple objects for inputting numbers, etc., an object 1601 for erasing the input content, and an object 1603 for deciding the input content, and an input content display area 1610 for displaying the input content.

[0200] In the input content display area 1610, the contents (e.g., numbers) input by touch operation are displayed in sequence on the floating-in-space image 3 from the left end to the right. The user can check the contents input by touch operation while looking at the input content display area 1610. Then, when the user has input all the desired numbers, he / she touches the object 1603. This registers the input contents displayed in the input content display area 1610. The touch operation on the floating-in-space image 3 is different from physical contact on the surface of the display device, and the user cannot feel the touch. Therefore, by separately displaying the input contents in the input content display area 1610, the user can proceed with the operation while checking whether or not his / her touch operation has been performed effectively, which is preferable.

[0201] On the other hand, if the user inputs content different from what he or she intended, for example by touching the wrong object, the user can erase the last input content (here, "9") by touching object 1601. Then, the user continues to perform touch operations on objects for inputting numbers and the like. When the user has input all the desired numbers, he or she touches object 1603.

[0202] In this way, by displaying the input contents in the input contents display area 1610, the user can confirm the input contents, and convenience can be improved. Also, if the user touches the wrong object, the input contents can be corrected, and convenience can be improved.

[0203] <<How to highlight input content to assist touch operation>> Next, it is also possible to assist the touch operation by highlighting the input contents. Fig. 38 is a diagram for explaining a method of assisting the touch operation by highlighting the input contents.

[0204] An example in which a number entered by a touch operation is highlighted is shown in Fig. 38. In accordance with Fig. 38, when an object corresponding to the number "6" is touched, the touched object is erased, and the input number "6" is displayed in the area where this object was displayed.

[0205] In this way, the number corresponding to the touched object is displayed instead of the object, so that the user can recognize that the object has been touched, and convenience can be improved. The number corresponding to the touched object may be called a replacement object that replaces the touched object.

[0206] As another method for highlighting the input contents, for example, the object touched by the user may be brightly lit or the object touched by the user may be blinking. Although not shown here, by recognizing the distance between the finger 210 and the display surface 3a described in the embodiment of Figs. 27-28, the object to be touched may become brighter than the surrounding objects as the finger approaches the display surface, and finally, when the finger is moved to the display surface, the degree of highlighting may reach a maximum, or the object may be lit even brighter or blink. Even with such a configuration, it is possible to make the user recognize that the object has been touched, and convenience can be improved.

[0207] <<Method of assisting touch operation with vibration (1)>> Next, a method of assisting a touch operation by vibration will be described. Fig. 39 is a diagram for explaining an example of a method of assisting a touch operation by vibration. Fig. 39 shows a case where a touch operation is performed using a touch pen (touch input device) 1700 instead of a finger 210. The touch pen 1700 is equipped with a communication unit that transmits and receives various information such as signals and data between a device such as a space floating image display device, and a vibration mechanism that vibrates based on an input signal.

[0208] Assume that the user operates the touch pen 1700 and touches an object displayed in the key input UI display area 1600 of the floating-in-space image 3 with the touch pen 1700. At this time, for example, the control unit 1100 transmits a touch detection signal indicating that a touch on the object has been detected from the communication unit 1132. When the touch pen 1700 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the touch pen 1700 to vibrate. The vibration of the touch pen 1700 is then transmitted to the user, and the user recognizes that the object has been touched. In this way, the vibration of the touch pen 1700 assists the touch operation.

[0209] According to this configuration, it is possible to make the user aware that he or she has touched the object by means of vibration.

[0210] Here, the case where the touch detection signal transmitted from the space floating image device is received by the touch pen 1700 has been described, but other configurations are also possible. For example, when a touch on an object is detected, the space floating image display device notifies the host device that a touch on the object has been detected. Then, the host device transmits a touch detection signal to the touch pen 1700.

[0211] Alternatively, the space floating image display device and the higher-level device may transmit the touch detection signal via a network. In this manner, the touch pen 1700 may indirectly receive the touch detection signal from the space floating image display device.

[0212] <<Method of assisting touch operation with vibration (2)>> Next, another method of assisting a touch operation by vibration will be described. Here, the user is made aware that he or she has touched an object by vibrating the terminal owned by the user. FIG. 40 is a diagram for explaining another example of the method of assisting a touch operation by vibration. In the example of FIG. 40, a user 230 wearing a wristwatch-type wearable terminal 1800 performs a touch operation.

[0213] The wearable terminal 1800 is equipped with a communication unit that transmits and receives various information such as signals and data to and from a device such as a space floating image display device, and a vibration mechanism that vibrates based on an input signal.

[0214] Assume that the user performs a touch operation with the finger 210 and touches an object displayed in the key input UI display area 1600 of the floating-in-space image 3. At this time, for example, the control unit 1100 transmits a touch detection signal indicating that a touch on the object has been detected from the communication unit 1132. When the wearable terminal 1800 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the wearable terminal 1800 to vibrate. Then, the vibration of the wearable terminal 1800 is transmitted to the user, and the user recognizes that the object has been touched. In this way, the vibration of the wearable terminal 1800 assists the touch operation. Here, a wristwatch-type wearable terminal has been described as an example, but a smartphone worn by the user may also be used.

[0215] The wearable terminal 1800 may receive a touch detection signal from a higher-level device, similar to the above-described touch pen 1700. The wearable terminal 1800 may receive a touch detection signal via a network. In addition to the wearable terminal 1800, it is also possible to assist the touch operation using an information processing terminal such as a smartphone owned by the user.

[0216] According to this configuration, it is possible to make a user aware that he or she has touched an object via various terminals such as the wearable terminal 1800 owned by the user.

[0217] <<Method of assisting touch operation with vibration (3)>> Next, another method of assisting a touch operation by vibration will be described. FIG. 41 is a diagram for explaining another example of a method of assisting a touch operation by vibration. In the example of FIG. 41, user 230 performs a touch operation while standing on diaphragm 1900. Diaphragm 1900 is placed at a predetermined position where user 230 performs a touch operation. In an actual usage form, diaphragm 1900 is placed, for example, under a mat (not shown), and user 230 stands on diaphragm 1900 via the mat.

[0218] As shown in FIG. 41, the diaphragm 1900 is connected to, for example, the communication unit 1132 of the space floating image display device 1000 via a cable 1910. When a touch on an object is detected, for example, the control unit 1110 supplies an AC voltage to the diaphragm 1900 for a predetermined time via the communication unit 1132. The diaphragm 1900 vibrates while the AC voltage is being supplied. That is, the AC voltage is a control signal for vibrating the diaphragm 1900, which is output from the communication unit 1132. The vibration generated by the diaphragm 1900 is transmitted from the feet to the user 230, and the user 230 can recognize that the object has been touched. In this way, the vibration of the diaphragm 1900 assists the touch operation.

[0219] The frequency of the AC voltage is set to a value within a range in which the user 230 can feel the vibration. The frequency of vibration that a person can feel is within a range of approximately 0.1 Hz to 500 Hz. For this reason, it is desirable to set the frequency of the AC voltage within this range.

[0220] Also, it is desirable to change the frequency of the AC voltage as appropriate depending on the characteristics of the diaphragm 1900. For example, when the diaphragm 1900 vibrates in the vertical direction, it is said that people have the highest sensitivity to vibrations of about 410 Hz. Also, when the diaphragm 1900 vibrates in the horizontal direction, it is said that people have the highest sensitivity to vibrations of about 12 Hz. Furthermore, at frequencies of 34 Hz ​​or higher, it is said that people have a higher sensitivity to vibrations in the vertical direction than in the horizontal direction.

[0221] Therefore, when the diaphragm 1900 vibrates in the vertical direction, it is desirable to set the frequency of the AC voltage to a value within a range including, for example, 410 Hz. When the diaphragm 1900 vibrates in the horizontal direction, it is desirable to set the frequency of the AC voltage to a value within a range including, for example, 12 Hz. Note that the peak voltage and frequency of the AC voltage may be appropriately adjusted depending on the performance of the diaphragm 1900.

[0222] According to this configuration, it is possible for the user 230 to recognize that an object has been touched by vibrations from the feet. In addition, in the case of this configuration, it is also possible to set the display of the floating image 3 not to change when an object is touched, so that even if a third party peeks at the touch operation, the possibility of the input contents being known is reduced, making it possible to further improve security.

[0223] <<Modification 1 of Object Display>> Another example of object display in the space floating image 3 by the space floating image display device 1000 will be described. The space floating image display device 1000 is configured to display the space floating image 3, which is an optical image of a rectangular image displayed by the display device 1. The rectangular image displayed by the display device 1 and the space floating image 3 are in a corresponding relationship. Therefore, when an image having brightness is displayed over the entire display range of the display device 1, the space floating image 3 displays an image having brightness over the entire display range. In this case, although the sense of floating in the air can be obtained for the rectangular space floating image 3 as a whole, there is a problem that it is difficult to obtain the sense of floating in the air for each object itself displayed in the space floating image 3. In response to this, there is also a method of displaying only the object part of the space floating image 3 as an image having brightness. However, the method of displaying only the object part as an image having brightness can obtain a floating sense of the object well, but on the other hand, there is a problem that it is difficult to recognize the depth of the object.

[0224] Therefore, in the display example of Fig. 42A according to this embodiment, two objects, a first button BUT1 displayed as "YES" and a second button BUT2 displayed as "NO", are displayed within a display range 4210 of the floating in space image 3. The two object regions of the first button BUT1 and the second button BUT2 displayed as "NO" are regions that contain images having luminance on the display device 1. A black display region 4220 is arranged around the display regions of these two objects so as to surround the display regions of the objects.

[0225] The black display area 4220 is an area that displays black on the display device 1. That is, the black display area 4220 is an area that has image information that does not have luminance on the display device 1. In other words, the black display area 4220 is an area that does not have image information that has luminance. An area where black is displayed on the display device 1 is a spatial area in which the user cannot see anything in the spatial floating image 3 that is an optical image. Furthermore, in the display example of FIG. 42A, a frame image display area 4250 is arranged in the display range 4210 so as to surround the black display area 4220.

[0226] The frame image display area 4250 is an area in the display device 1 that displays a pseudo frame using an image having luminance. Here, the pseudo frame in the frame image display area 4250 may be a frame image that displays a single color. Alternatively, the pseudo frame in the frame image display area 4250 may be a frame image that displays using an image with a design. Alternatively, the frame image display area 4250 may display a frame such as a dashed line.

[0227] By displaying the frame image of the frame image display area 4250 as described above, the user can easily recognize the plane to which the two objects of the first button BUT1 and the second button BUT2 belong, and can easily recognize the depth positions of the two objects of the first button BUT1 and the second button BUT2. However, since there is a black display area 4220 around these objects that the user cannot see, the floating feeling of the two objects of the first button BUT1 and the second button BUT2 can be emphasized. Note that in the space floating image 3, the frame image display area 4250 exists on the outermost periphery of the display range 4210, but in some cases it does not have to be on the outermost periphery of the display range 4210.

[0228] As described above, according to the display example of FIG. 42A, it is possible to more suitably achieve both the feeling of floating in the air of an object displayed in the space floating image 3 and the recognition of the depth position.

[0229] <<Modification 2 of Object Display>> Fig. 42B is a modified example of the object display in Fig. 42A. This is a display example in which a message stating "touch operation is possible" is displayed near an object that can be touched by the user, such as the first button BUT1 and the second button BUT2. Here, as shown in Fig. 42B, a mark such as an arrow pointing to the object that can be touched by the user may be displayed. In this way, the user can easily recognize the object that can be touched.

[0230] Here, by displaying such messages and marks so that they are surrounded by the black display area 4220, a sense of floating in the air can be achieved.

[0231] <<Modification of the space floating image display device>> Next, a modified example of the space-floating image display device will be described with reference to FIG. 43. The space-floating image display device of FIG. 43 is a modified example of the space-floating image display device of FIG. 3A. The same components as those in FIG. 3A are given the same reference numerals. In the description of FIG. 43, differences from the components in FIG. 3A will be described, and the same components as those in FIG. 3A have already been described in FIG. 3A, so repeated description will be omitted.

[0232] Here, the space-floating image display device of Figure 43, like the space-floating image display device of Figure 3A, converts the image light from the display device 1 into a space-floating image 3 through a polarization separation member 101, a λ / 4 plate 21, and a retroreflective member 2.

[0233] The space-floating image display device of Fig. 43 is different from the space-floating image display device of Fig. 3A in that a physical frame 4310 is provided to surround the space-floating image 3 from the periphery. Here, the physical frame 4310 has an opening window provided along the outer periphery of the space-floating image 3, and the user can visually recognize the space-floating image 3 at the position of the opening window of the physical frame 4310. When the space-floating image 3 is rectangular, the opening window of the physical frame 4310 also has a rectangular shape.

[0234] In the example of Fig. 43, an aerial operation detection sensor 1351 is provided in a part of an opening window of the physical frame 4310. As already explained in Fig. 3C, the aerial operation detection sensor 1351 can detect a touch operation with a user's finger on an object displayed in the floating-in-space image 3.

[0235] In the example of Fig. 43, the physical frame 4310 has a cover structure that covers the polarization separation member 101 on the upper surface of the space floating image display device. Note that the cover structure does not necessarily cover the polarization separation member 101, but may be configured to cover the storage section of the display device 1 and the retroreflective member 2. However, the physical frame 4310 in Fig. 43 is merely an example of this embodiment, and does not necessarily have to have a cover structure.

[0236] Here, Fig. 44 shows the physical frame 4310 and the opening window 4450 of the space floating image display device of Fig. 43 when the space floating image 3 is not displayed. At this time, the user cannot visually recognize the space floating image 3.

[0237] In contrast to this, an example of the configuration of the opening window 4450 of the physical frame 4310 of the space floating image display device of Fig. 43 of this embodiment and an example of the display of the space floating image 3 are shown in Fig. 45. In the example of Fig. 45, the opening window 4450 is configured to approximately coincide with the display range 4210 of the space floating image 3.

[0238] Furthermore, the display example of the floating-in-space image 3 in Fig. 45 displays objects similar to the example in Fig. 42A. Specifically, objects that can be touched by the user, such as the first button BUT1 and the second button BUT2, are displayed. These objects that can be touched by the user are surrounded by a black display area 4220, providing a suitable feeling of floating in space.

[0239] A frame image display area 4470 is provided on the outer periphery surrounding the black display area 4220. The outer periphery of the frame image display area 4470 is the display range 4210, and the edge of the opening window 4450 of the space floating image display device is disposed so as to substantially coincide with the display range 4210.

[0240] 45, the frame image in the frame image display area 4470 is displayed in a color similar to the color of the physical frame 4310 around the opening window 4450. For example, if the physical frame 4310 is white, the frame image in the frame image display area 4470 is also displayed in white. If the physical frame 4310 is gray, the frame image in the frame image display area 4470 is also displayed in gray. For example, if the physical frame 4310 is yellow, the frame image in the frame image display area 4470 is also displayed in yellow.

[0241] In this way, by displaying the frame image in the frame image display area 4470 in a color similar to the color of the physical frame 4310 surrounding the opening window 4450, the spatial continuity between the physical frame 4310 and the frame images in the frame image display area 4470 can be emphasized and conveyed to the user.

[0242] Generally, users can better recognize the space of a physical structure than a floating-in-space image. Therefore, as in the display example of FIG. 45, by displaying the floating-in-space image in such a way as to emphasize the spatial continuity of the physical frame, the user can more easily recognize the depth of the floating-in-space image.

[0243] Furthermore, in the display example of FIG. 45, the spatially floating images of objects that the user can touch, such as the first button BUT1 and the second button BUT2, are formed on the same plane as the frame image display area 4470, so the user can more appropriately recognize the depth of the first button BUT1 and the second button BUT2 based on the depth perception of the physical frame 4310 and the frame image display area 4470.

[0244] That is, according to the display example of Fig. 45, it is possible to more suitably achieve both the feeling of floating in the air and the recognition of the depth position of the object displayed in the floating-in-space image 3. Also, it is possible to more suitably facilitate the recognition of the depth position of the object displayed in the floating-in-space image 3 than in the display example of Fig. 42A.

[0245] Also in the display example of FIG. 45, a mark such as an arrow pointing to an object that can be touched by the user may be displayed as in the display example of FIG. 42B.

[0246] As a modified example of the configuration of the space-floating image display device of FIG. 43, as shown in FIG. 46, a light shielding plate 4610 or a light shielding plate 4620 having a black surface with low light reflectance may be provided inside the cover structure of the physical frame 4310. By providing a light shielding plate in this way, even if a user peers into the inside of the space-floating image display device through an opening window, it is possible to prevent the user from visually recognizing parts and the like unrelated to the space-floating image 3. This makes it possible to prevent a real object unrelated to the space-floating image 3 from being visually recognized behind the black display area 4220 of FIG. 42A or the like, making it difficult to visually recognize the space-floating image 3. In addition, the generation of stray light based on the space-floating image 3 can also be prevented.

[0247] Here, the light shielding plate 4610 and the light shielding plate 4620 form a cylindrical quadrangular prism corresponding to the rectangle of the space-floating image 3, and may be configured to extend from the vicinity of the opening window of the space-floating image display device toward the storage section of the display device 1 and the retroreflective member 2. In addition, in consideration of the divergence angle of light and ensuring the freedom of the user's viewpoint, the opposing light shielding plates may be configured to include a non-parallel quadrangular pyramid shape and extend from the vicinity of the opening window of the space-floating image display device toward the storage section of the display device 1 and the retroreflective member 2. In this case, the quadrangular pyramid shape will be a shape that expands from the vicinity of the opening window of the space-floating image display device toward the storage section of the display device 1 and the retroreflective member 2.

[0248] The cover structure and light shielding plate in Fig. 46 may be used in a space-floating image display device that performs display other than the display example in Fig. 45. In other words, it is not necessary to display the frame image display area 4470. If the physical frame 4310 of the cover structure of the space-floating image display device is arranged to surround the display range 4210 of the space-floating image 3, it can contribute to improving the recognition of the depth position of the displayed object even without the frame image display area 4470 in Fig. 45.

[0249] Although various embodiments have been described above in detail, the present invention is not limited to the above-described embodiments, and various modified examples are included. For example, the above-described embodiments are detailed descriptions of the entire system in order to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0250] The technology according to this embodiment displays high-resolution, high-brightness image information in a state of floating in space, which allows users to operate the system without worrying about contact infection. If the technology according to this embodiment is used in a system used by an unspecified number of users, it is possible to provide a contactless user interface that can be used without worry, reducing the risk of contact infection. This contributes to the achievement of "Good health and well-being for all," one of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0251] In addition, the technology according to this embodiment makes it possible to efficiently reflect only the normal reflected light from the retroreflective material by reducing the divergence angle of the emitted image light and aligning it to a specific polarization, which makes it possible to obtain a bright and clear floating image with high light utilization efficiency. The technology according to this embodiment makes it possible to provide a highly usable non-contact user interface that can significantly reduce power consumption. This contributes to the achievement of the Sustainable Development Goals (SDGs) proposed by the United Nations, "9 Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote technological innovation" and "11 Make cities and towns sustainable and livable."

[0252] Furthermore, the technology according to this embodiment makes it possible to form a floating image using highly directional (linear) image light. The technology according to this embodiment makes it possible to provide a non-contact user interface with little risk of people other than the user looking at the floating image, even when displaying images that require high security, such as those in bank ATMs and ticket machines at train stations, or highly confidential images that should be hidden from people directly facing the user, by displaying highly directional image light. This contributes to "Sustainable cities and towns" in the Sustainable Development Goals (SDGs) proposed by the United Nations. [Explanation of symbols]

[0253] 1...display device, 2...retroreflective member, 3...spatial image (floating image in space), 105...window glass, 100...transparent member, 101...polarized light separation member, 12...absorptive polarizing plate, 13...light source device, 54...light direction conversion panel, 151...retroreflective member, 102, 202...LED substrate, 203...light guide, 205, 271...reflective sheet, 206, 270...phase difference plate, 300...floating image in space, 301...ghost image of floating image in space, 302...floating image in space Ghost image of an image, 230...user, 1000...space floating image display device, 1110...control unit, 1160...image control unit, 1180...imaging unit, 1102...image display unit, 1350...air operation detection unit, 1351...air operation detection sensor, 1500...virtual light source, 1510...virtual shadow, 1610...input content display area, 1700...touch pen, 1800...wearable terminal, 1900...diaphragm, 4220...black display area, 4250...framed image display area.

Claims

1. A display unit for displaying an image; A retroreflector that reflects the image light from the display unit and forms a spatial floating image in the air by the reflected light; Equipped with In the display range of the floating image in space, there is an area in which an object is displayed, a black display area is disposed surrounding the area in which the object is displayed; A physical frame is disposed so as to surround the floating image in space, The physical frame forms an opening window having a cover structure that covers the display unit and a storage unit that stores the retroreflector, A light-shielding plate is disposed inside the cover structure between the opening window and the storage section that stores the display unit and the retroreflector, and the light-shielding plate extends from at least both the upper end and the lower end of the opening window toward the storage section that stores the display unit and the retroreflector. A floating image display device.

2. 2. The space floating image display device according to claim 1, The black display area is an area in which there is no image information having luminance in the display image of the display unit corresponding to the spatial floating image. A floating image display device.

3. 2. The space floating image display device according to claim 1, a sensor for detecting a position of a user's finger performing a touch operation on the object; A floating image display device.

4. 4. The space floating image display device according to claim 3, displaying a message near the object, indicating that the object is a touch-operable object; A floating image display device.

5. 5. The space floating image display device according to claim 4, displaying a mark indicating the object in addition to the message; A floating image display device.

6. 2. The space floating image display device according to claim 1, The light blocking plate forms a cylindrical rectangular prism. A floating image display device.

7. 2. The space floating image display device according to claim 1, The light-shielding plate forms a quadrangular pyramid. A floating image display device.

8. 8. The space floating image display device according to claim 7, The shape of the truncated quadrangular pyramid is a shape that expands from the vicinity of the opening window toward a storage section that stores the display section and the retroreflector. A floating image display device.

Images