Spatial floating image display device

Abstract

To provide a more suitable space floating video display device.SOLUTION: A space floating video display device comprises: a display device which displays video; a recurrent reflection member 2 which reflects video light from the display device, and forms space floating video 3 in the air by reflected light; a space operation detection sensor 1351 which detects touch operation of a finger 210 of a user 230 to one or more objects displayed on the space floating video; and a control unit. The control unit, when the user performs the touch operation to the object, assists the touch operation to the user, based on the detection result of the touch operation using the space operation detection sensor.SELECTED DRAWING: Figure 34

Description

Technical Field

[0001] The present invention relates to a spatial floating image display device.

Background Art

[0002] As a spatial floating information display system, a display method in which a video display device directly displays an image toward the outside and a display screen is already known. Also, a detection system for reducing false detection of operations on the operation surface of the displayed spatial image is disclosed in, for example, Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, a touch operation on a spatial floating image is not performed on a physical button, a touch panel, or the like. Therefore, there are cases where the user cannot recognize whether or not a touch operation has been performed.

[0005] Therefore, an object of the present invention is to provide a more suitable spatial floating image display device.

Means for Solving the Problems

[0006] To solve the above problems, for example, the configuration described in the claims may be adopted. The present application includes multiple means for solving the above problems, but to give one example, the floating image display device comprises a display device that displays an image, a retroreflective member that reflects the image light from the display device and forms a floating image in the air with the reflected light, a sensor that detects the touch operation of a user's finger on one or more objects displayed in the floating image, and a control unit, wherein the control unit is configured to assist the user in touching an object based on the detection result of the touch operation using the sensor when the user touches an object. [Effects of the Invention]

[0007] According to the present invention, a more suitable spatial floating image display device can be realized. Other problems, configurations, and effects will be clarified in the following description of embodiments. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of how to use a spatially floating image display device according to one embodiment of the present invention. [Figure 2] This figure shows an example of the main component configuration and retroreflective component configuration of a spatial floating image display device according to one embodiment of the present invention. [Figure 3A] This figure shows an example of how to install a floating image display device. [Figure 3B] This figure shows another example of a method for installing a floating image display device. [Figure 3C] This figure shows an example of the configuration of a floating image display device. [Figure 4] This figure shows another example of the main component configuration of a spatially floating image display device according to one embodiment of the present invention. [Figure 5] This is an explanatory diagram illustrating the function of the sensing device used in the floating image display device. [Figure 6] This is an explanatory diagram of the principle of 3D image display used in a spatially floating image display device. [Figure 7]It is an explanatory diagram of a measurement system for evaluating the characteristics of a reflective polarizing plate. [Figure 8] It is a characteristic diagram showing the transmittance characteristics of a reflective polarizing plate with respect to the light incident angle of the transmission axis. [Figure 9] It is a characteristic diagram showing the transmittance characteristics of a reflective polarizing plate with respect to the light incident angle of the reflection axis. [Figure 10] It is a characteristic diagram showing the transmittance characteristics of a reflective polarizing plate with respect to the light incident angle of the transmission axis. [Figure 11] It is a characteristic diagram showing the transmittance characteristics of a reflective polarizing plate with respect to the light incident angle of the reflection axis. [Figure 12] It is a cross-sectional view showing an example of the specific configuration of a light source device. [Figure 13] It is a cross-sectional view showing an example of the specific configuration of a light source device. [Figure 14] It is a cross-sectional view showing an example of the specific configuration of a light source device. [Figure 15] It is a layout diagram showing the main part of a spatial floating image display device according to an embodiment of the present invention. [Figure 16] It is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. [Figure 17] It is a cross-sectional view showing an example of the specific configuration of a light source device. [Figure 18] It is a cross-sectional view showing an example of the specific configuration of a light source device. [Figure 19] It is a cross-sectional view showing an example of the specific configuration of a light source device. [Figure 20] It is an explanatory diagram for explaining the light source diffusion characteristics of a video display device. [Figure 21] It is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Figure 22] It is an explanatory diagram for explaining the diffusion characteristics of a video display device. [Figure 23] It is a cross-sectional view showing the configuration of a video display device. [Figure 24] It is an explanatory diagram for explaining the generation principle of ghost images in the prior art. [Figure 25] It is a cross-sectional view showing the configuration of a display device according to an embodiment of the present invention. [Figure 26] This is a diagram for explaining an example of the display of a display device according to an embodiment of the present invention. [Figure 27] This is a diagram for explaining an example of an auxiliary method for touch operation using a virtual shadow. [Figure 28] This is a diagram for explaining an example of an auxiliary method for touch operation using a virtual shadow. [Figure 29] This is a diagram for explaining an example of an auxiliary method for touch operation using a virtual shadow. [Figure 30] This is a diagram for explaining another example of an auxiliary method for touch operation using a virtual shadow. [Figure 31] This is a diagram for explaining another example of an auxiliary method for touch operation using a virtual shadow. [Figure 32] This is a diagram for explaining another example of an auxiliary method for touch operation using a virtual shadow. [Figure 33] This is a diagram for explaining a method of setting a virtual light source. [Figure 34] This is a block diagram showing an example of a method for detecting the position of a finger. [Figure 35] This is a block diagram showing another example of a method for detecting the position of a finger. [Figure 36] This is a block diagram showing another example of a method for detecting the position of a finger. [[ID=3%3]] [Figure 37] This is a diagram for explaining a method of displaying the input content and assisting touch operation. [Figure 38] This is a diagram for explaining a method of highlighting the input content and assisting touch operation. [Figure 39] This is a diagram for explaining an example of a method of assisting touch operation by vibration. [Figure 40] This is a diagram for explaining another example of an auxiliary method for touch operation by vibration. [Figure 41] This is a diagram for explaining another example of an auxiliary method for touch operation by vibration.

Embodiments for Carrying Out the Invention

[0009] Embodiments of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to the examples described herein, and various modifications and alterations are possible by those skilled in the art within the scope of the technical ideas disclosed herein. Furthermore, in all the figures used to illustrate the present invention, components having the same function are given the same reference numerals, and repeated descriptions may be omitted.

[0010] The following embodiment relates to an image display device capable of transmitting an image generated by image light from an image light source through a transparent member that partitions a space, such as glass, and displaying it as a floating image in space outside the transparent member.

[0011] According to the following embodiment, a suitable video display device can be realized for applications such as bank ATMs, train station ticket machines, and digital signage. For example, currently, bank ATMs and train station ticket machines typically use touch panels, but by using a transparent glass surface or a light-transmitting plate material, high-resolution video information can be displayed on this glass surface or light-transmitting plate material in a state of floating in space. At this time, by making the divergence angle of the emitted video light small, i.e., acute, and further aligning it to a specific polarization, only the normal reflected light is efficiently reflected by the retroreflective material, resulting in high light utilization efficiency. This suppresses ghost images that occur in addition to the main floating image, which was a problem in conventional retroreflection methods, and a clear floating image can be obtained. Furthermore, the device including the light source of this embodiment can provide a novel and highly usable floating image display device (floating image display system) that can significantly reduce power consumption. In addition, for example, a floating image display device for vehicles can be provided that enables so-called unidirectional floating image display, which is visible inside and / or outside a vehicle.

[0012] On the other hand, in conventional technology, an organic EL panel or liquid crystal panel is combined with a retroreflective member 151 as a high-resolution color display image source 150. In conventional technology, because the image light is diffused at a wide angle, in addition to the reflected light that is normally reflected by the retroreflective member 151, ghost images 301 and 302 are generated by the image light that is incident on the retroreflective member 2a at an oblique angle, as shown in Figure 24, and the image quality of the floating image in space is impaired. Furthermore, as shown in Figure 23, in addition to the normal floating image in space 300, multiple ghost images such as the first ghost image 301 and the second ghost image 302 are generated. As a result, the same floating image in space, which is a ghost image, can be monitored by people other than the observer, which poses a major security challenge.

[0013] <Spatial Floating Image Display Device 1> Figure 1 is a diagram showing an example of how to use a spatially floating image display device according to one embodiment of the present invention, and is a diagram showing the overall configuration of the spatially floating image display device according to this embodiment. The specific configuration of the spatially floating image display device will be described in detail using Figure 2, etc., but light with narrow-angle directivity characteristics and specific polarization is emitted from the image display device 1 as an image light beam, enters the retroreflective member 2, is retroreflected and passes through a transparent member 100 (glass, etc.), and forms an aerial image (spatially floating image 3), which is a real image, on the outside of the glass surface.

[0014] Furthermore, in stores and other similar establishments, the space is partitioned by a translucent material such as glass, called a "show window" (also known as "window glass") 105. According to the spatial floating image display device of this embodiment, it is possible to transmit such a transparent material and display the floating image in one direction to the outside and / or inside of the store (space).

[0015] In Figure 1(A), the inside of the window glass 105 (inside the store) is shown in the depth direction, and the outside (for example, the sidewalk) is shown in the foreground. On the other hand, by providing means for reflecting specific polarizations in the window glass 105, it is also possible to reflect the light and form an aerial image at a desired location inside the store.

[0016] Figure 1(B) is a schematic block diagram showing the configuration of the video display device 1 described above. The video display device 1 includes a video display unit that displays the original image of the aerial image, a video control unit that converts the input video to match the resolution of the panel, and a video signal receiving unit that receives video signals. The video signal receiving unit supports wired input signals such as HDMI (High-Definition Multimedia Interface) input and wireless input signals such as Wi-Fi (Wireless Fidelity), and can function as a standalone video receiver and display device, and can also display video information from tablets, smartphones, etc. Furthermore, by connecting a stick PC, it can be equipped with the capability to perform calculation processing and video analysis processing.

[0017] Figure 2 shows an example of the main components and retroreflective components of a spatially floating image display device according to one embodiment of the present invention. The configuration of the spatially floating image display device will be explained in more detail using Figure 2. As shown in Figure 2(A), a display device 1 is provided that emits image light of a specific polarization at a narrow angle in the oblique direction of a transparent member 100 such as glass. The display device 1 comprises a liquid crystal display panel 11 and a light source device 13 that generates light of a specific polarization having narrow-angle diffusion characteristics.

[0018] The image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 (in the figure, the polarization separation member 101 is formed in a sheet shape and adhered to the transparent member 100) which has a film that selectively reflects the image light of the specific polarization, and is incident on the retroreflective member 2. A λ / 4 plate 21 is provided on the image light incident surface of the retroreflective member. The image light is polarized from the specific polarization to the other polarization by passing through the λ / 4 plate 21 twice, once when it is incident on the retroreflective member and once when it is emitted. Here, since the polarization separation member 101, which selectively reflects the image light of the specific polarization, has the property of transmitting the polarization of the other polarization after polarization conversion, the image light of the specific polarization after polarization conversion is transmitted through the polarization separation member 101. The image light that has been transmitted through the polarization separation member 101 forms a spatially floating image 3, which is a real image, on the outside of the transparent member 100.

[0019] The light that forms the floating image 3 is a collection of light rays converging from the retroreflective member 2 to the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, unlike the diffused image light formed on a screen by a typical projector, the floating image 3 is an image with high directivity. Thus, in the configuration shown in Figure 2, when a user views from the direction of arrow A, the floating image 3 is perceived as a bright image. However, when another person views from the direction of arrow B, the floating image 3 cannot be seen as an image at all. This characteristic is very suitable for use in systems that display images requiring high security or highly confidential images that should be hidden from people directly facing the user.

[0020] Depending on the performance of the retroreflective member 2, the polarization axes of the reflected image light may become misaligned. In this case, some of the image light with misaligned polarization axes is reflected by the polarization separation member 101 and returns to the display device 1. This light is re-reflected on the image display surface of the liquid crystal display panel 11 that constitutes the display device 1, potentially generating ghost images and degrading the image quality of the floating image. Therefore, in this embodiment, an absorbing polarizing plate 12 is provided on the image display surface of the display device 1. By allowing the image light emitted from the display device 1 to pass through the absorbing polarizing plate 12 and absorbing the reflected light returning from the polarization separation member 101, the above-mentioned re-reflection can be suppressed. This prevents image quality degradation due to ghost images of the floating image.

[0021] The polarization separation member 101 described above may be formed, for example, from a reflective polarizer or a multilayer metal film that reflects specific polarizations.

[0022] Next, Figure 2(B) shows the surface shape of a typical retroreflective material 2, manufactured by Nippon Carbide Industries Co., Ltd., which was used in this study. Light rays incident on the interior of the regularly arranged hexagonal prisms are reflected by the walls and bottom surfaces of the hexagonal prisms and emitted as retroreflected light in the direction corresponding to the incident light, displaying a floating image in space, which is a real image, based on the image displayed on the display device 1. The resolution of this floating image in space depends not only on the resolution of the liquid crystal display panel 11, but also largely on the outer diameter D and pitch P of the retroreflective part of the retroreflective material 2 shown in Figure 2(B). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) liquid crystal display panel, even if one pixel (one triplet) is approximately 80 μm, if, for example, the diameter D of the retroreflective part is 240 μm and the pitch is 300 μm, one pixel of the floating image in space will be equivalent to 300 μm. Therefore, the effective resolution of the floating image in space is reduced to about 1 / 3. Therefore, in order to make the resolution of the floating image in space equivalent to the resolution of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to that of one pixel of the liquid crystal display panel. On the other hand, in order to suppress the occurrence of moiré patterns caused by the retroreflective material and the pixels of the liquid crystal display panel, it is good to design the respective pitch ratios to be outside of integer multiples of one pixel. Also, the shape should be arranged so that none of the sides of the retroreflective portion overlap with any of the sides of one pixel of the liquid crystal display panel.

[0023] On the other hand, in order to manufacture retroreflective members at a low cost, it is preferable to use a roll press method for molding. Specifically, this method involves aligning the retroreflective portion and shaping it on a film. The inverse shape of the shape to be shaped is formed on the roll surface, ultraviolet-curing resin is applied to a fixing base material, and the film is passed between the rolls to shape the required form. The film is then cured by irradiation with ultraviolet light to obtain a retroreflective member 2 of the desired shape.

[0024] <<Installation Method for a Space-Floating Image Display Device>> Next, the installation method of the floating image display device will be described. The installation method of the floating image display device can be freely changed depending on the usage. Figure 3A is a diagram showing an example of the installation method of the floating image display device. The floating image display device shown in Figure 3A is installed horizontally so that the side on which the floating image 3 is formed faces upward. That is, in Figure 3A, the floating image display device is installed so that the transparent member 100 faces upward, and the floating image 3 is formed above the floating image display device.

[0025] Figure 3B shows another example of how to install the floating image display device. In Figure 3B, the floating image display device is installed vertically so that the side on which the floating image 3 is formed faces sideways (towards the user 230). That is, in Figure 3B, the floating image display device is installed so that the transparent member 100 faces sideways, and the floating image 3 is formed on the side of the floating image display device (towards the user 230).

[0026] <<Configuration of the floating image display device>> Next, the configuration of the floating image display device 1000 will be described. Figure 3C is a block diagram showing an example of the internal configuration of the floating image display device 1000.

[0027] The floating video display device 1 includes a retroreflective section 1101, a video display section 1102, a light guide 1104, a light source 1105, a power supply 1106, an operation input section 1107, a non-volatile memory 1108, a memory 1109, a control section 1110, a video signal input section 1131, an audio signal input section 1133, a communication section 1132, an aerial operation detection sensor 1351, an aerial operation detection section 1350, an audio output section 1140, a video control section 1160, a storage section 1170, an imaging section 1180, and the like.

[0028] Each component of the floating video display device 1000 is housed in the housing 1190. Note that the imaging unit 1180 and the aerial operation detection sensor 1351 shown in Figure 3C may be provided on the outside of the housing 1190.

[0029] The retroreflective section 1101 in Figure 3C corresponds to the retroreflective member 2 in Figure 2. The retroreflective section 1101 retroreflectively reflects light modulated by the image display section 1102. The floating image 3 is formed by the light from the retroreflective section 1101 that is output to the outside of the floating image display device 1000.

[0030] The image display unit 1102 in Figure 3C corresponds to the liquid crystal display panel 11 in Figure 2. The light source 1105 in Figure 3C corresponds to the light source device 13 in Figure 2. Furthermore, the image display unit 1102, light guide 1104, and light source 1105 in Figure 3C correspond to the display device 1 in Figure 2.

[0031] The video display unit 1102 is a display unit that generates an image by modulating transmitted light based on a video signal input under control by the video control unit 1160, which will be described later. The video display unit 1102 corresponds to the liquid crystal display panel 11 in Figure 2. For example, a transmissive liquid crystal panel can be used as the video display unit 1102. Alternatively, a reflective liquid crystal panel or a DMD (Digital Micromirror Device: registered trademark) panel that modulates reflected light may also be used as the video display unit 1102.

[0032] The light source 1105 generates light for the image display unit 1102 and is a solid-state light source such as an LED or laser light source. The power supply 1106 converts the AC current input from an external source into DC current and supplies power to the light source 1105. The power supply 1106 also supplies the necessary DC current to each part of the floating image display device 1000.

[0033] The light guide 1104 guides the light generated by the light source 1105 and illuminates the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be called the backlight of the image display unit 1102. Various combinations of the light guide 1104 and the light source 1105 are possible. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be explained in detail later.

[0034] The aerial operation detection sensor 1351 is a sensor that detects the operation of the floating video 3 by the user 230's finger. The aerial operation detection sensor 1351 senses, for example, the area that overlaps with the entire display range of the floating video 3. Alternatively, the aerial operation detection sensor 1351 may sense only the area that overlaps with at least a portion of the display range of the floating video 3.

[0035] Specific examples of the aerial operation detection sensor 1351 include distance sensors using invisible light such as infrared, invisible light lasers, and ultrasonic waves. The aerial operation detection sensor 1351 may also be configured by combining multiple sensors to detect coordinates on a two-dimensional plane. Furthermore, the aerial operation detection sensor 1351 may consist of a Time of Flight (ToF) LiDAR (Light Detection and Ranging) or an image sensor.

[0036] The aerial operation detection sensor 1351 only needs to be able to sense touch operations, etc., on an object displayed as a floating image 3 in space, using the user's finger. Such sensing can be performed using existing technologies.

[0037] The aerial operation detection unit 1350 acquires sensing signals from the aerial operation detection sensor 1351 and, based on the sensing signals, calculates whether or not the user 230's finger has made contact with an object in the floating spatial image 3, and the position where the user 230's finger made contact with the object (contact position). The aerial operation detection unit 1350 is composed of circuits such as an FPGA (Field Programmable Gate Array). In addition, some functions of the aerial operation detection unit 1350 may be implemented in software, for example, by a spatial operation detection program executed in the control unit 1110.

[0038] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be built into the floating video display device 1000, or they may be provided separately from the floating video display device 1000. When provided separately from the floating video display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to transmit information and signals to the floating video display device 1000 via wired or wireless communication lines or video signal transmission lines.

[0039] Furthermore, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided as separate components. This makes it possible to construct a system in which the floating video display device 1000 without the aerial operation detection function is used as the main unit, and only the aerial operation detection function can be added as an option. Alternatively, the aerial operation detection sensor 1351 may be a separate component, and the aerial operation detection unit 1350 may be built into the floating video display device 1000. When it is desired to place the aerial operation detection sensor 1351 more freely relative to the installation position of the floating video display device 1000, there is an advantage to having only the aerial operation detection sensor 1351 as a separate component.

[0040] The imaging unit 1180 is a camera equipped with an image sensor, and it captures images of the space near the floating video 3, and / or the face, arms, fingers, etc., of the user 230. Multiple imaging units 1180 may be provided. By using multiple imaging units 1180, or by using imaging units with depth sensors, the aerial operation detection unit 1350 can be assisted when detecting touch operations on the floating video 3 by the user 230.

[0041] For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that detects whether or not an object has entered a plane (intrusion detection plane) that includes the display surface of the floating spatial image 3, the aerial operation detection sensor 1351 may not be able to detect information such as how far away an object that has not entered the intrusion detection plane (for example, a user's finger) is from the intrusion detection plane, or how close an object is to the intrusion detection plane.

[0042] In such cases, the distance between the object and the intrusion detection plane can be calculated by using information such as depth calculation information of the object based on images captured by multiple imaging units 1180 and depth information of the object from a depth sensor. This information, as well as various other information such as the distance between the object and the intrusion detection plane, is then used for various display controls of the floating spatial image 3.

[0043] Alternatively, instead of using the aerial operation detection sensor 1351, the aerial operation detection unit 1350 may detect touch operations on the floating video 3 by the user 230 based on the image captured by the imaging unit 1180.

[0044] Alternatively, the imaging unit 1180 may capture an image of the face of the user 230 operating the floating video 3, and the control unit 1110 may perform user identification processing. Furthermore, in order to determine whether other people are standing around or behind the user 230 operating the floating video 3 and whether they are peeking at the user 230's operation of the floating video 3, the imaging unit 1180 may capture an area that includes the user 230 operating the floating video 3 and the area surrounding the user 230.

[0045] The operation input unit 1107 is, for example, an operation button or a light receiving unit for a remote controller, and inputs signals for operations other than aerial operations (touch operations) by the user 230. Separately from the aforementioned user 230 who touches the floating spatial image 3, the operation input unit 1107 may be used, for example, for an administrator to operate the floating spatial image display device 1000.

[0046] The video signal input unit 1131 receives video data by connecting an external video output device. The audio signal input unit 1133 receives audio data by connecting an external audio output device. The audio output unit 1140 can output audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may also output built-in operation sounds or error warning sounds.

[0047] The non-volatile memory 1108 stores various data used by the floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, data for various operations displayed on the floating image 3, display icons, data and layout information for objects that the user operates. Memory 1109 stores video data to be displayed as the floating image 3, control data for the device, and the like.

[0048] The control unit 1110 controls the operation of each connected part. The control unit 1110 may also work in cooperation with a program stored in the memory 1109 to perform calculations based on information acquired from each part of the floating image display device 1000. The communication unit 1132 communicates with external devices and external servers via a wired or wireless interface. Various types of data, such as video data, image data, and audio data, are transmitted and received through the communication unit 1132.

[0049] The storage unit 1170 is a storage device that records various types of data and information, such as video data, image data, and audio data. For example, the storage unit 1170 may have various types of data and information, such as video data, image data, and audio data, pre-recorded in it at the time of product shipment. The storage unit 1170 may also record various types of data and information, such as video data, image data, and audio data, acquired from external devices or external servers via the communication unit 1132.

[0050] The video data, image data, etc., recorded in the storage unit 1170 are output as floating-in-space video 3 via the video display unit 1102 and the retroreflective unit 1101. The video data, image data, etc., of display icons and user-operable objects, etc., that are displayed as floating-in-space video 3 are also recorded in the storage unit 1170.

[0051] Layout information such as display icons and objects shown as the floating spatial image 3, as well as various metadata information related to the objects, are also recorded in the storage unit 1170. Audio data recorded in the storage unit 1170 is output as audio from, for example, the audio output unit 1140.

[0052] The video control unit 1160 performs various controls related to the video signals input to the video display unit 1102. For example, the video control unit 1160 performs video switching control, such as determining which video signal to input to the video display unit 1102 from among the video signals stored in the memory 1109 and the video signals (video data) input to the video signal input unit 1131.

[0053] Alternatively, the video control unit 1160 may generate a superimposed video signal by superimposing the video signal to be stored in the memory 1109 and the video signal input from the video signal input unit 1131, and then input the superimposed video signal to the video display unit 1102 to form the composite image as a floating image 3.

[0054] Furthermore, the video control unit 1160 may perform image processing on video signals input from the video signal input unit 1131 and video signals stored in the memory 1109. Examples of image processing include scaling, which enlarges, reduces, and transforms images; brightness adjustment, which changes the brightness; contrast adjustment, which changes the contrast curve of an image; and retinex processing, which decomposes an image into its light components and changes the weighting of each component.

[0055] Furthermore, the video control unit 1160 may perform special effects video processing on the video signal input to the video display unit 1102 to assist the user 230's aerial operation (touch operation). Special effects video processing is performed, for example, based on the detection result of the user 230's touch operation by the aerial operation detection unit 1350 or on the image captured by the imaging unit 1180 of the user 230.

[0056] As explained above, the floating image display device 1000 is equipped with various functions. However, the floating image display device 1000 does not need to have all of these functions; any configuration is acceptable as long as it has the function of forming the floating image 3.

[0057] <Spatial Floating Image Display Device 2> Figure 4 shows another example of the main components of a spatially floating image display device according to one embodiment of the present invention. The image display element 11 constituting the display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates light of a specific polarization having narrow-angle diffusion characteristics. For example, it is composed of a small liquid crystal display panel with a screen size of about 5 inches to a large liquid crystal display panel exceeding 80 inches. A polarization separation member 101, such as a reflective polarizer, is provided on the surface of the folded mirror 22 to reflect the image light from the liquid crystal display panel 11 toward the retroreflective member 2. The image light of a specific polarization from the display device 1 is reflected by a film (a sheet 101 is attached in the figure) that selectively reflects the image light of a specific polarization provided on a transparent member 100, and incident on the retroreflective member 2.

[0058] A λ / 4 plate 21 is provided on the light incident surface of the retroreflective member, and by passing the image light through it twice, the polarization is converted, converting one polarization to the other polarization, thereby transmitting through the polarization separation member 101 and displaying the spatially floating image 3, which is a real image, on the outside of the transparent member 100. An absorbing polarizer is provided on the external light incident surface of the transparent member 100. In the polarization separation member 101 described above, retroreflection causes the polarization axes to become uneven, so some of the image light is reflected back to the display device 1. This light is reflected again on the image display surface of the liquid crystal display panel 11 that constitutes the display device 1, generating a ghost image and significantly degrading the image quality of the spatially floating image. Therefore, in this embodiment, an absorbing polarizer 12 is provided on the image display surface of the display device 1, allowing the image light to pass through and absorbing the reflected light described above, thereby preventing image quality degradation due to ghost images of the spatially floating image. Furthermore, to reduce image quality degradation due to sunlight or illumination light outside the set, it is preferable to provide an absorbing polarizer 12 on the surface of the transparent member 100. The polarization separation member 101 is formed from a reflective polarizer or a metal multilayer film that reflects specific polarizations.

[0059] Next, to sense the relationship between the distance and position of the object and the sensor 44 in relation to the spatial floating image display device described above, multiple layers of sensors 44 with TOF (Time of Fly) functionality are arranged as shown in Figure 5, making it possible to sense not only the coordinates of the object in the planar direction but also its coordinates in the depth direction, direction of movement, and speed of movement. To read the distance and position in two dimensions, multiple combinations of ultraviolet light-emitting and light-receiving units are arranged linearly, and light from the light-emitting point is shone onto the object, and the reflected light is received by the light-receiving unit. The distance to the object is clearly determined by the product of the difference between the time of emission and the time of reception and the speed of light. Furthermore, the coordinates on the plane can be read from the coordinates at the point where the difference between the time of emission and the time of reception is smallest using multiple light-emitting and light-receiving units. In this way, the coordinates of the object on a plane (2D) and 3D coordinate information can also be obtained by combining multiple sensors as described above.

[0060] Furthermore, the method for obtaining a three-dimensional floating image as a floating image display device described above will be explained using Figure 6. Figure 6 is an explanatory diagram of the principle of three-dimensional image display used in the floating image display device. Horizontal lenticular lenses are positioned according to the pixels of the image display screen of the liquid crystal display panel 11 of the display device 1 shown in Figure 4. As a result, as shown in Figure 6, in order to display motion parallax from three directions, P1, P2, and P3 in the horizontal direction of the screen, the images from the three directions are treated as one block every three pixels, and image information from the three directions is displayed for each pixel. The direction of light emission is controlled by the action of the corresponding lenticular lens (shown as vertical lines in Figure 6) to separate and emit light in three directions. As a result, a three-dimensional image with three parallaxes can be displayed.

[0061] <Reflective polarizing plate> In the floating image display device of this embodiment, the polarization separation member 101 is used to improve the contrast performance that determines the image quality compared to a general half-mirror. As an example of the polarization separation member 101 of this embodiment, the characteristics of a reflective polarizer will be described. Figure 7 is an explanatory diagram of the measurement system used to evaluate the characteristics of the reflective polarizer. The transmission and reflection characteristics of the reflective polarizer in Figure 7 with respect to the angle of incidence of light rays from a direction perpendicular to the polarization axis are shown as V-AOI in Figures 8 and 9, respectively. Similarly, the transmission and reflection characteristics of the reflective polarizer with respect to the angle of incidence of light rays from a direction horizontal to the polarization axis are shown as H-AOI in Figures 10 and 11, respectively.

[0062] As shown in Figures 8 and 9, the characteristics of a grid-structured reflective polarizer deteriorate with respect to light perpendicular to the polarization axis. For this reason, a specification aligned with the polarization axis is desirable, and the light source in this embodiment, which can emit the image light from the liquid crystal display panel at a narrow angle, is an ideal light source. Similarly, the characteristics in the horizontal direction also deteriorate with respect to light coming from an oblique angle. Considering these characteristics, the following describes an example configuration of this embodiment in which a light source that can emit the image light from the liquid crystal display panel at an even narrower angle is used as the backlight of the liquid crystal display panel. This makes it possible to provide high-contrast floating images.

[0063] <Display device> Next, the display device 1 of this embodiment will be described with reference to the figures. The display device 1 of this embodiment includes a video display element 11 (liquid crystal display panel) and a light source device 13 that constitutes its light source. Figure 12 shows the light source device 13 together with the liquid crystal display panel as an exploded perspective view.

[0064] As shown by arrow 30 in Figure 12, this liquid crystal display panel (image display element 11) obtains an illumination beam with narrow-angle diffusion characteristics from the light source device 13, which is a backlight device, that is, a laser beam with strong directivity (straight-line propagation) and characteristics similar to laser light with the polarization plane aligned in one direction. The image light, which is modulated according to the input image signal, is reflected by the retroreflective member 2 and transmitted through the transparent member 100 to form a floating image, which is a real image (see Figure 1). In Figure 12, the display device 1 is configured to include the liquid crystal display panel 11, an optical direction conversion panel 54 that controls the directional characteristics of the light beam emitted from the light source device 13, and a narrow-angle diffuser plate (not shown) as needed. That is, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light with a specific polarization is emitted with the intensity of the light modulated by the image signal (see arrow 30 in Figure 12). This allows the desired image to be projected as highly directional (straight-line) light of a specific polarization via the optical direction conversion panel 54 toward the retroreflective member 2. After being reflected by the retroreflective member 2, the light is transmitted toward the eyes of an observer outside the store (space) to form a floating image 3 in space. A protective cover 50 (see Figures 13 and 14) may be provided on the surface of the optical direction conversion panel 54.

[0065] In this embodiment, in order to improve the utilization efficiency of the luminous flux 30 emitted from the light source device 13 and to significantly reduce power consumption, the display device 1, which includes the light source device 13 and the liquid crystal display panel 11, can project light from the light source device 13 (see arrow 30 in Figure 12) toward the retroreflective member 2, and after reflection by the retroreflective member 2, the directivity can be controlled by a transparent sheet (not shown) provided on the surface of a transparent member 100 (such as a window glass 105) to form a floating image at a desired position. Specifically, this transparent sheet controls the image formation position of the floating image while providing high directivity with optical components such as a Fresnel lens or a linear Fresnel lens. As a result, the image light from the display device 1 reaches an observer outside the show window 105 (for example, on the sidewalk) efficiently with high directivity (straight-line propagation), like laser light, and as a result, it is possible to display a high-quality floating image in high resolution and significantly reduce the power consumption of the display device 1, including the LED element 201 of the light source device 13.

[0066] <Example of a display device 1> Figure 13 shows an example of the specific configuration of the display device 1. In Figure 13, a liquid crystal display panel 11 and a light direction conversion panel 54 are arranged on top of the light source device 13 shown in Figure 12. The light source device 13 is formed from, for example, plastic on the case shown in Figure 12, and houses LED elements 201 and a light guide 203 inside. The end face of the light guide 203 has a lens shape that gradually increases in cross-sectional area toward the light receiving part, in order to convert the divergent light emitted from each LED element 201 into a nearly parallel luminous beam, and has the effect of gradually decreasing the divergence angle by undergoing multiple total internal reflections as the light propagates through the interior. The liquid crystal display panel 11, which constitutes the display device 1, is mounted on its upper surface. Furthermore, an LED (Light Emitting Diode) element 201, which is a semiconductor light source, and an LED substrate 202 on which its control circuit is mounted are attached to one side of the case of the light source device 13 (the left end face in this example). In addition, a heat sink, which is a component for cooling the heat generated by the LED element and the control circuit, may be attached to the outer surface of the LED substrate 202.

[0067] Furthermore, the frame (not shown) of the liquid crystal display panel, which is mounted on the top surface of the case of the light source device 13, is configured by mounting the liquid crystal display panel 11 attached to the frame, and also by mounting an FPC (Flexible Printed Circuits) (not shown) electrically connected to the liquid crystal display panel. That is, the liquid crystal display panel 11, which is a liquid crystal display element, together with the LED element 201, which is a solid light source, generates a display image by modulating the intensity of transmitted light based on a control signal from a control circuit (not shown) that constitutes an electronic device. At this time, the generated image light has a narrow diffusion angle and consists only of specific polarization components, so a new and unprecedented image display device is obtained that is similar to a surface-emitting laser image source driven by an image signal. Currently, it is technically and safely impossible to obtain a laser beam of the same size as the image obtained by the above-described display device 1 using a laser device. Therefore, in this embodiment, for example, light similar to the surface-emitting laser image light described above is obtained from a light beam from a general light source equipped with an LED element.

[0068] Next, the configuration of the optical system housed within the case of the light source device 13 will be explained in detail with reference to Figure 13 and Figure 14.

[0069] Figures 13 and 14 are cross-sectional views, and therefore only one of the multiple LED elements 201 constituting the light source is shown. These are converted into approximately collimated light by the shape of the light-receiving end face 203a of the light guide 203. For this reason, the light-receiving portion of the end face of the light guide and the LED elements are mounted while maintaining a predetermined positional relationship. Each of these light guides 203 is formed from a light-transmitting resin such as acrylic. The LED light-receiving surface at the end of this light guide has, for example, a cone-convex outer surface obtained by rotating a parabolic cross-section, with a recess at its apex that forms a convex portion (i.e., a convex lens surface) in its center, and a convex lens surface (or a concave lens surface that is recessed inward) in the center of its planar portion (not shown). Furthermore, the outer shape of the light-receiving section of the light guide to which the LED element 201 is attached is a parabolic shape that forms a conical outer surface, and is set within an angle range in which light emitted from the LED element in the periphery direction can be totally reflected inside, or a reflective surface is formed.

[0070] On the other hand, the LED elements 201 are each positioned at predetermined locations on the surface of the LED substrate 202, which is their circuit board. The LED substrate 202 is fixed to the LED collimator (light-receiving end face 203a) such that the LED elements 201 on its surface are each positioned in the center of the aforementioned recess.

[0071] With this configuration, the shape of the light-receiving end face 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as substantially parallel light, thereby improving the utilization efficiency of the generated light.

[0072] As described above, the light source device 13 is configured by attaching a light source unit, which consists of multiple LED elements 201 arranged in a row, to a light-receiving end surface 203a, which is a light-receiving part provided on the end face of a light guide 203. The divergent light beam from the LED elements is converted into approximately parallel light by the lens shape of the light-receiving end surface 203a on the end face of the light guide, and guided through the inside of the light guide 203 (in the direction parallel to the drawing) as indicated by the arrows. The light beam direction conversion means 204 then emits the light towards the liquid crystal display panel 11, which is arranged approximately parallel to the light guide (in the direction perpendicular to the viewer in the drawing). By optimizing the distribution (density) of this light beam direction conversion means depending on the shape of the inside or surface of the light guide, the uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled. The aforementioned light beam direction conversion means 204, by shaping the surface of the light guide or by providing, for example, a portion with a different refractive index inside the light guide, causes the light beam propagating within the light guide to be emitted toward the liquid crystal display panel 11, which is positioned approximately parallel to the light guide (in a direction perpendicular to the viewer in the drawing). At this time, if the liquid crystal display panel 11 is facing directly toward the center of the screen and the viewpoint is positioned at the same position as the screen diagonal, a relative brightness ratio of 20% or more when comparing the brightness of the center and the periphery of the screen is sufficient for practical use, and exceeding 30% indicates even better performance.

[0073] Figure 13 is a cross-sectional arrangement diagram illustrating the configuration and operation of the light source in this embodiment, which performs polarization conversion, in the light source device 13 including the light guide 203 and LED element 201 described above. In Figure 13, the light source device 13 consists of a light guide 203 with a light beam direction conversion means 204 provided on the surface or inside, which is formed of plastic or the like, an LED element 201 as a light source, a reflective sheet 205, a phase difference plate 206, a lenticular lens, etc., and a liquid crystal display panel 11 equipped with polarizing plates on the light source light incident surface and the image light output surface is attached to its upper surface.

[0074] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13. This selectively reflects one side of the polarization (e.g., P-wave) 212 of the natural light beam 210 emitted from the LED element 201, reflects it off a reflective sheet 205 provided on one side (lower surface in the figure) of the light guide 203, and directs it back towards the liquid crystal display panel 52. Therefore, a phase difference plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, causing the reflected light beam to be reflected by the reflective sheet 205 and passed through twice, thereby converting the reflected light beam from P-polarized to S-polarized, and improving the efficiency of utilizing the light source light as image light. The video light beam, whose light intensity has been modulated by the video signal on the liquid crystal display panel 11 (arrow 213 in Figure 13), enters the retroreflective member 2 and, as shown in Figure 1, is reflected and then transmitted through the window glass 105 to obtain a spatially floating image, which is a real image, inside or outside the store (space).

[0075] Figure 14, similar to Figure 13, is a cross-sectional arrangement diagram illustrating the configuration and operation of the light source in this embodiment that performs polarization conversion, in a light source device 13 including a light guide 203 and an LED element 201. The light source device 13 is similarly composed of a light guide 203 with a light beam direction conversion means 204 provided on the surface or inside, for example, made of plastic, an LED element 201 as a light source, a reflective sheet 205, a phase difference plate 206, a lenticular lens, etc., and a liquid crystal display panel 11 is attached to its upper surface as an image display element, having polarizing plates on the light source light incident surface and the image light emission surface.

[0076] Furthermore, a film or sheet-like reflective polarizing plate 49 is provided on the light source light incident surface (lower surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13. This selectively reflects one side of the polarization (e.g., S-wave) 211 of the natural light beam 210 emitted from the LED light source 201, reflects it off a reflective sheet 205 provided on one side (lower surface in the figure) of the light guide 203, and returns it to the liquid crystal display panel 11. A phase difference plate (λ / 4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, causing the reflected light beam to be reflected by the reflective sheet 205 and passed through twice, thereby converting the reflected light beam from S-polarized to P-polarized, improving the efficiency of utilizing the light source light as image light. The video light beam, whose light intensity is modulated by the video signal on the liquid crystal display panel 11 (arrow 214 in Figure 14), enters the retroreflective member 2 and, as shown in Figure 1, is reflected and then transmitted through the window glass 105 to obtain a spatially floating image, which is a real image, inside or outside the store (space).

[0077] In the light source devices shown in Figures 13 and 14, in addition to the action of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, a reflective polarizer reflects one side of the polarization component. Therefore, the theoretically obtainable contrast ratio is the product of the reciprocal of the cross transmittance of the reflective polarizer and the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In actual experiments, it was confirmed that the contrast performance of the displayed image improved by more than 10 times. As a result, high-quality images comparable to those of self-emissive organic EL displays were obtained.

[0078] <Example of display device 2> Figure 15 shows another example of the specific configuration of the display device 1. The light source device 13 in Figure 15 is similar to the light source device in Figure 17, etc. This light source device 13 is constructed by housing LEDs, a collimator, a composite diffusion block, a light guide, etc., in a case made of, for example, plastic, and a liquid crystal display panel 11 is mounted on its top surface. In addition, an LED substrate 102 on which semiconductor light source LED (Light Emitting Diode) elements 14a, 14b and their control circuit is mounted is attached to one side of the case of the light source device 13, and a heat sink 103, which is a component for cooling the heat generated by the LED elements and control circuit, is attached to the outer surface of the LED substrate 102 (see also Figures 17, 18, etc.).

[0079] Furthermore, the liquid crystal display panel frame attached to the top surface of the case is configured with a liquid crystal display panel 11 mounted on the frame, and an FPC (Flexible Printed Circuits) 403 (see Figure 7) electrically connected to the liquid crystal display panel 11. In other words, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light based on control signals from a control circuit (not shown here) that constitutes the electronic device, together with LED elements 14a and 14b, which are solid light sources.

[0080] <Example of a display device 3> Next, another example of the specific configuration of the display device 1 will be explained using Figure 16. The light source device of this display device 1 converts the divergent luminous flux of natural light (a mixture of P-polarization and S-polarization) from the LEDs into a nearly parallel luminous flux by the LED collimator 18, and reflects it toward the liquid crystal display panel 11 by the reflective light guide 304. The reflected light is incident on a waveplate and a reflective polarizer 49 placed between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarization (e.g., S-polarization) is reflected by the reflective polarizer, its phase is converted by the waveplate, it returns to the reflective surface, passes through the phase difference plate again, and is converted into a polarization (e.g., P-polarization) that transmits through the reflective polarizer.

[0081] As a result, natural light from the LEDs is aligned to a specific polarization (e.g., P polarization), incident on the liquid crystal display panel 11, and its brightness is modulated according to the video signal to display an image on the panel surface. Multiple LEDs constituting the light source are shown as in the example described above (however, only one is shown in Figure 16 because it is a vertical cross-section), and these are mounted at predetermined positions on the LED collimator 18. Each of these LED collimators 18 is made of a translucent resin such as acrylic or glass. The LED collimator 18 has a cone-shaped outer surface obtained by rotating a parabolic cross-section, and at its apex, it has a recess with a convex portion (i.e., a convex lens surface) in the center. In addition, the center of its planar portion has a convex lens surface that protrudes outward (or a concave lens surface that is recessed inward). The parabolic surface forming the cone-shaped outer surface of the LED collimator 18 is set within an angle range in which light emitted from the LEDs in the peripheral direction can be totally reflected inside it, or a reflective surface is formed.

[0082] The above configuration is the same as the light source device of the image display device shown in Figures 17 and 18. Furthermore, the light converted to nearly parallel light by the LED collimator 15 shown in Figure 16 is reflected by the reflective light guide 304, and a reflective polarizer 49 transmits light of a specific polarization. The reflected light of the other polarization passes through the light guide 304 again and is reflected by a reflector 271 provided on the other side of the light guide that does not come into contact with the liquid crystal display panel 11. At this time, the polarization is changed by passing through a phase difference plate (λ / 4 plate) 270 placed between the reflector 271 and the liquid crystal display panel 11 twice, and then passes through the light guide 304 again, passes through the reflective polarizer 49 provided on the opposite side, aligning the polarization direction before being incident on the liquid crystal display panel 11. As a result, all of the light from the light source can be used, so the light utilization efficiency is doubled.

[0083] In conventional TV sets, the light emitted from a liquid crystal display panel has similar diffusion characteristics in both the horizontal direction (shown on the X-axis in Figure 22(a)) and the vertical direction (shown on the Y-axis in Figure 22(b)). In contrast, the diffusion characteristics of the light beam emitted from the liquid crystal display panel in this embodiment are such that, as shown in Example 1 in Figure 22, the viewing angle at which the brightness reaches 50% of the front view (0-degree angle) is 13 degrees, which is 1 / 5 of the conventional 62 degrees. Similarly, the vertical viewing angle is made uneven vertically, and the reflection angle of the reflective light guide and the area of ​​the reflective surface are optimized so that the upper viewing angle is about 1 / 3 of the lower viewing angle. As a result, the amount of image light directed towards the monitoring direction is significantly improved compared to conventional LCD TVs, and the brightness is more than 50 times higher.

[0084] Furthermore, with the viewing angle characteristics shown in Example 2 of Figure 22, the viewing angle at which the brightness is 50% of that of a front view (0-degree angle) is set to 5 degrees, which is 1 / 12 of the conventional 62 degrees. Similarly, the vertical viewing angle is made uniform both vertically and horizontally, and the reflection angle of the reflective light guide and the area of ​​the reflective surface are optimized to reduce the viewing angle to about 1 / 12 of that of conventional systems. As a result, the amount of image light directed towards the monitoring direction is significantly improved compared to conventional LCD TVs, and the brightness is more than 100 times higher. As described above, by making the viewing angle narrow, the amount of light flux directed towards the monitoring direction can be concentrated, so the efficiency of light utilization is greatly improved. As a result, even when using a conventional LCD display panel for TVs, a significant improvement in brightness can be achieved with similar power consumption by controlling the light diffusion characteristics of the light source device, making it possible to create an image display device that is suitable for information display systems directed towards bright outdoor environments.

[0085] When using a large LCD display panel, the overall brightness of the screen can be improved by directing the light from the edges of the screen inward so that it is directed towards the monitor when the monitor is facing the center of the screen. Figure 20 shows the convergence angles of the long and short sides of the panel, with the monitor's distance L from the panel and the panel size (screen aspect ratio 16:10) as parameters. When monitoring with the screen in portrait orientation, the convergence angle should be set to match the short side. For example, with a 22" panel used vertically and a monitoring distance of 0.8m, setting the convergence angle to 10 degrees will effectively direct the video light from the four corners of the screen towards the monitor.

[0086] Similarly, when monitoring with a 15" panel in portrait orientation, if the monitoring distance is 0.8m, a convergence angle of 7 degrees allows the image light from all four corners of the screen to be effectively directed towards the monitor. As described above, by directing the image light from the periphery of the screen towards the monitor who is in the optimal position to monitor the center of the screen, the overall brightness of the screen can be improved depending on the size of the LCD display panel and whether it is used in portrait or landscape orientation.

[0087] In its basic configuration, as shown in Figure 16, a light source device emits a luminous beam with narrow-angle directional characteristics onto the liquid crystal display panel 11. By modulating the brightness in accordance with the video signal, the video information displayed on the screen of the liquid crystal display panel 11 is reflected by a retroreflective member, and the resulting floating image is displayed outdoors or indoors via a transparent member 100.

[0088] <Example of a light source device 1> Next, the configuration of the optical system, including the light source device housed within the case, will be explained in detail with reference to Figure 17 and Figures 18(a) and (b).

[0089] Figures 17 and 18 show the LEDs 14a and 14b that constitute the light source, which are mounted in predetermined positions on the LED collimator 15. The LED collimator 15 is made of a translucent resin such as acrylic. As shown in Figure 18(b), the LED collimator 15 has a cone-shaped outer surface 156 obtained by rotating a parabolic cross section, and at its apex, it has a recess 153 with a convex portion (i.e., a convex lens surface) 157 formed in the center. In addition, the center of its planar portion has a convex lens surface (or a concave lens surface) 154 that protrudes outward (or is recessed inward). The parabolic surface 156 that forms the cone-shaped outer surface of the LED collimator 15 is set within an angle range in which light emitted from the LEDs 14a and 14b in the peripheral direction can be totally reflected inside it, or a reflective surface is formed thereon.

[0090] Furthermore, LEDs 14a and 14b are positioned at predetermined locations on the surface of the LED substrate 102, which is the circuit board for the LEDs. The LED substrate 102 is fixed to the LED collimator 15 such that LEDs 14a or 14b on its surface are positioned in the center of the recess 153.

[0091] With this configuration, the light emitted from LED 14a or 14b, particularly the light emitted upward (to the right in the diagram) from its central portion, is focused by the two convex lens surfaces 157 and 154 that form the outer shape of the LED collimator 15, becoming parallel light. Similarly, the light emitted from other parts toward the periphery is reflected by the parabolic surface forming the conical outer surface of the LED collimator 15, and is also focused to become parallel light. In other words, with an LED collimator 15 having a convex lens in its center and a parabolic surface around its periphery, it becomes possible to extract almost all of the light generated by LED 14a or 14b as parallel light, thereby improving the utilization efficiency of the generated light.

[0092] Furthermore, a polarization conversion element 21 is provided on the light output side of the LED collimator 15. As is clear from Figure 18, this polarization conversion element 21 is constructed by combining a translucent member with a parallelogram cross-section (hereinafter referred to as a parallelogram prism) and a translucent member with a triangular cross-section (hereinafter referred to as a triangular prism), and arranging multiple such elements in an array parallel to a plane perpendicular to the optical axis of the parallel light from the LED collimator 15. In addition, polarizing beam splitters (hereinafter abbreviated as "PBS film") 211 and reflective films 212 are alternately provided at the interfaces between adjacent translucent members arranged in this array, and a λ / 2 phase plate 213 is provided on the output surface from which light incident on the polarization conversion element 21 and transmitted through the PBS film 211 is emitted.

[0093] The emission surface of this polarization conversion element 21 is further provided with a rectangular composite diffusion block 16, as shown in Figure 18(a). That is, the light emitted from LED 14a or 14b becomes parallel light due to the action of the LED collimator 15, enters the composite diffusion block 16, is diffused by the emission-side texture 161, and then reaches the light guide 17.

[0094] The light guide 17 is a rod-shaped member formed from a translucent resin such as acrylic, with a roughly triangular cross-section (see Figure 18(b)). As is clear from Figure 17, it comprises a light guide light incident portion (surface) 171 facing the emission surface of the composite diffusion block 16 via a first diffuser plate 18a, a light guide light reflecting portion (surface) 172 forming a slope, and a light guide light emission portion (surface) 173 facing the liquid crystal display panel 11, which is a liquid crystal display element, via a second diffuser plate 18b.

[0095] As shown in Figure 17, a magnified view of the light guide 17, the light guide's light-reflecting portion (surface) 172 has numerous reflective surfaces 172a and connecting surfaces 172b alternately formed in a sawtooth pattern. The reflective surfaces 172a (sloping line segments in the figure) form αn (n is a natural number, for example, 1 to 130) with respect to the horizontal plane shown by the dashed line in the figure. As an example, αn is set to 43 degrees or less (but greater than or equal to 0 degrees).

[0096] The light guide's light incident portion (surface) 171 is formed in a curved convex shape that is inclined toward the light source. As a result, parallel light from the output surface of the composite diffusion block 16 is diffused and incident via the first diffusion plate 18a, and as is clear from the figure, it is slightly bent (deflected) upward by the light guide's light incident portion (surface) 171 as it reaches the light guide's light reflecting portion (surface) 172, where it is reflected and reaches the liquid crystal display panel 11 provided on the upper output surface in the figure.

[0097] As described in detail above, the display device 1 improves light utilization efficiency and its uniform illumination characteristics, while also enabling the manufacture of a compact and low-cost device, including a modularized S-polarized wave light source. In the above description, the polarization conversion element 21 was described as being installed after the LED collimator 15, but the present invention is not limited to this, and similar effects and benefits can be obtained by providing it in the optical path leading to the liquid crystal display panel 11.

[0098] Furthermore, the light guide's light-reflecting section (surface) 172 has numerous reflective surfaces 172a and connecting surfaces 172b alternately formed in a sawtooth pattern. The illumination luminous beam is totally reflected on each reflective surface 172a and directed upward. In addition, a narrow-angle diffuser plate is provided on the light guide's light-emitting section (surface) 173 to control the directional characteristics as a substantially parallel diffused luminous beam, which is then incident on the light direction conversion panel 54 and incident on the liquid crystal display panel 11 from an oblique direction. In this embodiment, the light direction conversion panel 54 is provided between the light guide's light-emitting section (surface) 173 and the liquid crystal display panel 11, but the same effect can be obtained by providing it on the light-emitting surface of the liquid crystal display panel 11.

[0099] <Example of a light source device 2> Figure 19 shows another example of the optical system configuration of the light source device 13. Similar to the example shown in Figure 18, multiple (two in this example) LEDs 14a and 14b constituting the light source are shown, and these are attached to the LED collimator 15 at predetermined positions. Each of these LED collimators 15 is formed from a translucent resin such as acrylic. Similar to the example shown in Figure 18, the LED collimator 15 has a cone-shaped outer surface 156 obtained by rotating a parabolic cross-section, and at its apex, it has a recess 153 with a convex portion (i.e., a convex lens surface) 157 formed in the center. In addition, the center of its planar portion has a convex lens surface (or a concave lens surface that is recessed inward) 154 that protrudes outward. The parabolic surface 156 that forms the cone-shaped outer surface of the LED collimator 15 is set within an angle range in which light emitted from the LED 14a in the peripheral direction can be totally reflected inside it, or a reflective surface is formed thereon.

[0100] Furthermore, LEDs 14a and 14b are positioned at predetermined locations on the surface of the LED substrate 102, which is the circuit board for the LEDs. The LED substrate 102 is fixed to the LED collimator 15 such that LEDs 14a or 14b on its surface are positioned in the center of the recess 153.

[0101] With this configuration, the light emitted from LED 14a or 14b, particularly the light emitted upward (to the right in the diagram) from its central portion, is focused by the two convex lens surfaces 157 and 154 that form the outer shape of the LED collimator 15, becoming parallel light. Similarly, the light emitted from other parts toward the periphery is reflected by the parabolic surface forming the conical outer surface of the LED collimator 15, and is also focused to become parallel light. In other words, with an LED collimator 15 having a convex lens in its center and a parabolic surface around its periphery, it becomes possible to extract almost all of the light generated by LED 14a or 14b as parallel light, thereby improving the utilization efficiency of the generated light.

[0102] Furthermore, a light guide 170 is provided on the light output side of the LED collimator 15 via a first diffuser plate 18a. The light guide 170 is a rod-shaped member formed from a translucent resin such as acrylic, with a roughly triangular cross-section (see Figure 19(a)). As is clear from Figure 19(a), it comprises a light incident portion (surface) 171 facing the output surface of the diffusion block 16 via the first diffuser plate 18a, a light reflection portion (surface) 172 forming a slope, and a light output portion (surface) 173 facing the liquid crystal display panel 11, which is a liquid crystal display element, via a reflective polarizing plate 200.

[0103] If a reflective polarizer 200 is selected that, for example, reflects P-polarized light (and transmits S-polarized light), it will reflect P-polarized light from the natural light emitted from the LED light source, pass through the λ / 4 plate 202 provided in the light guide light reflecting section 172 shown in Figure 19(b), be reflected by the reflective surface 201, and pass through the λ / 4 plate 202 again to be converted to S-polarized light. As a result, all the light beam incident on the liquid crystal display panel 11 will be unified into S-polarized light.

[0104] Similarly, if a reflective polarizer 200 is selected that has the characteristic of reflecting S-polarized light (and transmitting P-polarized light), the S-polarized light from the natural light emitted from the LED light source will be reflected, pass through the λ / 4 plate 202 provided in the light guide light reflecting section 172 shown in Figure 19(b), be reflected by the reflective surface 201, and then pass through the λ / 4 plate 202 again to be converted to P-polarized light. As a result, all the light beam incident on the liquid crystal display panel 52 will be unified to P-polarized light. Polarization conversion can also be achieved with the configuration described above.

[0105] <Example of a light source device 3> Another example of the optical system configuration for a light source device will be explained using Figure 16. In the third example, as shown in Figure 16, the divergent luminous flux of natural light (a mixture of P-polarized and S-polarized light) from the LED 102 is converted into a nearly parallel luminous flux by the collimator lens 18 and reflected toward the liquid crystal display panel 11 by the reflective light guide 304. The reflected light is incident on a reflective polarizer 206 placed between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarization (e.g., S-polarization) is reflected by the reflective polarizer 206, passes through the surface connecting the reflective surfaces of the light guide 304, is reflected by a reflector 271 placed facing the opposite side of the light guide 304, and is polarized by passing through a phase plate (λ / 4 wave plate) 270 twice, passing through the light guide and the reflective polarizer before being incident on the liquid crystal display panel 11 and modulated into image light. At this time, by aligning the specific polarization with the polarization-converted polarization plane, the light utilization efficiency becomes twice that of normal, and the polarization degree (extinction ratio) of the reflective polarizer is also incorporated into the extinction ratio of the entire system. Therefore, by using the light source device of this embodiment, the contrast ratio of the information display system is greatly improved.

[0106] As a result, natural light from the LEDs is aligned to a specific polarization (e.g., P polarization). Multiple LEDs constituting the light source are provided, similar to the example described above (however, only one is shown in Figure 16 due to the vertical cross-section), and these are mounted at predetermined positions relative to the LED collimator 18. Each LED collimator 18 is formed from a translucent resin such as acrylic or glass. The LED collimator 18 has a cone-shaped outer surface obtained by rotating a parabolic cross-section, and at its apex, it has a recess with a convex portion (i.e., a convex lens surface) in its center. In addition, the center of its planar portion has a convex lens surface that protrudes outward (or a concave lens surface that is recessed inward). The parabolic surface forming the cone-shaped outer surface of the LED collimator 18 is set within an angle range in which light emitted from the LED 18 in the peripheral direction can be totally reflected inside it, or a reflective surface is formed.

[0107] Furthermore, the LEDs are each positioned at predetermined locations on the surface of the circuit board, the LED substrate 102. The LED substrate 102 is fixed to the LED collimator 18 such that the LEDs on its surface are each positioned in the center of their respective recesses.

[0108] With this configuration, the light emitted from the LED, particularly the light emitted from its central portion, is focused by the two convex lens surfaces that form the outer shape of the LED collimator 18, becoming parallel light. Similarly, the light emitted from other parts toward the periphery is reflected by the parabolic surface that forms the conical outer surface of the LED collimator 18, and is also focused to become parallel light. In other words, with an LED collimator 18 having a convex lens in its center and a parabolic surface around its periphery, it becomes possible to extract almost all of the light generated by the LED as parallel light, thereby improving the utilization efficiency of the generated light.

[0109] <Example of a light source device 4> Furthermore, another example of the optical system configuration for a light source device will be explained using Figure 25. Two optical sheets 207 are used on the light output side of the LED collimator 18 to convert the diffusion characteristics in the vertical and horizontal directions (front and back directions, not shown in the figure), and the light from the LED collimator 18 is incident between the two optical sheets 207 (diffusion sheets). If the optical sheet 207 is made of a single sheet, the vertical and horizontal diffusion characteristics are controlled by the fine shape of the front and back surfaces. Alternatively, multiple diffusion sheets may be used to share the function. By adjusting the surface and back surface shapes of the optical sheet 207, the diffusion angle of the light from the LED collimator 18 in the vertical direction of the screen should be matched to the width of the vertical surface of the reflective surface of the diffusion sheet, and in the horizontal direction, the surface density of the light beam emitted from the liquid crystal display panel 11 should be uniform. The number of LEDs and the divergence angle from the LED substrate (optical element) 102 should be optimally designed as design parameters. In other words, the diffusion characteristics are controlled by the surface shapes of multiple diffusion sheets instead of a light guide. In this embodiment, polarization conversion is performed in the same way as in Example 3 of the light source device described above. Alternatively, a polarization conversion element 21 may be placed between the LED collimator 18 and the diffusion film 207 to perform polarization conversion before the light source is incident on the diffusion sheet 207.

[0110] If the aforementioned reflective polarizer 206 is selected to have the characteristic of reflecting S-polarized light (and transmitting P-polarized light), it will reflect S-polarized light from the natural light emitted from the LED light source, pass through the phase difference plate 270 shown in Figure 25, be reflected by the reflective surface 271, and pass through the phase difference plate 270 again to be converted to P-polarized light before being incident on the liquid crystal display panel 11. The thickness of this phase difference plate needs to be selected to an optimal value depending on the angle of incidence of the light rays onto the phase difference plate, and the optimal value exists in the range of λ / 16 to λ / 4.

[0111] <Lenticular lenses> To control the diffusion distribution of image light from the liquid crystal display panel 11, the unidirectional emission characteristics can be controlled by optimizing the lens shape by providing a lenticular lens between the light source device 13 and the liquid crystal display panel 11, or on the surface of the liquid crystal display panel 11. Furthermore, by arranging a microlens array in a matrix, the emission characteristics of the image light beam from the display device 1 can be controlled in the X and Y axes, resulting in an image display device with desired diffusion characteristics.

[0112] The function of lenticular lenses will now be explained. By optimizing the lens shape, lenticular lenses can efficiently obtain a floating image in space by transmitting or reflecting light emitted from the display device 1 through the transparent member 100. Specifically, by combining two lenticular lenses or arranging a microlens array in a matrix to control the diffusion characteristics of the image light from the display device 1, the brightness (relative brightness) of the image light can be controlled in the X and Y axes according to its reflection angle (0 degrees in the vertical direction). In this embodiment, by using such a lenticular lens, the vertical luminance characteristics are made steeper compared to conventional methods, as shown in Figure 22(b). Furthermore, by changing the balance of the directional characteristics in the up and down directions (positive and negative directions of the Y axis), the luminance of light due to reflection and diffusion (relative luminance) is increased. This results in image light with a narrow diffusion angle (high directivity) and containing only specific polarization components, similar to image light from a surface-emitting laser image source. This suppresses ghost images that were generated by retroreflective members when using conventional image display devices, and allows for efficient control so that a spatially floating image due to retroreflection reaches the observer's eye.

[0113] Furthermore, the aforementioned light source device enables a significantly narrower directional characteristic in both the X and Y axes compared to the typical light diffusion characteristics of a liquid crystal display panel shown in Figures 22(a) and (b) (indicated as "conventional" in the figure). This allows for the realization of a video display device that emits light with a specific polarization, emitting an image light beam nearly parallel to a specific direction.

[0114] Figure 21 shows an example of the characteristics of the lenticular lens used in this embodiment. In this example, the characteristics in the X direction (vertical direction) are shown in particular. Characteristic O shows a symmetrical brightness characteristic with the peak of the light emission direction at an angle of approximately 30 degrees upward from the vertical (0 degrees). Furthermore, characteristics A and B in Figure 21 show examples of characteristics in which the image light above the peak brightness at approximately 30 degrees is collected to increase the brightness (relative brightness). As a result, in characteristics A and B, the brightness (relative brightness) of the light decreases sharply at angles exceeding 30 degrees compared to characteristic O.

[0115] In other words, with the optical system including the lenticular lens described above, when the image light beam from the display device 1 is incident on the retroreflective member 2, the emission angle and viewing angle of the image light aligned to a narrow angle by the light source device 13 can be controlled, significantly improving the freedom of installation of the retroreflective sheet (retroreflective member 2). As a result, the degree of freedom in the relationship of the image formation position of the floating image that is reflected or transmitted through the transparent member 100 and formed at a desired position can be significantly improved. As a result, it becomes possible to efficiently deliver light with a narrow diffusion angle (high directivity) and only specific polarization components to the eyes of an observer outside or inside the room. This means that even if the intensity (brightness) of the image light from the image display device is reduced, the observer can accurately perceive the image light and obtain information. In other words, by reducing the output of the image display device, it becomes possible to realize a floating image display device with low power consumption.

[0116] <Touch operation assistance function> Next, we will explain the touch operation assistance functions for the user. First, we will explain touch operation when no assistance functions are provided. Here, we will explain using the example of a user selecting and touching one of two buttons (objects), but the following content can be suitably applied to, for example, ATMs in banks, ticket vending machines in train stations, digital signage, etc.

[0117] Figure 26 illustrates an example of the display and touch operation of the floating image display device 1000. The floating image 3 shown in Figure 26 includes a first button BUT1 displaying "YES" and a second button BUT2 displaying "NO". The user selects "YES" or "NO" by moving their finger 210 toward the floating image 3 and touching either the first button BUT1 or the second button BUT2. In the examples in Figures 26 and 27-29, the first button BUT1 and the second button BUT2 are displayed in different colors. Here, areas of the floating image 3 other than the first button BUT1 and the second button BUT2 may be made transparent without displaying an image, but in that case, only the button areas (the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2) will be displayed, and the range affected by the virtual shadow effect described later will be visible. Therefore, in the following explanation, as a more preferred example, it is assumed that in the area of ​​the floating spatial image 3 other than the first button BUT1 and the second button BUT2, a wider area including the display area of ​​the first button BUT1 and the display area of ​​the second button BUT2 is displayed, with an image of a different color or brightness than that of the first button BUT1 and the second button BUT2.

[0118] In a typical touch-panel video display device that is not a floating-image display device, the buttons selected by the user consist of image buttons displayed on the touch panel surface. Therefore, by visually observing the touch panel surface, the user can perceive the distance between the object (e.g., a button) displayed on the touch panel surface and their own finger. However, in a floating-image display device, because the floating-image 3 is suspended in the air, it may not be easy for the user to perceive the depth of the floating-image 3. Therefore, when performing touch operations on the floating-image 3, it may not be easy for the user to perceive the distance between the button displayed on the floating-image 3 and their own finger. Furthermore, in a typical touch-panel video display device that is not a floating-image display device, the user can easily determine whether or not they have touched a button by the feel of the touch. However, when performing touch operations on the floating-image 3, there is no tactile sensation when touching an object (e.g., a button), so the user may not be able to determine whether or not they have touched an object. Considering the above circumstances, this embodiment provides a function to assist the user with touch operations.

[0119] The following explanation describes processing based on the user's finger position, but the specific method for detecting the user's finger position will be described later.

[0120] <<Assisting touch operations using virtual shadows (1)>> Figures 27 to 29 illustrate an example of a method for assisting touch operation using a virtual shadow. In the example shown in Figures 27 to 29, the user touches the first button BUT1 to select "YES". The floating image display device 1000 of this embodiment assists the user's touch operation by displaying a virtual shadow on the floating image 3. Here, "displaying a virtual shadow on the floating image 3" is an image display process that makes it appear as if a shadow is projected onto the image by reducing the brightness of the video signal in a part of the image that mimics the shape of a finger. Specifically, this process can be performed by calculations in the image control unit 1160 or control unit 1110. In the virtual shadow display process, the brightness of the video signal may be completely set to 0 in a part of the shape that mimics the shape of a finger. However, it is preferable to display the image with reduced brightness in a part of the image that mimics the shape of a finger, rather than completely setting the brightness of the video signal to 0 in that part of the image, as this is perceived as a shadow more naturally. In this case, during the virtual shadow display process, not only may the brightness of the video signal be reduced for a portion of the shape resembling a finger, but the saturation of the video signal may also be reduced.

[0121] The floating image 3 exists in the air where there is no physical contact surface, and in a normal environment, the shadow of a finger would inevitably be projected onto it. However, according to the virtual shadow display processing of this embodiment, even in the air where the shadow of a finger would not normally be projected, it is possible to make it appear as if a shadow exists within the floating image 3, thereby improving the user's perception of the depth of the floating image 3 and enhancing the sense of its real presence.

[0122] Figure 27 shows the state at the first point in time when the user attempts to touch the first button BUT1 on the display surface 3a of the floating image 3 with their finger 210, Figure 28 shows the state at the second point in time when the finger 210 is closer to the floating image 3 than in Figure 27, and Figure 29 shows the state at the third point in time when the finger 210 has touched the first button BUT1 on the display surface 3a of the floating image 3. Furthermore, (A) in Figures 27 to 29 shows the state of the display surface 3a of the floating image 3 when viewed from the front (normal direction of the display surface 3a), and (B) in Figures 27 to 29 shows the state of the display surface 3a of the floating image 3 when viewed from the side (direction parallel to the display surface 3a). In Figures 27 to 29, the x-direction is the horizontal direction on the display surface 3a of the floating image 3, the y-direction is the direction perpendicular to the x-axis within the display surface 3a of the floating image 3, and the z-direction is the normal direction of the display surface 3a of the floating image 3 (the height direction relative to the display surface 3a). In the explanatory diagrams of Figures 27 to 33, the floating image 3 is depicted as having thickness in the depth direction for ease of explanation, but in reality, if the image display surface of the display device 1 is flat, the floating image 3 is also flat and has no thickness in the depth direction. In this case, the floating image 3 and the display surface 3a are on the same plane. In the description of this embodiment, the display surface 3a means the surface on which the floating image 3 can be displayed, and the floating image 3 means the part on which the floating image is actually displayed.

[0123] In Figures 27, 28, and 29, the finger 210 detection process is performed, for example, using the captured image generated by the imaging unit 1180 and the sensing signal from the aerial operation detection sensor 1351. In the finger 210 detection process, for example, the position (x and y coordinates) of the tip of the finger 210 on the display surface 3a of the floating spatial image 3, and the height position (z coordinate) of the tip of the finger 210 relative to the display surface 3a are detected. Here, the position (x and y coordinates) of the tip of the finger 210 on the display surface 3a of the floating spatial image 3 is the position coordinate on the display surface 3a of the intersection of the perpendicular line from the tip of the finger 210 to the display surface 3a of the floating spatial image 3. The height position of the tip of the finger 210 relative to the display surface 3a is also depth information representing the depth of the finger 210 relative to the display surface 3a. The arrangement of the imaging unit 1180 and the aerial operation detection sensor 1351, which perform detection of the finger 210, will be explained in detail later.

[0124] At the first time point shown in Figure 27, the finger 210 is assumed to be at its furthest point from the display surface 3a of the floating image 3 compared to the second time point shown in Figure 28 and the third time point shown in Figure 28. The distance (height position) between the tip of the finger 210 and the display surface 3a of the floating image 3 at this time is denoted as dz1. In other words, the distance dz1 represents the height of the finger 210 relative to the display surface 3a of the floating image 3 in the z direction.

[0125] Note that the distance dz1 shown in Figure 27 and the distance dz2 shown in Figure 28 (described later) are defined with respect to the display surface 3a of the floating spatial image 3 as positive on the user side and negative on the side opposite the user to the display surface 3a. That is, if the finger 210 is on the user side of the display surface 3a, the distances dz1 and dz2 will be positive values, and if the finger 210 is on the opposite side of the display surface 3a from the user, the distances dz1 and dz2 will be negative values.

[0126] In this embodiment, it is assumed that the virtual light source 1500 is located on the user side relative to the display surface 3a of the floating spatial image 3. Here, the setting of the installation direction of the virtual light source 1500 may actually be stored as information in the non-volatile memory 1108 or memory 1109 of the floating spatial image display device 1000. Alternatively, the setting of the installation direction of the virtual light source 1500 may be a parameter that exists only in the design. Even if the setting of the installation direction of the virtual light source 1500 is a parameter that exists only in the design, the design installation direction of the virtual light source 1500 is uniquely determined from the relationship between the position of the user's finger and the display position of the virtual shadow, which will be described later. Here, in the example of Figures 27 to 29, the virtual light source 1500 is located on the user side relative to the display surface 3a, and is provided to the right of the display surface 3a as seen from the user. Then, a virtual shadow 1510 that mimics the shadow of the finger 210 formed by the light emitted from the virtual light source 1500 is displayed on the floating spatial image 3. In the examples shown in Figures 27-29, the virtual shadow 1510 is displayed to the left of the finger 210. This virtual shadow 1510 assists the user with touch operations.

[0127] In the state shown in Figure 27(B), the tip of the finger 210 is furthest away in the normal direction from the display surface 3a of the floating spatial image 3 compared to the states shown in Figure 28(B) and Figure 29(B). Therefore, in Figure 27(A), the tip of the virtual shadow 1510 is formed at the position furthest horizontally from the first button BUT1 that is to be touched, compared to the states shown in Figure 28(A) and Figure 29(A). Consequently, in Figure 27(A), the horizontal distance between the tip of the finger 210 and the tip of the virtual shadow 1510 when viewing the display surface 3a of the floating spatial image 3 from the front is the largest compared to the states shown in Figure 28(A) and Figure 29(A). In Figure 27(A), dx1 is defined as the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating spatial image 3.

[0128] In Figure 28(B), the finger 210 is closer to the floating image 3 than in Figure 27(B). Therefore, in Figure 28(B), the normal distance dz2 between the tip of the finger 210 and the display surface 3a of the floating image 3 is smaller than dz1. In this case, in Figure 28(A), the virtual shadow 1510 is displayed at a position where the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating image 3 is dx2, which is smaller than dx1. That is, in the example of Figure 28, since the virtual light source 1500 is located on the user side of the display surface 3a and to the right of the display surface 3a from the user's perspective, the horizontal distance between the tip of the finger 210 and the tip of the virtual shadow 1510 when viewing the display surface 3a of the floating image 3 from the front changes in conjunction with the normal distance between the tip of the finger 210 and the display surface 3a of the floating image 3.

[0129] Then, when the tip of finger 210 and the tip of virtual shadow 1510 come into contact, the distance in the normal direction between the tip of finger 210 and the display surface 3a of the floating spatial image 3 becomes 0, as shown in Figure 29. At this time, the virtual shadow 1510 is displayed such that the distance between finger 210 and the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating spatial image 3 becomes 0. As a result, the user can recognize that finger 210 has touched the display surface 3a of the floating spatial image 3. At this time, if the tip of finger 210 is touching the area of ​​the first button BUT1, the user can recognize that they have touched the first button BUT1. In other words, even in the example in Figure 29, since the virtual light source 1500 is located on the user side of the display surface 3a and to the right of the display surface 3a from the user's perspective, the horizontal distance between the tip of the finger 210 and the tip of the virtual shadow 1510, when viewed from the front of the display surface 3a of the floating spatial image 3, changes in conjunction with the normal distance between the tip of the finger 210 and the display surface 3a of the floating spatial image 3. That is, the display position of the tip of the virtual shadow 1510 is determined by the positional relationship between the position of the virtual light source 1500 and the position of the tip of the user's finger 210, and changes in conjunction with the change in the position of the tip of the user's finger 210.

[0130] According to the configuration and processing of "Assistance for touch operation using virtual shadows (1)" described above, during touch operation, the user can more effectively recognize the distance (depth) in the normal direction between the finger 210 and the display surface 3a of the floating image 3, based on the horizontal positional relationship between the finger 210 and the virtual shadow 1510 on the display surface 3a of the floating image 3. Furthermore, if the finger 210 touches an object (e.g., a button) which is the floating image 3, the user can recognize that they have touched the object. This makes it possible to provide a more suitable floating image display device.

[0131] <<Assisting touch operations using virtual shadows (2)>> Next, as another example of a method to assist touch operation using virtual shadows, we will describe the case where the virtual light source 1500 is located to the left of the display surface 3a as seen from the user's perspective. Figures 30 to 32 illustrate another example of a method to assist touch operation using virtual shadows. Figure 30 corresponds to Figure 27 and shows the state at the first point in time when the user attempts to touch the first button BUT1 on the display surface 3a of the floating spatial image 3 with their finger 210. Figure 31 corresponds to Figure 28 and shows the state at the second point in time when the finger 210 is closer to the floating spatial image 3 than in Figure 30. Figure 32 corresponds to Figure 29 and shows the state when the finger 210 has touched the floating spatial image 3. Note that in Figures 30 to 32 (B), for the sake of explanation, the view is shown from the opposite direction to Figures 27 to 29 (B).

[0132] In Figures 30 to 32, the virtual light source 1500 is located on the user side of the display surface 3a, specifically to the left of the display surface 3a from the user's perspective. A virtual shadow 1510, which mimics the shadow of the finger 210 formed by the light emitted from the virtual light source 1500, is displayed on the floating image 3. In Figures 30 to 32, the virtual shadow 1510 is displayed to the right of the finger 210. This virtual shadow 1510 assists the user with touch operations.

[0133] In the state shown in Figure 30(B), the tip of the finger 210 is furthest from the display surface 3a of the floating spatial image 3 in the normal direction, compared to the states shown in Figures 31(B) and 32(B). In Figure 30(B), the distance in the normal direction between the tip of the finger 210 and the display surface 3a of the floating spatial image 3 is dz10. Also, in Figure 30(A), the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating spatial image 3 is dx10.

[0134] In Figure 31(B), the finger 210 is closer to the floating image 3 than in Figure 27(B). Therefore, in Figure 31(B), the normal distance dz20 between the tip of the finger 210 and the display surface 3a of the floating image 3 is smaller than dz10. In this case, in Figure 31(A), the virtual shadow 1510 is displayed at a position where the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating image 3 is dx20, which is smaller than dx10. That is, in the example of Figure 31, since the virtual light source 1500 is located on the user side of the display surface 3a and to the left of the display surface 3a from the user's perspective, the horizontal distance between the tip of the finger 210 and the tip of the virtual shadow 1510 when the display surface 3a of the floating image 3 is viewed from the front changes in conjunction with the normal distance between the tip of the finger 210 and the display surface 3a of the floating image 3.

[0135] Then, when the tip of finger 210 and the tip of virtual shadow 1510 come into contact, the distance in the normal direction between the tip of finger 210 and the display surface 3a of the floating spatial image 3 becomes 0, as shown in Figure 32. At this time, the virtual shadow 1510 is displayed such that the distance between finger 210 and the virtual shadow 1510 in the horizontal direction of the display surface 3a of the floating spatial image 3 becomes 0. As a result, the user can recognize that finger 210 has touched the display surface 3a of the floating spatial image 3. At this time, if the tip of finger 210 is touching the area of ​​the first button BUT1, the user can recognize that they have touched the first button BUT1. In other words, even in the example in Figure 32, since the virtual light source 1500 is located on the user side of the display surface 3a and to the left of the display surface 3a from the user's perspective, the horizontal distance between the tip of the finger 210 and the tip of the virtual shadow 1510, when the display surface 3a of the floating spatial image 3 is viewed from the front, changes in conjunction with the normal distance between the tip of the finger 210 and the display surface 3a of the floating spatial image 3.

[0136] The configuration and processing described above for "Assisting touch operations using virtual shadows (2)" also yield the same effects as the configurations shown in Figures 27 to 29.

[0137] Here, if the spatial floating image display device 1000 implements the processing described above as "assistance with touch operation using virtual shadows (1)" and / or "assistance with touch operation using virtual shadows (2)", there are several possible implementation examples as follows.

[0138] As a first implementation example, the floating spatial image display device 1000 is equipped with only the "assistance to touch operation using virtual shadows (1)". In this case, the virtual light source 1500 is located on the user side of the display surface 3a, to the right of the display surface 3a from the user's perspective, so the virtual shadow 1510 is displayed to the left of the tip of the user's finger 210 from the user's perspective. Therefore, if the user's finger 210 is a finger of the right hand, the visibility of the virtual shadow 1510 is not obstructed by the user's right hand or right arm, which is preferable. Thus, given the statistical tendency for right-handed users to be more numerous, even if only the "assistance to touch operation using virtual shadows (1)" is implemented in the floating spatial image display device 1000, the probability that the virtual shadow 1510 will be clearly visible is sufficiently high, making it preferable.

[0139] Furthermore, as a first implementation example, both the "Touch operation assistance using virtual shadow (1)" process and the "Touch operation assistance using virtual shadow (2)" process may be implemented, and the system may be configured to switch which process is performed depending on whether the user uses their right or left hand to perform the touch operation. In this case, the probability of the virtual shadow 1510 being clearly visible can be further increased, improving user convenience.

[0140] Specifically, when the user is performing a touch operation with their right hand, the virtual shadow 1510 is displayed to the left of the finger 210 using the configuration shown in Figures 27 to 29. In this case, the visibility of the virtual shadow 1510 is favorable as it is not obstructed by the user's right hand or arm. On the other hand, when the user is performing a touch operation with their left hand, the virtual shadow 1510 is displayed to the right of the finger 210 using the configuration shown in Figures 30 to 32. In this case, the visibility of the virtual shadow 1510 is favorable as it is not obstructed by the user's left hand or arm. As a result, whether the user is performing a touch operation with their right hand or left hand, the virtual shadow 1510 is displayed in a position that is easily visible to the user, improving user convenience.

[0141] Here, the determination of whether the user is performing a touch operation with their right or left hand can be made, for example, based on the captured image generated by the imaging unit 1180. For example, the control unit 1110 performs image processing on the captured image and detects the user's face, arms, hands, and fingers from the captured image. The imaging unit 1180 then estimates the user's posture or movement from the arrangement of these detected parts (face, arms, hands, and fingers) and determines whether the user is performing a touch operation with their right or left hand. In this determination, if the vicinity of the center of the user's body in the left-right direction can be determined from other parts, it is not necessarily required to capture the face. Alternatively, the above determination may be made based only on the arrangement of the arms. Alternatively, the above determination may be made based only on the arrangement of the hands. Alternatively, the above determination may be made based on a combination of the arrangement of the arms and hands. Furthermore, the arrangement of the face may be combined with these determinations.

[0142] Figures 27-29 and 30-32 show a virtual shadow 1510 extending at an angle corresponding to the actual extension direction of the finger 210. The actual extension direction of the finger 210 can be calculated by imaging the finger with one of the imaging units described earlier. Alternatively, the virtual shadow 1510 may be displayed with its extension direction fixed at a predetermined angle, without reflecting the angle corresponding to the extension direction of the finger 210. This reduces the load on the video control unit 1160 or control unit 1110 that controls the display of the virtual shadow 1510.

[0143] For example, if finger 210 is a finger of the right hand, it is natural for the user to extend their arm from the front right side of the display surface 3a of the floating spatial image 3 and attempt to touch the display surface 3a of the floating spatial image 3 with finger 210 pointing to the upper left towards the display surface 3a of the floating spatial image 3. Therefore, if finger 210 is a finger of the right hand, configuring the virtual shadow 1510 to display the shadow of the finger in a predetermined direction indicating the upper right towards the display surface 3a of the floating spatial image 3 will result in a natural display without reflecting the angle corresponding to finger 210.

[0144] Furthermore, for example, if finger 210 is a finger of the left hand, it is natural for the user to extend their arm from the left front of the display surface 3a of the floating spatial image 3 and attempt to touch the display surface 3a of the floating spatial image 3 with finger 210 pointing to the upper right toward the display surface 3a of the floating spatial image 3. Therefore, if finger 210 is a finger of the left hand, configuring the virtual shadow 1510 to display the shadow of the finger in a predetermined direction indicating the upper left toward the display surface 3a of the floating spatial image 3 will result in a natural display without reflecting the angle corresponding to finger 210.

[0145] Furthermore, if the user's finger 210 is on the opposite side of the display surface 3a of the floating image 3 from the user, it is sufficient to display a message that allows the user to recognize that the finger 210 is behind the floating image 3 and cannot be touched. For example, a message informing the user that the finger 210 is behind the floating image 3 and cannot be touched may be displayed on the floating image 3. Alternatively, the virtual shadow 1510 may be displayed in a different color than usual, such as red. This makes it possible to more effectively prompt the user to return the finger 210 to the appropriate position.

[0146] <<Example of virtual light source settings>> Here, we will explain how to set up the virtual light source 1500. Figure 33 is a diagram illustrating how to set up the virtual light source. Figure 33 shows a situation where the user is performing touch operations with their left hand, but the content described below is also suitably applicable when the user is performing touch operations with their right hand.

[0147] Figure 33 shows the normal vector L1 of the display surface 3a of the floating spatial image 3, extending toward the user from point C in the center of the display surface 3a; line L2 connecting the virtual light source 1500 to point C where the normal vector L1 intersects the display surface 3a; and the virtual light source installation angle α, which is defined by the angle between the normal vector L1 and line L2. For the sake of simplicity, Figure 33 shows the moment when the tip of the user's finger 210 is on line L2.

[0148] Here, in Figures 27-33, for the sake of simplicity, the virtual light source 1500 is depicted as being positioned not far from the display surface 3a of the floating spatial image 3 or the user's finger 210. While it is acceptable to set the virtual light source 1500 in such a position, the most suitable setting is as follows: It is desirable to set the distance between the virtual light source 1500 and the central point C of the display surface 3a of the floating spatial image 3 to infinity. The reason for this is as follows: If there were an object plane with a contact surface in the same coordinate system as the display surface 3a of the floating spatial image 3 in Figures 27-32, and the light source was the sun instead of a virtual light source, the distance to the sun could be approximated as almost infinity. Therefore, the horizontal position (x direction) of the tip of the shadow of the user's finger on the actual object plane would change linearly with respect to the change in the distance (z direction) between the tip of the user's finger and the object plane. Therefore, even in the setting of the virtual light source 1500 shown in Figures 27-33 of this embodiment, if the distance between the virtual light source 1500 and the central point C on the display surface 3a of the floating spatial image 3 is set to infinity, and the position of the tip of the virtual shadow 1510 on the floating spatial image 3 changes linearly in response to changes in the distance (z direction) between the tip of the user's finger 210 and the display surface 3a of the floating spatial image 3, a virtual shadow that can be perceived more naturally by the user can be represented.

[0149] If the virtual light source 1500 is positioned not too far from the display surface 3a of the floating image 3 or the user's finger 210, the horizontal position (x direction) of the tip of the virtual shadow 1510 in the floating image 3 changes non-linearly in response to changes in the distance (z direction) between the tip of the user's finger 210 and the display surface 3a of the floating image 3, making the calculation of the horizontal position (x direction) of the tip of the virtual shadow 1510 somewhat complicated. On the other hand, if the distance between the virtual light source 1500 and the central point C on the display surface 3a of the floating image 3 is set to infinity, the horizontal position (x direction) of the tip of the virtual shadow 1510 in the floating image 3 changes linearly in response to changes in the distance (z direction) between the tip of the user's finger 210 and the display surface 3a of the floating image 3, thus simplifying the calculation of the horizontal position (x direction) of the tip of the virtual shadow 1510.

[0150] If the virtual light source installation angle α is small, the angle between the line connecting the virtual light source 1500 and the finger 210 and the normal vector L1 cannot be increased from the user's perspective. As a result, the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 in the horizontal direction (x direction) of the display surface 3a of the floating spatial image 3 becomes shorter. This makes it difficult for the user to perceive the change in the position of the virtual shadow 1510 when the tip of the finger 210 performs a touch operation, potentially reducing the effectiveness of the user's depth perception during touch operation. To avoid this, it is desirable that the virtual light source 1500 be installed such that the angle between the line L2 connecting the virtual light source 1500 and point C and the normal vector L1 is, for example, 20° or more.

[0151] On the other hand, if the angle between the line connecting the virtual light source 1500 and the finger 210 and the normal vector L1 approaches 90°, the distance between the tip of the finger 210 and the tip of the virtual shadow 1510 becomes very long. This increases the probability that the display position of the virtual shadow 1510 will be outside the range of the floating spatial image 3, and thus increases the probability that the virtual shadow 1510 cannot be displayed in the floating spatial image 3. For this reason, it is desirable that the installation angle α of the virtual light source 1500 be 70° or less, so that the angle between the line L2 connecting the virtual light source 1500 and point C and the normal vector L1 does not approach 90°, for example.

[0152] In other words, it is desirable that the virtual light source 1500 be positioned so as not to be too close to the plane containing the normal vector passing through the finger 210, and not too close to the plane containing the display surface 3a of the floating spatial image 3.

[0153] The floating image display device 1000 of this embodiment can display virtual shadows as described above. This is a more physically natural image processing method than superimposing predetermined marks onto the image to assist the user's touch operation. Therefore, the touch operation assistance technology using the virtual shadow display described above in the floating image display device 1000 of this embodiment can provide a situation in which the user can perceive depth in touch operations more naturally.

[0154] <<Method for detecting finger position>> Next, we will explain how to detect the position of the finger 210. Below, we will specifically describe the configuration for detecting the position of the user's finger 210.

[0155] <<<Method for detecting finger position (1)>>> Figure 34 is a diagram illustrating an example of a method for detecting the position of a finger. In the example shown in Figure 34, the position of the finger 210 is detected using one imaging unit 1180 and one air-operated sensor 1351. Note that all imaging units in the embodiments of the present invention have imaging sensors.

[0156] The first imaging unit 1180a (1180) is installed on the opposite side of the floating spatial image 3 from the user 230. The first imaging unit 1180a may be installed in the housing 1190 as shown in Figure 34, or it may be installed in a location away from the housing 1190.

[0157] The imaging area of ​​the first imaging unit 1180a is set to include, for example, the display area of ​​the floating image 3, and the fingers, hands, arms, and face of the user 230. The first imaging unit 1180a images the user 230 performing a touch operation on the floating image 3 and generates a first image. Note that even if the display area of ​​the floating image 3 is imaged from the first imaging unit 1180a, the image will be taken from the opposite side of the direction of propagation of the directional light beam of the floating image 3, so the floating image 3 itself cannot be seen as an image. In the example of finger position detection method (1), the first imaging unit 1180a is not just an imaging unit, but incorporates a depth sensor in addition to the imaging sensor. Existing technology can be used for the configuration and processing of the depth sensor. The depth sensor of the first imaging unit 1180a detects the depth of each part (for example, the user's fingers, hands, arms, and face) in the image captured by the first imaging unit 1180a and generates depth information.

[0158] The aerial operation sensor 1351 is installed in a position where it can sense the display surface 3a of the floating spatial image 3 as the sensing target surface. In Figure 34, the aerial operation sensor 1351 is installed below the display surface 3a of the floating spatial image 3, but it may also be installed to the side or above the display surface 3a. The aerial operation sensor 1351 may be installed in the housing 1190 as shown in Figure 34, or it may be installed in a location away from the housing 1190.

[0159] In Figure 34, the aerial operation detection sensor 1351 is a sensor that detects the position where the finger 210 comes into contact with or overlaps with the display surface 3a of the floating image 3. That is, when the tip of the finger 210 approaches the display surface 3a of the floating image 3 from the user's side, the aerial operation detection sensor 1351 can detect contact between the finger 210 and the display surface 3a of the floating image 3.

[0160] For example, the control unit 1110 shown in Figure 3C reads a program for image processing and a program for displaying the virtual shadow 1510 from the non-volatile memory 1108. The control unit 1110 performs first image processing on the first captured image generated by the imaging sensor of the first imaging unit 1180a to detect the finger 210 and calculate the position (x coordinate, y coordinate) of the finger 210. Based on the first captured image generated by the imaging sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, the control unit 1110 calculates the position (z coordinate) of the tip of the finger 210 relative to the floating image 3.

[0161] In the example shown in Figure 34, the first imaging unit 1180a's imaging sensor and depth sensor, the aerial operation sensor 1351, the aerial operation detection unit 1350, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects touches to objects in the floating spatial image 3. This calculates the position (x, y, and z coordinates) of the finger 210. Furthermore, the touch detection result is calculated from the detection result of the aerial operation detection unit 1350 or from a combination of the detection result of the aerial operation detection unit 1350 and the information generated by the first imaging unit 1180a.

[0162] The control unit 1110 then calculates the position (display position) where the virtual shadow 1510 will be displayed based on the position of the finger 210 (x, y, and z coordinates) and the position of the virtual light source 1500, and generates video data of the virtual shadow 1510 based on the calculated display position.

[0163] The control unit 1110 may calculate the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated. Alternatively, instead of calculating the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated, display position map data, which pre-calculates the display positions of the virtual shadow 1510 corresponding to multiple locations on the finger 210, may be stored in the non-volatile memory 1108. After calculating the position of the finger 210, video data of the virtual shadow 1150 may be generated based on the display position map data stored in the non-volatile memory 1108. Furthermore, the control unit 1110 may calculate the tip of the finger 210 and the extension direction of the finger 210 in the first image processing, calculate the display position of the tip of the finger 210 and the extension direction of the virtual shadow 1510 corresponding to the extension direction, and generate video data of the virtual shadow 1510 adjusted to a display angle corresponding to the actual orientation of the finger 210 based on these.

[0164] The control unit 1110 outputs the video data of the generated virtual shadow 1510 to the video control unit 1160. The video control unit 1160 generates superimposed video data (superimposed video data) by superimposing the video data of the virtual shadow 1510 with other video data such as objects, and outputs the superimposed video data including the video data of the virtual shadow 1510 to the video display unit 1102.

[0165] The video display unit 1102 displays a video based on superimposed video data, which includes the video data of the virtual shadow 1510, thereby displaying a floating video 3 in which the virtual shadow 1510 and objects are superimposed.

[0166] Touch detection on an object is performed, for example, as follows: The aerial operation detection unit 1350 and the aerial operation detection sensor 1351 are configured as shown in Figure 3. When a finger 210 touches or overlaps with the plane including the display surface 3a of the floating spatial image 3, the unit detects its position and outputs touch position information indicating the position where the finger 210 touched or overlapped with the display surface 3a to the control unit 1110. When the control unit 1110 receives the touch position information, it determines whether the position of the finger 210 (x coordinate, y coordinate) calculated by the first image processing is included in the display range of each object displayed on the display surface 3a of the floating spatial image 3. If the position of the finger 210 is included in the display range of any object, the control unit 1110 determines that a touch has been made to that object.

[0167] According to the detection method described above, it is possible to detect the position of the finger 210 and the touch operation with a simple configuration that combines one imaging unit 1180 (first imaging unit 1180a) having an imaging sensor and a depth sensor, and one air operation detection sensor 1351.

[0168] As a variation of the finger position detection method (1), the control unit 1110 may detect a touch operation by the finger 210 solely based on the first captured image generated by the imaging sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, without using the detection results from the air operation detection unit 1350 and the air operation detection sensor 1351. For example, during normal operation, the system is configured to detect touch operations by a finger 210 by combining the captured image from the first imaging unit 1180a's imaging sensor, the detection result from the depth sensor, and the detection result from the aerial operation detection sensor 1351. If there is any malfunction in the operation of the aerial operation detection sensor 1351 or the aerial operation detection unit 1350, the control unit 1110 may switch to a mode in which it detects touch operations by a finger 210 using only the first captured image generated by the first imaging unit 1180a's imaging sensor and the depth information generated by the first imaging unit 1180a's depth sensor, without using the detection results from the aerial operation detection unit 1350 and the aerial operation detection sensor 1351.

[0169] <<Method for detecting finger position (2)>> Figure 35 is a diagram illustrating another example of a method for detecting the position of a finger. In the example shown in Figure 35, the position of the finger 210 is detected using two imaging units. The second imaging unit 1180b (1180) and the third imaging unit 1180c (1180) are both located on the side opposite to the user 230 with respect to the floating image 3.

[0170] The second imaging unit 1180b is installed, for example, on the right side as viewed from the user 230. The imaging area of ​​the second imaging unit 1180b is set to include, for example, the floating image 3, the user 230's fingers, hands, arms, face, etc. The second imaging unit 1180b captures the user 230 performing a touch operation on the floating image 3 from the right side of the user 230 and generates a second image.

[0171] The third imaging unit 1180c is installed, for example, on the left side as viewed from the user 230. The imaging area of ​​the third imaging unit 1180c is set to include, for example, the floating image 3, the user 230's fingers, hands, arms, face, etc. The third imaging unit 1180c captures the user 230 performing a touch operation on the floating image 3 from the left side of the user 230 and generates a third image. In this way, in the example of Figure 35, the second imaging unit 1180b and the third imaging unit 1180c constitute a so-called stereo camera.

[0172] The second imaging unit 1180b and the third imaging unit 1180c may be installed in the housing 1190 as shown in Figure 35, or they may be installed at a location away from the housing 1190. Alternatively, one imaging unit may be installed in the housing 1190, and the other imaging unit may be installed at a location away from the housing 1190.

[0173] The control unit 1110 performs a second image processing on the second captured image and a third image processing on the third captured image. Based on the results of the second image processing (second image processing result) and the third image processing (third image processing result), the control unit 1110 calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210.

[0174] In the example shown in Figure 35, the second imaging unit 1180b, the third imaging unit 1180c, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects touches to objects in the floating spatial image 3. The position of the finger 210 (x, y, and z coordinates) is then calculated as either a position detection result or a touch detection result.

[0175] As shown in Figure 35, in the example, a virtual shadow 1510 is generated based on the position of the finger 210 calculated from the second and third image processing results. Furthermore, a determination is made as to whether or not the object has been touched, based on the position of the finger 210 calculated from the second and third image processing results.

[0176] This configuration eliminates the need for an imaging unit with a depth sensor. Furthermore, by using the second imaging unit 1180b and the third imaging unit 1180c as a stereo camera, it is possible to improve the accuracy of detecting the position of the finger 210. In particular, the detection accuracy of the x and y coordinates can be improved compared to the example in Figure 34. As a result, it becomes possible to more accurately determine whether or not an object has been touched.

[0177] Furthermore, as a modification of the finger position detection method (2), the detection of the user's finger position (x, y, and z coordinates) is performed based on the second image captured by the second imaging unit 1180b and the third image captured by the third imaging unit 1180c, as described above, and the display of the virtual shadow 1510 is controlled accordingly. The presence or absence of touch on an object in the floating spatial image 3 is detected by the aerial operation detection unit 1350 or the control unit 1110 based on the detection result by the aerial operation detection sensor 1351. According to this modification, since the aerial operation sensor 1351 senses the display surface 3a of the floating spatial image 3 as the sensing target surface, it is possible to detect contact of the user's finger 210 with the display surface 3a of the floating spatial image 3 with higher accuracy than the depth direction detection accuracy of the stereo camera by the second imaging unit 1180b and the third imaging unit 1180c.

[0178] <<<Method for detecting finger position (3)>>> Figure 36 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in Figure 36, the position of the finger 210 is detected using two imaging units. Unlike the example in Figure 35, the example in Figure 36 is configured such that one of the imaging units, the fourth imaging unit 1180d (1180), is positioned to image the display surface 3a of the floating spatial image 3 from the side. Also, as in the example in Figure 34, the first imaging unit 1180a (1180) is installed on the opposite side of the floating spatial image 3 from the user 230. In the example in Figure 36, the first imaging unit 1180a (1180) only needs to be able to take images and does not need to be equipped with a depth sensor.

[0179] Therefore, the fourth imaging unit 1180d is installed around the display surface 3a of the floating spatial image 3. In Figure 36, the fourth imaging unit 1180d is installed on the lower side of the display surface 3a of the floating spatial image 3, but it may also be installed to the side or above the display surface 3a. The fourth imaging unit 1180d may be installed in the housing 1190 as shown in Figure 36, or it may be installed in a location away from the housing 1190.

[0180] The imaging area of ​​the fourth imaging unit 1180d is set to include, for example, the floating image 3, the fingers, hands, arms, and face of the user 230. The fourth imaging unit 1180d captures the user 230 performing a touch operation on the floating image 3 from around the display surface 3a of the floating image 3 and generates a fourth image.

[0181] The control unit 1110 performs a fourth image processing on the fourth captured image to calculate the distance (z coordinate) between the display surface 3a of the floating spatial image 3 and the tip of the finger 210. Then, based on the position of the finger 210 (x coordinate, y coordinate) calculated by the first image processing on the first captured image by the first imaging unit 1180a described above, and the position of the finger 210 (z coordinate) calculated by the fourth image processing, the control unit 1110 performs processing related to the virtual shadow 1510 and determines whether or not there is a touch on the object.

[0182] In the example shown in Figure 36, the first imaging unit 1180a, the fourth imaging unit 1180d, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects touches on an object. The position of the finger 210 (x, y, and z coordinates) is then calculated as the position detection result or touch detection result.

[0183] This configuration makes it possible to improve the detection accuracy of the distance between the display surface 3a of the floating image 3 and the tip of the finger 210, that is, the depth of the finger 210 relative to the display surface 3a of the floating image 3, compared to the example of the stereo camera configuration in Figure 35.

[0184] Furthermore, as a modification of the finger position detection method (3), the detection of the user's finger position (x, y, and z coordinates) is performed based on the first image captured by the first imaging unit 1180a and the fourth image captured by the fourth imaging unit 1180d, as described above, thereby controlling the display of the virtual shadow 1510. The presence or absence of touch on an object in the floating spatial image 3 is detected by the aerial operation detection unit 1350 or the control unit 1110 based on the detection result of the aerial operation detection sensor 1351. According to this modification, since the aerial operation sensor 1351 senses the display surface 3a of the floating spatial image 3 as the sensing target surface, it is possible to detect contact of the user's finger 210 with the display surface 3a of the floating spatial image 3 with higher accuracy than detection accuracy using the fourth image captured by the fourth imaging unit 1180d.

[0185] <<How to display input content to assist with touch operation>> This section describes examples of how to assist user touch operations in other ways. For example, it is possible to assist touch operations by displaying the entered content. Figure 37 illustrates how to assist touch operations by displaying the entered content. Figure 37 shows an example of entering numbers using touch operations.

[0186] The floating spatial image 3 in Figure 37 includes a key input UI (user interface) display area 1600 which includes multiple objects such as multiple objects for inputting numbers, an object 1601 for erasing input content, and an object 1603 for determining input content, and an input content display area 1610 which displays the input content.

[0187] In the input content display area 1610, the content entered via touch operation (e.g., numbers) is sequentially displayed on the floating spatial image 3, starting from the left edge and moving to the right. The user can confirm the content entered via touch operation while looking at the input content display area 1610. Then, after entering all the desired numbers, the user touches object 1603. This registers the input content displayed in the input content display area 1610. Unlike physical contact on the surface of a display device, touch operation on the floating spatial image 3 does not allow the user to feel the contact. Therefore, displaying the input content separately in the input content display area 1610 is preferable because it allows the user to proceed with the operation while confirming whether their touch operation was successful or not.

[0188] On the other hand, if the user touches the wrong object or enters content different from what was intended, they can erase the last entered content (in this case, "9") by touching object 1601. The user then continues touching the input objects for numbers, etc. Once the user has entered all the desired numbers, they touch object 1603.

[0189] In this way, by displaying the input content in the input content display area 1610, the user can be allowed to confirm the input content, thereby improving convenience. Furthermore, if the user touches the wrong object, the user can be allowed to correct the input content, further improving convenience.

[0190] <<How to highlight input content to assist with touch operation>> Next, it is also possible to highlight the input content to assist with touch operations. Figure 38 illustrates a method for assisting touch operations by highlighting the input content.

[0191] Figure 38 shows an example where a number entered via touch is highlighted. Following Figure 38, when the object corresponding to the number "6" is touched, the touched object is erased, and the entered number "6" is displayed in the area where the object was previously displayed.

[0192] In this way, by displaying a number corresponding to the touched object in place of the object itself, it becomes possible to make the user aware that they have touched an object, thereby improving usability. The number corresponding to the touched object may also be called a replacement object, as it replaces the touched object.

[0193] As another way to highlight input content, for example, the object touched by the user may be made to light up brightly or blink. Although not shown here, by recognizing the distance between the finger 210 and the display surface 3a as described in the embodiments of Figures 27-28, the object that the user is about to touch will become brighter than the surrounding objects as the finger approaches the display surface, and when it finally touches the display surface, the level of emphasis may reach its maximum, or it may light up even brighter or blink. In such a configuration, it is possible to make the user aware that they have touched an object, thereby improving convenience.

[0194] <<Method to assist touch operation using vibration (1)>> Next, a method for assisting touch operations using vibration will be described. Figure 39 is a diagram illustrating an example of a method for assisting touch operations using vibration. Figure 39 shows a case where touch operations are performed using a touch pen (touch input device) 1700 instead of a finger 210. The touch pen 1700 is equipped with a communication unit that sends and receives various information such as signals and data with a device such as a floating image display device, and a vibration mechanism that vibrates based on the input signal.

[0195] Suppose the user operates the stylus 1700 and touches an object displayed in the key input UI display area 1600 of the floating spatial image 3 with the stylus 1700. At this time, for example, the control unit 1100 transmits a touch detection signal from the communication unit 1132 to indicate that it has detected a touch on the object. When the stylus 1700 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. As a result, the stylus 1700 vibrates. The vibration of the stylus 1700 is then transmitted to the user, and the user recognizes that they have touched an object. In this way, the vibration of the stylus 1700 assists in touch operation.

[0196] This configuration allows the user to recognize when they have touched an object through vibration.

[0197] This description assumes that the touch detection signal transmitted from the floating image display device is received by the touch pen 1700, but other configurations are also possible. For example, when a touch on an object is detected, the floating image display device notifies the host device that a touch on the object has been detected. The host device then transmits a touch detection signal to the touch pen 1700.

[0198] Alternatively, the floating image display device and the host device may transmit a touch detection signal via a network. In this way, the touch pen 1700 may indirectly receive a touch detection signal from the floating image display device.

[0199] <<Method to assist touch operation using vibration (2)>> Next, we will describe other methods of assisting touch operations using vibration. In this method, the user's device vibrates to allow the user to recognize that they have touched an object. Figure 40 illustrates another example of a method of assisting touch operations using vibration. In the example in Figure 40, user 230, wearing a wristwatch-type wearable device 1800, performs a touch operation.

[0200] The wearable terminal 1800 is equipped with a communication unit that transmits and receives various information such as signals and data to and from devices such as a spatially floating image display device, and a vibration mechanism that vibrates based on the input signal.

[0201] Suppose the user performs a touch operation with their finger 210 and touches an object displayed in the key input UI display area 1600 of the floating spatial image 3. At this time, for example, the control unit 1100 transmits a touch detection signal from the communication unit 1132 to indicate that a touch on the object has been detected. When the wearable terminal 1800 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. As a result, the wearable terminal 1800 vibrates. The vibration of the wearable terminal 1800 is then transmitted to the user, and the user recognizes that they have touched an object. In this way, the vibration of the wearable terminal 1800 assists with the touch operation. Here, a wristwatch-type wearable terminal was used as an example, but a smartphone or other device worn by the user could also be used.

[0202] The wearable terminal 1800 may also receive touch detection signals from a higher-level device, similar to the stylus pen 1700 described above. Furthermore, the wearable terminal 1800 may receive touch detection signals via a network. In addition to the wearable terminal 1800, it is also possible to use other information processing terminals, such as a smartphone owned by the user, to assist with touch operations.

[0203] This configuration allows users to recognize when they have touched an object via various devices, such as the wearable device 1800 that they own.

[0204] <<Method to assist touch operation using vibration (3)>> Next, other methods of assisting touch operation with vibration will be described. Figure 41 illustrates another example of a method of assisting touch operation with vibration. In the example in Figure 41, user 230 stands on the vibrating plate 1900 and performs touch operation. The vibrating plate 1900 is installed in a predetermined position where user 230 will perform touch operation. In actual use, the vibrating plate 1900 would be placed, for example, under a mat (not shown), and user 230 would stand on the vibrating plate 1900 via the mat.

[0205] As shown in Figure 41, the diaphragm 1900 is connected to, for example, the communication unit 1132 of the floating image display device 1000 via the cable 1910. When a touch on an object is detected, for example, the control unit 1110 supplies an AC voltage to the diaphragm 1900 via the communication unit 1132 for a predetermined time. The diaphragm 1900 vibrates while the AC voltage is supplied. That is, the AC voltage is a control signal output from the communication unit 1132 to vibrate the diaphragm 1900. The vibration generated by the diaphragm 1900 is transmitted from the user's feet to the user 230, allowing the user 230 to recognize that they have touched an object. In this way, the vibration of the diaphragm 1900 assists in touch operations.

[0206] The frequency of the AC voltage is set to a value within the range in which user 230 can perceive vibration. The frequency range of vibrations that humans can perceive is approximately 0.1 Hz to 500 Hz. Therefore, it is desirable to set the frequency of the AC voltage within this range.

[0207] Furthermore, it is desirable that the frequency of the AC voltage be appropriately changed according to the characteristics of the diaphragm 1900. For example, when the diaphragm 1900 vibrates vertically, humans are said to be most sensitive to vibrations of around 410 Hz. When the diaphragm 1900 vibrates horizontally, humans are said to be most sensitive to vibrations of around 12 Hz. Moreover, at frequencies above 34 Hz, humans are said to be more sensitive to vertical vibrations than to horizontal vibrations.

[0208] Therefore, when the diaphragm 1900 vibrates vertically, it is desirable to set the frequency of the AC voltage to a value within a range that includes, for example, 410 Hz. Also, when the diaphragm 1900 vibrates horizontally, it is desirable to set the frequency of the AC voltage to a value within a range that includes, for example, 12 Hz. Note that the peak voltage and frequency of the AC voltage may be adjusted as appropriate depending on the performance of the diaphragm 1900.

[0209] This configuration allows the user 230 to recognize when an object has been touched through vibrations from their feet. Furthermore, this configuration allows the display of the floating image 3 to remain unchanged when an object is touched, reducing the possibility of others seeing the input content and thus improving security.

[0210] Although various embodiments have been described in detail above, the present invention is not limited to the embodiments described above, but includes various modifications. For example, the embodiments described above are detailed explanations of the entire system in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. [Explanation of Symbols]

[0211] 1…Display device, 2…Retroreflective member, 3…Spatial image (floating image), 105…Wind glass, 100…Transparent member, 101…Polarization separation member, 12…Absorbing polarizer, 13…Light source device, 54…Light direction conversion panel, 151…Retroreflective member, 102, 202…LED substrate, 203…Light guide, 205, 271…Reflective sheet, 206, 270…Phase difference plate, 300…Floating image, 301…Ghost of floating image 302...Ghost image of floating image, 230...User, 1000...Floating image display device, 1110...Control unit, 1160...Image control unit, 1180...Imaging unit, 1102...Image display unit, 1350...Air operation detection unit, 1351...Air operation detection sensor, 1500...Virtual light source, 1510...Virtual shadow, 1610...Input content display area, 1700...Touch pen, 1800...Wearable terminal, 1900...Diaphragm.

Claims

1. a display device for displaying an image; a retroreflective member that reflects image light from the display device and forms a floating image in the air using the reflected light; a sensor for detecting the position of a user's finger performing a touch operation on one or more objects displayed on the floating image; A control unit; Equipped with Based on the position of the user's finger detected using the sensor, the control unit controls image processing for the image displayed on the display device, thereby displaying a virtual shadow of the user's finger on the display surface of the space floating image, which has no physical contact surface. A floating image display device.

2. 2. The space floating image display device according to claim 1, When the position of the tip of the user's finger changes in the normal direction on the front side of the display surface of the space floating image as seen from the user, the position of the tip of the virtual shadow displayed on the space floating image in the left-right direction on the display surface of the space floating image changes. A floating image display device.

3. 3. The space floating image display device according to claim 2, With respect to a change in the normal direction of the position of the tip of the user's finger, a left-right position of the tip of the virtual shadow displayed on the space floating image on the display surface of the space floating image changes linearly. A floating image display device.

4. 2. The space floating image display device according to claim 1, an imaging unit that captures an image of a user's hand or arm; When a finger of a user performing a touch operation on one or more objects displayed in the floating image in space is a finger of the user's right hand, the virtual shadow is displayed in the floating image in space at a position to the left of the tip of the finger as seen from the user, When a finger of a user performing a touch operation on one or more objects displayed in the floating-in-space image is a finger of the left hand, the virtual shadow is displayed in the floating-in-space image at a position to the right of the tip of the finger as seen from the user. A floating image display device.

5. 2. The space floating image display device according to claim 1, The control unit detects the position of the tip of the finger on the display surface of the space floating image and the height position of the tip of the finger relative to the display surface using the sensor that detects the position of the user's finger. A floating image display device.

6. 2. The space floating image display device according to claim 1, The presence or absence of the user's finger touching the display surface of the space floating image is detected by a sensor different from a sensor that detects the position of the user's finger. A floating image display device.

7. 2. The space floating image display device according to claim 1, The position of the virtual shadow displayed on the display surface of the space floating image is a position specified by the positional relationship between the position of a virtual light source and the position of the user's finger detected using the sensor. A floating image display device.

8. 8. The space floating image display device according to claim 7, The position of the virtual light source is a virtual light source installation angle defined by the angle between a normal line extending from a central point on the display surface of the space floating image toward the user and a line connecting the virtual light source and the central point on the display surface of the space floating image is 20° or more; A floating image display device.

9. 2. The space floating image display device according to claim 1, The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image changes in conjunction with the angle of the user's finger imaged by the imaging unit of the space floating image display device. A floating image display device.

10. 2. The space floating image display device according to claim 1, The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image is a fixed angle, not linked to the angle of the user's finger imaged by the imaging unit of the space floating image display device, A floating image display device.

11. a display device for displaying an image; a retroreflective member that reflects image light from the display device and forms a floating image in the air using the reflected light; a sensor that detects a touch operation of a user's finger on one or more objects displayed on the floating image in space; A control unit; Equipped with When the user performs a touch operation on the object, the control unit assists the user in the touch operation based on a detection result of the touch operation using the sensor. A floating image display device.

12. 12. The space floating image display device according to claim 11, The floating image in space includes an input content display area that displays the content input by the touch operation at a position different from the object. A floating image display device.

13. 12. The space floating image display device according to claim 11, When the object is touched, the touched object is erased and a replacement object showing the content corresponding to the touched object is displayed. A floating image display device.

14. 12. The space floating image display device according to claim 11, When the object is touched, the touched object is lit up. A floating image display device.

15. 12. The space floating image display device according to claim 11, When the object is touched, the touched object blinks. A floating image display device.

16. 12. The space floating image display device according to claim 11, the user performs the touch operation using a touch input device, and when the object is touched, the touch input device is vibrated; A floating image display device.

17. 12. The space floating image display device according to claim 11, vibrating a terminal held by the user when the object is touched; A floating image display device.

18. 18. The space floating image display device according to claim 17, The terminal is a wearable terminal. A floating image display device.

19. 18. The space floating image display device according to claim 17, The terminal is a smartphone. A floating image display device.

20. 12. The space floating image display device according to claim 11, When the object is touched, a control signal for vibrating a diaphragm placed at the user's feet is output from a communication unit of the space floating image display device. A floating image display device.