Space-floating image display device

Abstract

To provide a technology which, when a spatial floating video is used as a contactless user interface, is easier to use for users, has higher visibility and operability, and is suitable for preventing or reducing misoperation or erroneous input, thus contributing to sustainable development goals "3 health and welfare for all people," "9 creating infrastructure for industry and technological innovation," and "11 community development worth continuing to live."SOLUTION: A spatial floating video display device 1000 comprises a video display device 1 and a retroreflective member 2, forming a spatial floating video 3 on the basis of reflected light from the retroreflective member 2, and further includes a sensor 4 for detecting a state of an operation, including the position of a finger UH of a user U, upon a space area that includes a plane of the spatial floating video 3 or an object displayed on the plane, and a finger tip touch sense generation device 6 for forming a sound pressure by an ultrasonic wave near the position of the finger UH, on the basis of information detected by the sensor 4, so as to generate a tactile sensation to the finger UH.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to a technology for a space floating image display device. [Background technology]

[0002] As a space-floating information display system, there are already known image display devices that display space-floating images toward the outside, and display methods that display space-floating images that can be input by the user as a user interface or man-machine interface. In contrast to "contact user interfaces" such as conventional liquid crystal touch panels, which require the user to touch the physical display surface with their fingers, such space-floating images can be used as "non-contact user interfaces."

[0003] As an example of prior art, JP 2019-128722 A (Patent Document 1) discloses providing a display and operation device that can provide an operator who operates an image of an immaterial operation object displayed in the air with a reliable sense of operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-128722 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the prior art example, it is described that in order to reduce false positives of operations on an image formed in the air, when a user approaches the image from a predetermined direction, the purpose is achieved by accepting the user's movement as an operation.

[0006] A floating-in-space image, which is an image displayed by a floating-in-space image display device, is displayed as if it is floating in space. The user can visually recognize the floating-in-space image. However, there is no "image as a substance" at the position in the space where the floating-in-space image is displayed. In other words, there is no surface or the like that can be touched as an object other than air at that position.

[0007] Therefore, when using floating images as a non-contact user interface, the user cannot get a definite tactile sensation (e.g., a touch sensation) of "touching an object" like, for example, a push button (an object configured as a GUI image) on a conventional touch panel screen. This can easily lead to erroneous operation or erroneous input. For example, there is a possibility that a touch operation may not be determined to have been performed even if the user intended to touch a button, or that a touch operation may be determined to have been performed even if the user did not intend to touch a button.

[0008] The object of the present invention is to provide a technology for a space-floating image display device that is easy for users to use, has high visibility and operability, and is suitable for preventing or reducing erroneous operations and erroneous inputs when the generated space-floating image is used as a non-contact user interface. [Means for solving the problem]

[0009] A representative embodiment of the present invention has the following configuration. The space-floating image display device of the embodiment is a space-floating image display device that forms a space-floating image, and includes a display device that displays an image, a retroreflective member that retroreflects image light from the display device, forms the space-floating image based on the reflected light from the retroreflective member, and includes a sensor for detecting an operation state including the position of a user's fingers relative to a spatial region including a surface of the space-floating image or an object displayed on the surface, and a tactile sensation generating device that generates a tactile sensation in the fingers by forming ultrasonic sound pressure near the position of the fingers based on the information detected by the sensor. Effect of the Invention

[0010] According to a representative embodiment of the present invention, the present invention provides a technology for a space-floating image display device that is easy to use for users, has high visibility and operability, and is suitable for preventing or reducing erroneous operations and erroneous inputs when the generated space-floating image is used as a non-contact user interface. Other issues, configurations, effects, etc. will be described in [Mode for carrying out the invention]. [Brief description of the drawings]

[0011] [Figure 1] 1 shows an example of a functional block configuration of a space floating image display device according to an embodiment of the present invention. [Diagram 2] 1 shows an example of the configuration of a main part of a space floating image display device according to an embodiment. [Diagram 3] 3 shows an example of the configuration of a retroreflective member. [Figure 4] 1 shows an example of incidence and reflection of light on a retroreflective member. [Diagram 5] 1A and 1B are schematic explanatory diagrams showing a normal image and a ghost image in a space floating image display device. [Figure 6] 13 shows another example of the main configuration of the space floating image display device according to the embodiment. [Figure 7] 1 shows an example of the configuration of a light blocking member in a space floating image display device according to an embodiment. [Figure 8] 11 shows another example of the configuration of the light blocking member in the space floating image display device according to the embodiment. [Figure 9] 1 shows an example of the configuration of a non-contact user interface using a space-floating image in a space-floating image display device according to an embodiment. [Figure 10] 10 shows an example of the user's gaze direction with respect to the space floating image of FIG. [Figure 11] 13 shows another example of the configuration of a non-contact user interface using a space-floating image in the space-floating image display device according to the embodiment. [Figure 12] 12 shows an example of the user's gaze direction with respect to the space floating image of FIG. 11. [Figure 13] 1 shows an example of the arrangement of super-directional speakers in a space floating image display device according to an embodiment. [Figure 14] 13 shows another example of the arrangement of superdirectional speakers in the space floating image display device according to the embodiment. [Figure 15] 1 shows an example of the arrangement of a space-floating image, a super-directional speaker, and a camera in a space-floating image display device according to an embodiment. [Figure 16] 13 shows another example of the arrangement of the space-floating image, the super-directional speaker, and the camera in the space-floating image display device according to the embodiment. [Figure 17] 1 shows a configuration of a space floating image display device according to an embodiment, in which a user, a space floating image, a fingertip tactile sense generating device, etc. are seen from the side. [Figure 18] 1 shows an example of the configuration of a fingertip tactile sense generating device which is a fingertip tactile sense generating unit in a space floating image display device according to an embodiment of the present invention. [Figure 19] 1 shows an example of the configuration of an inductance circuit in a space floating image display device according to an embodiment. [Figure 20] 1 shows an example of a configuration of phase groups on a plane of an ultrasonic element array in a space floating image display device according to an embodiment. [Figure 21] 1 shows an example of the configuration of a non-contact user interface using a floating-in-space image in a floating-in-space image display device according to an embodiment, and an example of the arrangement of a fingertip tactile sensation generating device, etc. [Figure 22] 1A and 1B show a configuration example of a non-contact user interface using a floating-in-space image in a floating-in-space image display device according to an embodiment, and other arrangement examples of a fingertip tactile sense generating device and the like. [Figure 23] 13 shows another example of the arrangement of the fingertip tactile sense generating device in the space floating image display device according to the embodiment. [Figure 24] 13 shows an example of an operation for a space-floating image in the space-floating image display device according to the embodiment. [Diagram 25] FIG. 2 is an explanatory diagram relating to light source diffusion characteristics of a display device according to an embodiment. [Figure 26] FIG. 2 is an explanatory diagram relating to light source diffusion characteristics of a display device according to an embodiment. [Figure 27] 1 shows an example of the configuration of a display device according to an embodiment. [Figure 28] 1 is a cross-sectional view showing an example of the configuration of a light source device according to an embodiment. [Figure 29] 1 is a cross-sectional view showing an example of the configuration of a light source device according to an embodiment. [Diagram 30] 1 is a layout diagram showing a configuration of a main part of a space floating image display device according to an embodiment of the present invention; [Diagram 31] 1 is a cross-sectional view showing a configuration example of an image display device in a space floating image display device according to an embodiment. [Diagram 32] 1 is a cross-sectional view showing an example of the configuration of a light source device according to an embodiment. [Diagram 33] 1 is a cross-sectional view showing an example of the configuration of a light source device according to an embodiment. [Diagram 34] 1 is a cross-sectional view showing an example of the configuration of a light source device according to an embodiment. [Diagram 35] 1 is a cross-sectional view showing an example of the configuration of a light source device according to an embodiment. [Diagram 36] FIG. 2 is an enlarged cross-sectional view of a light guide according to one embodiment. [Figure 37] FIG. 2 is an explanatory diagram relating to the diffusion characteristics of a display device according to an embodiment. [Figure 38] FIG. 2 is an explanatory diagram relating to the diffusion characteristics of a display device according to an embodiment. [Figure 39] 1 is a cross-sectional view showing a configuration example of a display device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same parts are generally given the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of each component may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention, and the present invention is not limited to the configurations disclosed in the drawings.

[0013] For the purpose of explanation, when describing processing by a program, the program, functions, processing units, etc. may be described as the main components, but the main hardware components of these are the processor, or a controller, device, computer, system, etc. that is composed of the processor, etc. The computer executes processing according to the program read into the memory by the processor, appropriately using resources such as memory and communication interfaces. This realizes predetermined functions, processing units, etc. The processor is composed of semiconductor devices such as a CPU or GPU, for example. The processor is composed of devices or circuits that are capable of performing predetermined calculations. Processing is not limited to software program processing, and can also be implemented by dedicated circuits. Dedicated circuits such as FPGA, ASIC, CPLD, etc. can be applied.

[0014] The program may be installed as data in advance on the target computer, or may be distributed as data from a program source to the target computer and installed. The program source may be a program distribution server on a communication network, or a non-transitory computer-readable storage medium. The program may be composed of multiple program modules. The computer system may be composed of multiple computers. The program may be composed of multiple program modules. The computer system may be composed of a client-server system or a cloud computing system.

[0015] <Example of conventional technology> A conventional example of a space-floating image display device is a device that combines an image display device such as an organic EL panel or a liquid crystal panel as a high-resolution color display image source with a retroreflective member. In the conventional example of a space-floating image display device, the image light is diffused at a wide angle. In addition, in the conventional example of a space-floating image display device, the retroreflective portion 2a constituting the retroreflective member 2 is a hexahedron as shown in Figs. 3 and 4. Therefore, in the conventional example of a space-floating image display device, in addition to the reflected light that is normally reflected, as shown in Fig. 4, the image light that is obliquely incident on the retroreflective member 2 (multiple retroreflective portions 2a) generates multiple ghost images, from the first ghost image G1 to the sixth ghost image G6, as ghost images, in addition to the normal image R1 that is the normal space-floating image 3, as shown in Fig. 5. This impairs the image quality of the space-floating image.

[0016] FIG. 5 shows an example of how a floating image in space appears when viewed from the viewpoint of a regular user (an eye point in a standard correct position). In the conventional technology example, ghost images G1 to G6 are generated. In the embodiment, such ghost images are prevented, and only a regular image R1 is obtained. The occurrence of ghost images is not only annoying for the user, but also a person other than the original user (for example, a stranger near the user) may view a ghost image having the same content as the floating image in space, which has been a major issue.

[0017] In addition, in the non-contact user interface using the floating image in the conventional floating image display device, a unique problem has been revealed in that an erroneous input occurs, such as selecting an object (e.g., a push button) different from the object the user intended to select and operate, due to a misalignment between the floating image and the user's line of sight. The conventional technology example may induce such erroneous input against the user's will in devices and systems to which a non-contact user interface using floating image is applied. For example, when applied to a bank ATM device, it may cause a major error such as an incorrect input of the amount. In addition, if the user is a person with relatively poor eyesight, such as an elderly person, the frequency of erroneous inputs as described above may increase. The above-mentioned erroneous inputs and erroneous operations are also a major issue from the perspective of the future spread and application of floating image display devices to various uses.

[0018] On the other hand, the space floating image display device of the embodiment has a configuration that improves visibility by eliminating ghost images that significantly reduce the visibility of the space floating image in the space floating image display device of the conventional technology example and improving the brightness of the space floating image. In addition, the space floating image display device of the embodiment has a function of generating a physical contact sensation (touch sensation, etc.) for the fingertip when the user performs an operation such as touching the surface of the space floating image with the fingertip at the place where the space floating image in the external air is displayed. For example, the space floating image is used as a non-contact user interface in a method of touching at least one button. In that case, when the user touches the button with the fingertip, the space floating image display device generates and imparts a touch sensation to the fingertip as if the user had touched a physical button.

[0019] Furthermore, the embodiment of the floating-in-space image display device has a function of emitting a sound (e.g., a sound of a number written on a button) corresponding to the location touched by the fingertip (e.g., the touched button) from near the location when the user touches the surface of the floating-in-space image.

[0020] In the embodiment of the space floating image display device described below, the generated space floating image is used as a non-contact user interface including objects such as push buttons. Note that these objects are elements and parts that make up the space floating image and the graphical user interface (GUI), and are images and images that can be seen even though they have no physical substance other than air.

[0021] <Space-floating image display device> Fig. 1 shows an example of a functional block configuration of a space-floating image display device according to an embodiment. The space-floating image display device 1000 in Fig. 1 includes a retroreflection unit 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power source 1106, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, an image signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an air-operation detection sensor 1351, an air-operation detection unit 1350, a fingertip tactile sense generation unit (in other words, a touch sensation generation unit) 1230, an audio signal output unit 1240, a super-directional speaker 1242, a normal speaker 1243, an image control unit 1160, a storage unit 1170, and an imaging unit 1180. These elements are connected to each other via a bus or the like. The main components of the space floating image display device 1000 are housed in a housing 1190. The imaging unit 1180 and the mid-air operation detection sensor 1351 may be provided as part of the housing 1190 or outside the housing 1190.

[0022] The retroreflective portion 1101 in FIG. 1 corresponds to the retroreflective member 2 in FIG. 2. The retroreflective portion 1101 retroreflects light modulated by the image display portion 1102. The space-floating image 3 is formed by the light reflected from the retroreflective portion 1101 and outputted to the outside of the space-floating image display device 1000. The image display portion 1102 in FIG. 1 corresponds to the liquid crystal display panel 11 in FIG. 2, and is equivalent to a color display image source. The light source 1105 in FIG. 1 corresponds to the light source device 13 in FIG. 2. The image display portion 1102, the light guide 1104, and the light source 1105 in FIG. 1 correspond to the display device 1 (image display device) in FIG. 2.

[0023] The image display unit 1102 is a display unit that generates an image by modulating transmitted light based on an image signal input under the control of the image control unit 1160. For example, a transmissive liquid crystal panel is used as the image display unit 1102. For example, a reflective liquid crystal panel using a method of modulating reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as the image display unit 1102. The light source 1105 generates light for the image display unit 1102, and is, for example, a solid light source such as an LED light source or a laser light source. The power source 1106 converts an AC current input from the outside into a DC current and supplies power to the light source 1105. The power source 1106 also supplies the necessary DC current to each unit in the space floating image display device 1000.

[0024] The light guide 1104 guides the light generated by the light source 1105 and irradiates the light to the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can be called a backlight of the image display unit 1102. There are various types of combination of the light guide 1104 and the light source 1105. Specific configuration examples of the combination of the light guide 1104 and the light source 1105 will be described later.

[0025] The aerial operation detection sensor 1351 is a sensor for detecting an operation of the floating-in-space image 3 by the finger UH of the user U in FIG. 2. The aerial operation detection sensor 1351 senses, for example, a range that overlaps with the entire display range of the floating-in-space image 3. The aerial operation detection sensor 1351 may sense only a range that overlaps with at least a part of the display range of the floating-in-space image 3. Specific examples of the aerial operation detection sensor 1351 include a distance sensor using invisible light such as infrared light, invisible light laser, or ultrasonic waves. The aerial operation detection sensor 1351 may also be configured to detect position coordinates of a two-dimensional plane corresponding to the main surface of the floating-in-space image 3 by combining multiple sensors. The aerial operation detection sensor 1351 may also be configured with a ToF (Time Of Flight) LiDAR (Light Detection and Ranging) or an image sensor (in other words, a camera). The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect operations such as a touch operation performed by the fingers UH of the user U on an object displayed as the floating-in-space image 3. Existing technology can also be applied to such sensing.

[0026] The aerial operation detection unit 1350 acquires a sensing signal (in other words, detection information) from the aerial operation detection sensor 1351, and determines the state including the presence or absence of contact with an object in the floating in space image 3 by the fingers UH of the user U based on the sensing signal, and calculates the position where the fingertip contacts the object. The aerial operation detection unit 1350 may be configured with a circuit such as an FPGA. Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized by software based on a spatial operation detection program executed by the processor of the control unit 1110, for example.

[0027] The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided separately from the space-floating image display device 1000 and provided externally. When provided separately, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to transmit information and signals to the space-floating image display device 1000 (for example, a control device described later) via a wired or wireless communication connection path or signal transmission path. When the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are provided separately, it is possible to construct a system in which the space-floating image display device without the aerial operation detection function is used as the main body, and only the aerial operation detection function can be added as an option. Also, only the aerial operation detection sensor 1351 may be provided separately, and the aerial operation detection unit 1350 may be built into the space-floating image display device. In cases where it is desired to more freely arrange the aerial operation detection sensor 1351 relative to the installation position of the space floating image display device, there is an advantage to a configuration in which only the aerial operation detection sensor 1351 is a separate body.

[0028] Each element, such as the aerial operation detection sensor 1351, the imaging unit 1180 (particularly the camera), the ultradirectional speaker and the ultrasonic element array described below, is basically configured to be arranged in a pre-designed fixed position and orientation, but the position and orientation may also be variable and adjustable by the user.

[0029] The imaging unit 1180 is configured using, for example, a camera having an image sensor, and captures the space near the floating in space image 3 and / or the body of the user U who operates the floating in space image 3 (at least a part of the head, face, eyes, arms, fingers, etc.). A plurality of imaging units 1180 may be provided. For example, the imaging unit 1180 may be provided as a stereo camera with two or more cameras. The imaging unit 1180 may be an imaging unit with a depth sensor. The imaging unit 1180 may also assist the mid-air operation detection sensor 1351 and the mid-air operation detection unit 1350 in detecting the operation of the floating in space image 3 by the user U. In particular, by using a plurality of imaging units 1180 or an imaging unit with a depth sensor, it is possible to assist the mid-air operation detection unit 1350 in making the detection process easier.

[0030] For example, the aerial operation detection sensor 1351 is configured as an object intrusion sensor that detects the presence or absence of an object intrusion into the intrusion detection plane (also described as an intrusion detection plane) including the display surface of the floating in space image 3, or a contact detection sensor that detects the presence or absence of the finger UH contacting the surface of the floating in space image 3. In this case, it may be difficult or impossible for the aerial operation detection sensor 1351 alone to detect information such as how far an object (e.g., the finger UH) that has not intruded into the intrusion detection plane is from the intrusion detection plane, or how close the object is to the intrusion detection plane. In such a case, the distance between the object and the intrusion detection plane can be calculated, or can be calculated with higher accuracy, by using object depth calculation information based on the captured image of the camera of the imaging unit 1180, object depth information by a depth sensor, etc. Then, various information including such distances detected and calculated using various sensor devices can be effectively used for various display controls for the floating in space image 3, etc.

[0031] In a modified example, the aerial operation detection unit 1350 may detect the operation of the floating-in-space image 3 by the user U based on the captured image of the imaging unit 1180 without using the aerial operation detection sensor 1351. In other words, the position of the aerial operation detection sensor 1351 may be a position on the opposite side to the floating-in-space image 3 and the user U (a position that can capture the two-dimensional plane of the floating-in-space image 3) like the position of the imaging unit 1180, instead of a position on the side closer to the floating-in-space image 3 and the user U as in FIG. 2 (a position on an extension of the plane of the floating-in-space image 3).

[0032] Also, the imaging unit 1180 may capture an image of the face of the user U who operates the space-floating image 3, and the control unit 1110 may perform an identification process of the user U (for example, user authentication based on face recognition) based on the captured image. Alternatively, the imaging unit 1180 may simply realize a function such as a human sensor. Also, there is a possibility that a person standing around or behind the legitimate user U who operates the space-floating image 3 facing the space-floating image 3 may peek at the contents of the space-floating image 3 or the operation of the user U on the space-floating image 3. If it is desired to ensure the confidentiality of the contents and operation of the space-floating image 3, such peeking should be prevented. Therefore, in order to determine whether or not a person is peeking, the imaging unit 1180 may capture an image of the range including the user U who operates the space-floating image 3 and its surrounding area, and determine whether or not a person is peeking.

[0033] The operation input unit 1107 is, for example, an operation button or a remote controller light receiving unit, and inputs a signal for an operation different from the aerial operation on the space floating image 3 by the user U. The operation input unit 1107 may be used by a person other than the user U who operates the space floating image 3, for example, an administrator, to operate the space floating image display device 1000.

[0034] The video signal input unit 1131 inputs video data from a connected external video output device. The audio signal input unit 1133 inputs audio data from a connected external audio output device. Meanwhile, the audio signal output unit 1240 can output an audio signal based on the audio data input to the audio signal input unit 1133. The audio signal output unit 1240 may also output an audio signal based on data previously recorded in the storage unit 1170 or data built into the non-volatile memory 1108. For example, audio data such as numbers and character strings, and audio data of other operation sounds and error warning sounds are included. The audio data includes data for generating an audio signal associated with the floating in space image 3 and an object. An example of the operation sound is a sound (such as "pong") output when an object such as a push button in the floating in space image 3 is touched.

[0035] The audio signal output unit 1240 may be connected to a speaker (normal speaker 1243 in FIG. 1) that outputs audio in the normal audible band. In the case of audio that does not need to be kept confidential, the normal speaker 1243 may be used. The audio signal output unit 1240 may also be connected to the superdirectional speaker 1242 in FIG. 1. When the space floating image display device 1000 is applied to a system that requires consideration of security, such as an ATM device in a bank, it is effective to use the superdirectional speaker 1242 so that no one other than the user U can hear the audio. The superdirectional speaker 1242 in FIG. 1 corresponds to the superdirectional speaker 30 in FIG. 2.

[0036] The superdirectional speaker 1242 is configured with an array of ultrasonic output elements, each capable of generating an ultrasonic signal of, for example, about 40 kHz, arranged on a plane. In this case, generally, the more ultrasonic output elements used, the louder the volume of the sound obtained by the superdirectional speaker becomes.

[0037] An ultradirectional speaker is a speaker that outputs ultradirectional sound so that only people (corresponding ears) present in a specific limited spatial region can hear the sound in the audible band. In other words, an ultradirectional speaker has a characteristic that people (corresponding ears) present outside the specific limited spatial region cannot hear the sound. The principle of an ultradirectional speaker will be briefly explained. As is well known, ultrasonic waves have a high degree of directivity compared to sounds in the audible band (e.g., human speech). Therefore, by using the above-mentioned 40 kHz ultrasonic signal as a carrier and modulating (e.g., amplitude modulating) the carrier with an audio signal in the audible band, it is possible to make the sound audible only in a specific limited spatial region.

[0038] For example, the space-floating image display device 1000 uses the imaging unit 1180 (e.g., multiple cameras) to identify the positions of the face, eyes, ears, etc. of the user U in space. Then, according to the identification result, the space-floating image display device 1000 can control so that the sound from the superdirectional speaker 1242 is heard only in a specific area near the ears of the user U. Specifically, the space-floating image display device 1000 controls the phase (in other words, the delay time) of each ultrasonic signal input to each ultrasonic output element constituting the superdirectional speaker 1242. This makes it possible to make the sound heard only in a specific limited spatial area.

[0039] Furthermore, as a configuration of superdirectional speaker 1242, a plurality of ultrasonic output elements may be arranged on, for example, a concave curved surface, rather than on a flat surface, so that sound can be heard only in the above-mentioned specific limited spatial region. Superdirectional speaker 1242 may be configured as a part of housing 1190, or may be configured as a separate body from housing 1190. A specific example will be described later.

[0040] The fingertip tactile sense generating unit 1230 is a part having a function of generating and imparting a tactile sensation to the fingertips when the user U's fingers UH, including the fingertips, operate the floating-in-space image 3. When a touch operation by the user U's fingertips on an object displayed as the floating-in-space image 3 is detected, the fingertip tactile sense generating unit 1230 generates and imparts a touch sensation as if there is an object other than air on the fingertips. The fingertip tactile sense generating unit 1230 is configured, for example, using an ultrasonic element array in which a plurality of ultrasonic output elements are arranged on a plane. An example of the fingertip tactile sense generating unit 1230 will be described later.

[0041] The fingertip haptic generation unit 1230 uses information on the touch position of the user U's fingertip on the object of the floating-in-space image 3, detected by the mid-air operation detection sensor 1351 and the mid-air operation detection unit 1350. The fingertip haptic generation unit 1230 has a function of emitting ultrasonic waves of a predetermined sound pressure from the ultrasonic element array toward the touch position. This ultrasonic wave has a very strong directivity. When this ultrasonic wave hits the fingertip of the user U, the user U can get a touch sensation with his / her fingertip as if he / she had touched some actual object. As a result, even when the user U performs a touch operation as an mid-air operation on an object such as a push button displayed as the floating-in-space image 3 that is actually nothing but air, he / she can get a touch sensation as if his / her fingertip had touched a physical button. That is, the user U can feel that he / she has performed a touch operation more reliably.

[0042] Furthermore, in one embodiment, the fingertip tactile sense generating unit 1230 has a function of modulating an ultrasonic signal with an audio signal in an audible band, in other words, a function of superimposing an audio signal on an ultrasonic wave. As a result, when an ultrasonic signal is modulated with an audio signal, a sound based on the audio signal is emitted from the vicinity of the fingertip where the user U touches an object. This sound travels in all directions. The user U can hear this sound in conjunction with the touch operation. Therefore, the user U can not only feel a touch sensation on his / her fingertip, but also can more reliably recognize the touch operation he / she has performed from the sound emitted near the fingertip.

[0043] The fingertip tactile sense generation unit 1230 may be configured as a part of the housing 1190, or may be configured as a device having a housing separate from the housing 1190. The fingertip tactile sense generation unit 1230 may be disposed at a predetermined position outside the housing 1190, or may be configured such that its position and orientation can be variably adjusted. The fingertip tactile sense generation unit 1230 in Fig. 1 corresponds to the fingertip tactile sense generation device 6 in Fig. 17 described later.

[0044] The non-volatile memory 1108 stores and holds various data used in the space-floating image display device 1000. The data stored in the non-volatile memory 1108 includes, for example, video / image data for displaying the space-floating image 3. This data includes data for configuring objects (characters, buttons, icons, etc.) to be displayed as at least a part of the space-floating image 3. This data may include data for various operations, layout information of the objects, metadata of the objects, control information, related information, etc. The memory 1109 stores video data for displaying as the space-floating image 3, data for controlling the space-floating image display device 1000, etc. Note that the video / image may be a still image or a moving image.

[0045] The control unit 1110 corresponds to a controller of the space floating image display device 1000 and controls the operation of each unit connected to it. The control unit 1110 includes a processor and performs calculations and the like based on information acquired from each unit in the space floating image display device 1000 in cooperation with a program stored in the memory 1109.

[0046] The communication unit 1132 communicates with external devices, servers, etc. via a wired or wireless communication interface. Various data such as video / image data, audio data, etc. are sent and received by communication via the communication unit 1132. The space floating image display device 1000 may obtain instructions, video data, etc. from the outside via the communication unit 1132, and may output and store information on the result of user operations, etc., to the outside.

[0047] The storage unit 1170 is a storage device that records various data and information such as video / image data, audio data, etc. The storage unit 1170 may hold the same data as the non-volatile memory 1108. For example, various data and information such as video / image data, audio data, etc. may be recorded in advance in the storage unit 1170 at the time of product shipment. The storage unit 1170 may record various data and information such as video / image data, audio data, etc. acquired from an external device, server, etc. via the communication unit 1132. The audio data recorded in the storage unit 1170 can be output as audio through the audio signal output unit 1140, for example.

[0048] Video / image data and audio data are associated in a specific way. For example, a certain push button object is an object on which characters such as "A", "1", "Yes", "No", and "Redo" are displayed, and the image data associated with that object has character information and display control information for those characters, or has character images of those characters. The audio data associated with the image data of that object has audio data for the characters being emitted as audio in response to a specific operation on that object. Also, another object is an object on which no characters are displayed, but even in that case, audio data for the audio being emitted in response to a specific operation on that object can be associated.

[0049] The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. For example, the video control unit 1160 creates a video signal, switches the video signal, and performs other controls. For example, the video control unit 1160 performs control to switch the source of the video signal to be input to the video display unit 1102 from among the memory 1109, the storage unit 1170, the video signal input unit 1131, the communication unit 1132, and the like. The video control unit 1160 can also generate a superimposed video signal by superimposing the video signal on the memory 1109 and the video signal input from the video signal input unit 1131, and control the superimposed video signal to be input to the video display unit 1102. As a result, a composite video based on the superimposed video signal can be formed as the space floating video 3.

[0050] The video control unit 1160 may also control image processing for the video signals of each source. Examples of image processing include scaling processing for enlarging, reducing, transforming, etc. an image, brightness adjustment processing for changing luminance, contrast adjustment processing for changing the contrast curve of an image, and Retinex processing for decomposing an image into light components and changing the weighting of each component. The video control unit 1160 may also perform special effect video processing, etc., for the video signal input to the video display unit 1102 to assist the user U in performing an aerial operation, such as a touch operation on the floating-in-space video 3.

[0051] The special effect video processing is performed, for example, based on the detection result of the touch operation by the aerial operation detection unit 1350 and the captured image of the user U by the imaging unit 1180. Examples of the special effect video may include an animation in which the button sinks in the depth direction when a push button object is touched, or an animation in which ripples are generated around the button on the surface of the floating in space video 3, in order to emphasize the touch operation.

[0052] The space-floating image display device 1000 (particularly the image control unit 1160) creates data for displaying the space-floating image 3 (in other words, image signals) based on the image and image data of the memory 1109, the storage unit 1170, etc., and inputs it to the image display unit 1102 of the image display device 1. Then, the image light generated and emitted by the image display unit 1102 is reflected by the retroreflective unit 1101 and emitted as image light with high directivity toward the outside of the space-floating image display device 1000. As a result, the space-floating image 3 is output and formed at a predetermined position outside. As shown in FIG. 2, a regular user U facing this space-floating image 3 can view the space-floating image 3 in the direction of the arrow A from an eye point UP (in other words, the eye) corresponding to a predetermined standard position in the space.

[0053] As described above, various functions are installed in the space-floating image display device 1000. However, the space-floating image display device 1000 does not need to have all of these functions, and only needs to have the function of forming the space-floating image 3, and various forms are possible.

[0054] <Example 1 of a space floating image display device> Fig. 2 shows the main components of the space-floating image display device 1000 according to an embodiment, and a configuration example of a retroreflector 1101. Fig. 2 shows the configuration when viewed from the side in the direction in which the space-floating image display device 1000 and a regular user U face each other in space. As shown in Fig. 2, a display device 1 (image display device) that diverges specific polarized image light at a narrow angle is provided in the oblique direction (direction having an angle A with respect to the horizontal plane) of a transparent member 100 arranged on a horizontal plane. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light having a narrow-angle diffusion characteristic.

[0055] Image light of a specific polarization from display device 1 is reflected by polarization separation member 101, which is provided on transparent member 100 (e.g., glass) and has a film that selectively reflects image light of a specific polarization, and enters retroreflective member 2. Retroreflective member 2 is disposed in the other diagonal direction (direction having angle B with respect to the horizontal plane) of transparent member 100. In the drawing, polarization separation member 101 is a sheet-shaped polarization separation member adhered to the back side of transparent member 100 (the lower surface side in the vertical direction).

[0056] A λ / 4 plate 2b is provided on the image light incident surface of the retroreflective member 2 as a wavelength plate. The image light passes through the λ / 4 plate 2b twice, at the time of incidence to the retroreflective member 2 and at the time of emission. As a result, the image light is polarized and converted from a specific polarized wave (in other words, one polarized wave) to the other polarized wave. Here, the polarization separation member 101, which selectively reflects the image light of the specific polarized wave, has the property of transmitting the polarized light of the other polarized wave after the polarization conversion. Therefore, the image light of the specific polarized wave after the polarization conversion is transmitted to the outside through the polarization separation member 101 in an oblique direction corresponding to the angle B. The image light transmitted through the polarization separation member 101 forms a space-floating image 3, which is a real image, at a predetermined position outside the transparent member 100. The space-floating image 3 in this example is arranged so that its main plane is oblique to the horizontal plane of the transparent member 100 at an angle C (an angle corresponding to the angle B). As shown in the figure, the mid-air operation detection sensor 1351 in this example is disposed in a position near the transparent member 100 on an extension of the plane of the floating-in-space image 3. The main plane of the floating-in-space image 3 also corresponds to the range where touch operation is possible.

[0057] The light forming the floating image 3 is a collection of light rays that converge from the retroreflective member 2 to the optical image of the floating image 3. These light rays continue to travel straight even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is an image with high directivity, unlike the diffuse image light formed on a screen by a general projector or the like. Therefore, in the configuration of FIG. 2, when the user U (corresponding eye point UP) views the floating image 3 from the direction of the arrow A corresponding to the direction of the highly directional image light (shown by the dashed line arrow), the floating image 3 is viewed as a bright image. On the other hand, when another person views the floating image 3 from a direction different from the direction of the arrow A, for example, the direction of the arrow B (the horizontal direction facing the floating image 3 from the back), the floating image 3 cannot be viewed as an image at all. This characteristic is very suitable for use in a system that displays images that require high security or highly confidential images that should not be seen by people facing the user U.

[0058] Depending on the performance of the retroreflective member 2, the polarization axis of the reflected image light may become misaligned. In this case, a part of the image light with the misaligned polarization axis is reflected by the above-mentioned polarization separation member 101 and returns to the display device 1. This light is reflected again on the image display surface of the liquid crystal display panel 11 constituting the display device 1, which may cause a ghost image (FIG. 5) and reduce the image quality of the spatial floating image 3. Therefore, in this embodiment, an absorbing polarizing plate 12 is provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorbing polarizing plate 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorbing polarizing plate 12. This makes it possible to suppress the above-mentioned re-reflection, and as a result, it is possible to prevent the image quality from being reduced by the ghost image. The polarization separation member 101 may be formed, for example, of a reflective polarizing plate or a metal multilayer film that reflects a specific polarized wave.

[0059] In this embodiment, a superdirectional speaker 30 (corresponding to the superdirectional speaker 1242 in FIG. 1) is disposed on the transparent member 100 in FIG. 2 or in a predetermined position around the transparent member 100 so as not to obstruct the optical path for forming the floating-in-space image 3. The superdirectional speaker 30 outputs superdirectional sound in the direction of the dashed-dotted arrow in the figure (i.e., toward the ear UE). As described above, the superdirectional speaker 30 has a characteristic that the user U can hear the sound in the audible band only in a very limited spatial region near the ear UE of the user U, and the people around the user U cannot hear the sound. This characteristic is particularly suitable for outputting highly confidential sound information (such as a PIN number or an amount of money) by reading it out.

[0060] Next, FIG. 3 shows the surface shape of a retroreflective member manufactured by Nippon Carbide Industrial Co., Ltd. as a typical example of the configuration of the retroreflective member 2. The surface of this retroreflective member has a plurality of regularly arranged hexagonal columns as retroreflective parts 2a (in other words, retroreflective elements). A light ray incident on the inside of the hexagonal column is reflected by the wall and bottom surfaces of the hexagonal column and is emitted as retroreflected light in a direction corresponding to the incident light. As a result, a normal image R1 as shown in FIG. 5 is formed as a space floating image 3. On the other hand, as shown in FIG. 4, ghost images G1 to G6 are formed in addition to the normal image R1 depending on the image light from the display device 1 that is obliquely incident on the retroreflective member 2 (hexagonal column).

[0061] The space-floating image display device 1000 according to the embodiment displays a space-floating image 3, which is a real image, based on an image displayed on the display device 1. The resolution of this space-floating image 3 depends greatly on the diameter D and pitch P of the hexagonal prisms that are the retroreflective portion 2a of the retroreflective member 2 shown in FIG. 3, in addition to the resolution of the liquid crystal display panel 11. For example, when a 7-inch WUXGA (1920×1200 pixels) liquid crystal display panel 11 is used, even if one pixel (corresponding one triplet) is about 80 μm, if the diameter D is 240 μm and the pitch P is 300 μm, one pixel of the space-floating image 3 is equivalent to 300 μm. Therefore, the effective resolution of the space-floating image 3 is reduced to about 1 / 3.

[0062] Therefore, in order to make the resolution of the spatial floating image 3 equal to that of the display device 1, it is desirable to configure the diameter D and pitch P of the retroreflective portion 2a to be close to one pixel of the liquid crystal display panel 11. On the other hand, in order to suppress the occurrence of moire caused by the retroreflective portion 2a of the retroreflective member 2 and the pixels of the liquid crystal display panel 11, it is preferable to design the pitch ratio of each to a value other than an integer multiple of one pixel. In addition, it is preferable to arrange the shape so that none of the sides of the retroreflective portion 2a overlaps with any of the sides of one pixel of the liquid crystal display panel 11.

[0063] On the other hand, in order to manufacture the retroreflective member 2 at a low cost, it is preferable to use a roll press method. Specifically, this is a method in which the retroreflective portions 2a are aligned and shaped on a film. In this method, the inverse shape of the shape to be shaped is formed on the roll surface, and a UV-curable resin is applied onto a base material for fixing and passed between the rolls to form the required shape, and then the resin is irradiated with UV rays to cure. In this way, a retroreflective member 2 of the desired shape can be obtained.

[0064] In this embodiment, the display device 1 is configured using a liquid crystal display panel 11 and a light source device 13 that generates light of a specific polarization having a narrow-angle diffusion characteristic described later. This reduces the possibility that an image will be obliquely incident on the retroreflective member 2. This makes it possible to provide a structurally excellent system in which even if a ghost image occurs, the brightness of the ghost image is low.

[0065] <Example of a space floating image display device 2> 6 shows another example of the main components of the space floating image display device according to the embodiment. The display device 1 is configured with a liquid crystal display panel 11 as an image display element, and a light source device 13 that generates light of a specific polarized wave having a narrow-angle diffusion characteristic. The liquid crystal display panel 11 is configured with a screen size selected from small ones having a screen size of about 5 inches to large ones having a screen size of more than 80 inches. The image light from the liquid crystal display panel 11 is reflected toward the retroreflective member 2 by a polarization separation member 101 such as a reflective polarizing plate.

[0066] A λ / 4 plate 2b is provided on the light incidence surface of the retroreflective member 2, and the image light is passed through it twice to perform polarization conversion (converting a specific polarized wave into the other polarized wave). As a result, the image light after polarization conversion passes through the polarization separation member 101 and forms a space-floating image 3, which is a real image, outside the transparent member 100. In the polarization separation member 101, the polarization axis may become uneven due to retroreflection, and a part of the image light is reflected and returned to the display device 1. This light is reflected again by the image display surface of the liquid crystal display panel 11, which may result in the generation of a ghost image and significantly degrade the image quality of the space-floating image 3. Therefore, in this embodiment, an absorbing polarizing plate 12 is provided on the image display surface of the display device 1. The image light is transmitted through the absorbing polarizing plate 12, and the reflected light is absorbed by the absorbing polarizing plate 12. This prevents the image quality of the space-floating image 3 from being degraded by the ghost image.

[0067] Furthermore, in this embodiment, in order to reduce degradation of image quality due to external light such as sunlight or illumination outside the set of the spatial floating image device 1000, it is advisable to provide an absorptive polarizing plate 12B as shown in the figure on the surface (external light incident surface) of the transparent member 100. The polarization separation member 101 can be formed, for example, of a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

[0068] 6 and 2, a difference is that a light shielding member 24 and a light shielding member 23 are provided in the middle of the optical path between the polarized light separation member 101 and the liquid crystal display panel 11 to shield oblique image light other than normal image light that forms the space floating image 3. The light shielding member 24 is disposed at a distance L2 on the side closer to the liquid crystal display panel 11, and the light shielding member 23 is disposed on the side closer to the polarized light separation member 101. Also, a light shielding member 22 is provided in the middle of the optical path between the retroreflective member 2 and the polarized light separation member 101 to shield oblique image light other than normal image light. The light shielding member 22 is disposed at a distance L1 from the retroreflective member 2. These light shielding members shield oblique image light that causes ghost images. The dashed lines in the light shielding members 22, 23, and 24 indicate the transmission (non-shielding) areas. As a result, the generation of ghost images as described above can be further suppressed.

[0069] The inventors have confirmed through experiments that the effect of shading can be enhanced by providing the light shielding member 24 and the light shielding member 23 in the space between the liquid crystal display panel 11 and the polarization separation member 101. In this experiment, it was confirmed that the inner diameter (diameter of the dashed line portion) of the light shielding member 23 and the light shielding member 24 can be manufactured and assembled with a part precision within the range of mechanical tolerance by setting the inner diameter (diameter of the dashed line portion) to 110% in area of ​​the area through which the normal image light beam that forms the space floating image 3 passes. In addition, it was confirmed that in order to further reduce the occurrence of ghost images, the occurrence of ghost images can be suppressed to a level that does not cause practical problems if the inner diameter is set to 104% or less of the area through which the normal image light beam of the light shielding member passes. On the other hand, if the light shielding member 22 provided between the retroreflective member 2 and the polarization separation member 101 is installed at a position where the distance L1 between the light shielding member 22 and the retroreflective member 2 is 50% or less of the distance between the retroreflective member 2 and the polarization separation member 101, the occurrence of ghost images can be further reduced. Furthermore, if the light blocking member 22 is placed at a position where the distance L1 between the light blocking member 22 and the retroreflecting member 2 is 30% or less of the distance between the retroreflecting member 2 and the polarization separating member 101, the occurrence of ghost images can be reduced to a level that is practically ok when viewed visually. It was confirmed that the level of ghost images can be further reduced by arranging the above-mentioned three light blocking members 22, 23, and 24 side by side.

[0070] Fig. 7 shows another example of the configuration of the light blocking member for reducing the occurrence of ghost images in the space floating image display device of one embodiment. Fig. 7(A) shows the shape of the cross section of the light blocking member 25 in this embodiment. Fig. 7(B) shows the shape of the plane (surface viewed in a direction perpendicular to the optical axis) of the light blocking member 25, and shows the effective area of ​​the light blocking member 25 with respect to the area 27 through which the normal image light beam that forms the space floating image 3 passes. In the example of Fig. 7(B), the area of ​​the area 27 through which the normal image light beam passes is set to be smaller than the area of ​​the inner diameter of the outer frame 25a (area having light blocking properties) of the light blocking member 25.

[0071] FIG. 8 shows another example of a light shielding member that reduces the occurrence of ghost images in the space floating image display device of one embodiment. FIG. 8 (A) shows the cross-sectional shape of the light shielding member 26 in this embodiment. FIG. 8 (B) shows the planar shape of the light shielding member 26, and shows a configuration in which the effective area of ​​the light shielding member 26 is approximately the same size as the area 27 through which the normal image light beam that forms the space floating image 3 passes. In this example, a beam 26b is provided from the outer frame 26a of the light shielding member 26 toward the inside. The tip of the beam 26b extends to the outer shape of the area 27. This causes abnormal light such as external light that causes the formation of ghost images to be reflected multiple times on the surface of the beam 26b. This allows the abnormal light to be further absorbed. In this example, the area of ​​the area 27 through which the normal image light beam passes is smaller than the area of ​​the inner diameter of the outer frame 26a of the light shielding member 26, and is set to be equal to the area of ​​the surface inscribed by the beam 26b.

[0072] In the above embodiment, the shape of the main surface of the retroreflective member 2 is a planar shape facing the display device 1 (including reflection in the optical path). On the other hand, as a modified example, the shape of the retroreflective member 2 may be a concave or convex surface with a curvature radius of, for example, 200 mm or more, from the planar shape facing the display device 1. In this configuration, even if a ghost image occurs due to the oblique image light reflected by the retroreflective member 2, the ghost image generated after reflection can be made invisible by moving it away from the field of view of the user U. If the curvature radius is set to 100 mm or less, the amount of light reflected normally among the light reflected at the periphery of the retroreflective member 2 (the periphery of the area where the λ / 4 plate 2b is arranged in FIG. 2 and the like, which mainly reflects the image light) decreases, and a new problem occurs in that the peripheral light amount of the obtained floating image 3 decreases. For this reason, in order to reduce the ghost image to a level that does not cause a practical problem, it is good to select and apply the above-mentioned technical means or use them in combination.

[0073] <Image display method of the space floating image display device> FIG. 9 is an explanatory diagram of an image display method for preventing erroneous input in a space-floating image display device according to an embodiment. FIG. 9 shows an example of a space-floating image 3 displayed on a space-floating image display device 1000, and shows how the space-floating image 3 appears when viewed from the viewpoint of a user U. This example of the space-floating image 3 corresponds to a non-contact user interface having a plurality of objects such as a numeric keypad, and capable of inputting numbers and the like. As described above, the display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates light of a specific polarized wave having a narrow-angle diffusion characteristic, and can be configured with a screen size selected from a small one having a screen size of about 5 inches to a large one having a screen size of more than 80 inches. For example, the image light from the liquid crystal display panel 11 is reflected toward the retroreflective member 2 by a polarized light separation member 101 such as a reflective polarizing plate.

[0074] FIG. 10 is a schematic diagram showing how the user U views the space-floating image 3 when the space-floating image display device 1000 is viewed from the side. In particular, the user U's eye point UP is in three different positions in the height direction, namely, upper, middle, and lower, and the user U views the space-floating image 3 in each of the line-of-sight directions LU, LM, and LL. In this example, the angles A, B, and C described above (FIG. 2) are about 45 degrees. Since the space-floating image 3 is formed by light rays with high directivity, when the space-floating image 3 is viewed from the viewing direction shown in FIG. 10 (for example, line-of-sight direction LM corresponding to the middle position), the space-floating image 3 is viewed as a bright image. In particular, when the user U operates the space-floating image display device 1000 at the correct position and the line of sight is in the line-of-sight direction ML, the user U can view all of the space-floating image 3 favorably, as shown in FIG. 9.

[0075] On the other hand, as a result of experiments by the inventor, it was found that in a configuration combining a light source device 13 with a narrow-angle diffusion characteristic and a liquid crystal display panel 11 as an image display element, when the position of the user U's eyes (eye point UP) is not in the optimal position (the position corresponding to the line of sight LM in Fig. 10), for example, when viewing along the line of sight LU on the upper side in Fig. 10, it is difficult to see the lower image of the floating in space image 3. Considering this from the opposite perspective, there is little risk that people other than the legitimate user U will peek at the floating in space image 3, and it can be said to be effective from the viewpoint of security, etc.

[0076] On the other hand, the present inventor has studied a method for easily performing touch operations on the space-floating image 3 from an optimal viewing position. In the space-floating image display device 1000 of this embodiment, as shown in FIG. 10, a camera unit 55 (corresponding to the imaging unit 1180 in FIG. 1) for determining whether the user U's standing position is appropriate or not is provided in a part (the rear position as seen from the user U) of the housing 50 (housing 1190 in FIG. 1) of the space-floating image display device 1000. The space-floating image display device 1000 determines the position and direction of the user U's face in the space, and further the position of the pupil (eye point UP) as necessary, by the camera unit 55. Then, the space-floating image display device 1000 changes the display state of the optimal viewing position display units 3a, 3b, 3c, and 3d, which are frame display units arranged on the four outer sides of the image area of ​​the main objects such as the numeric keypad, in the space-floating image 3 of FIG. 9. This guides the position and line of sight of the user U to the line of sight direction ML corresponding to the optimal position. For example, when the position of the eye point UP of the user U is below the optimal position and the viewing state is in a line of sight LL, the space floating image display device 1000 switches the display state of the four optimal viewing position display sections, such as turning off the optimal viewing position display section 3a corresponding to the upper side of the four sides. The state in which the four optimal monitoring position display sections 3a, 3b, 3c, and 3d are all visible from the user U's perspective, that is, the state in which it looks like a picture frame, corresponds to the optimal viewing state. This allows the user U's line of sight to be guided to the line of sight ML corresponding to the optimal position. As another example of display control, it is also possible to display an arrow image or the like that encourages the user to move the viewpoint in the frame display section.

[0077] In addition, the space-floating image display device 1000 automatically adjusts the brightness of the displayed image based on the information on the brightness of the outside world detected by the camera unit 55, thereby adjusting the brightness, etc. of the space-floating image 3, thereby reducing power consumption and improving visibility.

[0078] Furthermore, it was also found that when the user U uses the space-floating image display device 1000, it may be difficult to recognize and distinguish the position in the space where the space-floating image 3 is floating. In order to solve this problem, the inventor devised an embodiment as shown in FIG. 11 and FIG. 12. In this embodiment, a member having a physical substance (other than air) for position recognition, for example, a transparent structural member 60 made of plastic, is provided at the position where the space-floating image 3 is displayed. In FIG. 11, a frame-shaped transparent structural member 60 is arranged on the four outer sides of the space-floating image 3 (including the frame display portion) similar to that in FIG. 9. For example, the lower side of this transparent structural member 60 is fixed to one side of the front side of the housing 50. It was found that this makes it easier for the user U to recognize and distinguish the display position of the space-floating image 3 in the three-dimensional space. Note that instead of the transparent structural member 60, a non-transparent frame-shaped structural member made of metal, for example, may be provided. When a frame-shaped non-transparent structural member is provided, the user U can easily recognize the non-transparent structural member, and can more quickly recognize the display position of the floating image 3 even from a sideways direction or diagonally upward direction several meters away. After the recognition, the user U can operate the floating image 3 with his / her fingers from a front position.

[0079] In the configuration example of FIG. 11 and FIG. 12, a TOF sensor 56 (a distance sensor of a TOF method) is provided as the aerial operation detection sensor 1351. The TOF sensor 56 senses the relationship between the TOF sensor 56 and an object such as the finger of the user U, and the position of the object, for example, with respect to the floating in space image 3. This TOF sensor 56 is disposed at the bottom of the floating in space image 3. This TOF sensor 56 is disposed in a part of the housing 50 (50a) corresponding to the lower side part of the transparent structural member 60. This TOF sensor 56 can detect the state of the touch position of the fingertip, etc. by scanning the entire area within the plane of the floating in space image 3. This TOF sensor 56 and the control unit 1110 using it can detect and measure the position coordinates (e.g., touch position) of an object such as a finger in the coordinate system in the plane direction of the floating in space image 3, and can also detect the state of the object, such as the moving direction and moving speed. The TOF sensor 56 has a configuration in which multiple combinations of infrared light emitters and light receivers are arranged in a straight line in order to read the distance and position on a two-dimensional plane, as shown in FIG. 11. The TOF sensor 56 receives the light reflected from the infrared light emitter by the target object and multiplies the time difference from light emission to light reception by the speed of light to clarify the distance to the target object. In addition, the coordinates on the plane of the floating image 3 in space can be read from the coordinates of the part where the time difference is smallest among the multiple infrared light emitters and light receivers on the straight line.

[0080] As another sensing method different from the above example, a method using a configuration including an infrared laser light generating unit and an imaging unit using an infrared camera may be used. Here, the shape of the irradiation area of ​​the infrared laser light generated from the infrared laser light generating unit is a thin sheet shape, and the shape is close to the display surface of the floating in space image 3 with a gap of, for example, several mm or less. This irradiation area of ​​the infrared laser light is also called a "laser sheet." The infrared camera of the imaging unit captures the light reflected by an object such as the finger of the user U from a laser sheet formed by the infrared laser light generated from the infrared laser light generating unit (in other words, a laser sheet generating unit). Then, the control unit 1110 can identify the position of the finger on the laser sheet by analyzing the captured image generated by the imaging unit.

[0081] Furthermore, a configuration of sensors that can detect the position coordinates of an object on a two-dimensional plane as described above may be combined in a vertical direction (in other words, the depth direction or front-back direction) to the surface of the floating image 3. This makes it possible to obtain information such as the position coordinates of an object in a three-dimensional area including the vertical direction of the floating image 3. In other words, the state of an operation such as touching the floating image 3 with a finger or the like can be obtained in more detail.

[0082] The inventor also considered a display method that allows the user U to more clearly view the floating image 3. As a result, as shown in FIG. 9 and other figures, if a part of the floating image 3 is designed to overlap the outer frame 50a of the housing 50 on the outer periphery of the transparent member 100, which is the window through which the image light is emitted, the floating image 3 can be more clearly viewed when viewed from the user U. In addition, in order to make the floating amount of the floating image 3 appear larger, the layout of the entire optical system can be designed so that the lower end of the floating image 3 overlaps the outer frame 50a.

[0083] <Example 3 of a floating image display device: super-directional speaker> Next, the inventors have studied a configuration that can prevent erroneous input such as touch operation on an object that a user wants to select when the object displayed as a space floating image formed by a space floating image display device is used as a non-contact user interface and can perform input operation reliably. In order to achieve this, the inventors have studied a method of outputting sound in a suitable manner corresponding to the user's input operation, for example, a method of providing user operation assistance or operation guidance by sound.

[0084] Here, in a system that handles highly confidential information such as a PIN number or input amount information, such as a bank ATM device, simply outputting a sound corresponding to an object (e.g., a numeric button) selected by a user through a touch operation or the like from a normal speaker poses a major problem in terms of security, etc. Therefore, the present inventor has investigated a sound output method that uses a superdirectional speaker (superdirectional speaker 1242 in FIG. 1, superdirectional speaker 30 in FIG. 2) to enable only authorized users to hear the sound.

[0085] As shown in Fig. 2 and other figures, when viewed from the user U side (the direction of the arrow A), the planar area where the floating image 3 is displayed does not actually contain any objects (especially solids or liquids) other than air. Therefore, we considered a configuration in which the superdirectional speaker 1242 (Fig. 1) is placed at a position off the optical path for forming the floating image 3, so that the sound can be heard only near the user's ears. As a result, we considered two methods, for example, as shown in Figs. 13 and 14.

[0086] FIG. 13 shows a configuration example in which the superdirectional speaker 30 is arranged on the outer frame 50a of the housing 50 in the space floating image display device 1000 according to an embodiment. An application example is an ATM device. In this configuration example, sound from the superdirectional speaker 30 is output directly toward the ear UE of the user U. The propagation path of the superdirectional sound is a straight path, in other words, a non-reflective path. In this configuration, the superdirectional speaker 30 forms a sound field 31 with strong directionality. The sound emitted from the superdirectional speaker 30 directly reaches the area near the ear UE of the user U. The area near the ear UE of the user U who is in the correct position is the area where the strong sound field 31 is formed. Therefore, only the user U can hear the sound, and others cannot hear it.

[0087] 14 shows a configuration example in which superdirectional speaker 30 is arranged on wall 57 on the opposite side to user U in space floating image display device 1000 according to an embodiment. In this configuration example, sound emitted from superdirectional speaker 30 is first reflected by the flat surface of transparent member 100 (e.g., glass), and the reflected sound reaches an area near user U's ear UE.

[0088] In both the configurations of FIG. 13 and FIG. 14, the superdirectional speaker 30 is disposed at a position that is off the optical path for forming the floating-in-space image 3 (such as the optical path of the image light that travels from the retroreflective member 2 through the transparent member 100 to the floating-in-space image 3 or the eye point UP). Therefore, the superdirectional speaker 30 does not obstruct the optical path for forming the floating-in-space image 3. Also, comparing these configurations, the optical path of the image light and the path of the sound have a higher similarity in direction in FIG. 14 than in FIG. 13. Therefore, the configuration of FIG. 14 is more effective in terms of making it easier to hear the sound as if it were coming from the floating-in-space image 3 from the position of the regular user U (the position in the line of sight LM).

[0089] Conventionally, a technology has been known in which a superdirectional speaker is configured with a plurality of ultrasonic output elements arranged regularly on a plane as an array, and the three-dimensional position (in other words, the sound field) at which the sound output from the superdirectional speaker can be heard is controlled by the phase difference (or time difference) of ultrasonic signals input to the plurality of ultrasonic output elements. Note that the technology related to the superdirectional speaker is described, for example, in the paper "Highly realistic sound field reproduction: Latest research trends using parametric speakers" (IEICE Fundamentals Review Vol. 10 No. 1, pages 57 to 64).

[0090] The superdirectional speaker 30 in the embodiment may apply a technology for forming a sound field like such a three-dimensional position. Also, a configuration may be applied in which a plurality of ultrasonic output elements are arranged on a concave curved surface, not on a flat surface, and the curvature of the concave surface is changed. This configuration also makes it possible to control a sound field like a three-dimensional position where the sound output from the superdirectional speaker 30 can be heard. By using the above technology, it is possible to form an optimal sound field so that the sound can be heard only in the area very close to the user's ear.

[0091] Therefore, the space floating image display device 1000 according to one embodiment has a configuration as shown in Fig. 15. In this configuration, for example, one superdirectional speaker 30 is arranged in a position on the back side of the housing 50 as seen from the user and in a central position in the left-right direction. The fixing of the superdirectional speaker 30 can be similarly applied to the configuration examples of Figs. 13 and 14. Furthermore, a camera 55CL is arranged on the left side of the superdirectional speaker 30 as seen from the user, and a camera 55CR is arranged on the right side. The cameras 55CL and 55CR are configuration examples of the imaging unit 1180 described above.

[0092] In this configuration example, stereo photography is possible using two cameras (55CL, 50CR) on the left and right. Therefore, based on the images captured by each of the two cameras, the position of the user's face in space and the distance from the superdirectional speaker 30 to the position of the face can be calculated and obtained. Based on the calculation result, the control unit 1110 controls the phase difference (or time difference) of the ultrasonic signals input to the multiple ultrasonic output elements constituting the superdirectional speaker 30. This makes it possible to form an optimal sound field so that the sound can be heard only in the area near the user's face or ears. As a result, the user can hear the sound from the superdirectional speaker 30 without being heard by others. In particular, when a non-contact user interface using the space floating image 3 is used in a system requiring high security such as an ATM device, the configuration as shown in FIG. 15 is very suitable from the viewpoint of security.

[0093] The ultra-directional speaker 30 and cameras 55CL, 55CR may be provided at a position away from the housing 50 of the space-floating image display device 1000, or may be installed in the housing 50, or may be fixed to the housing 50 at a predetermined position.

[0094] Moreover, the space floating image display device 1000 according to the embodiment has a configuration as shown in FIG. 16. In this configuration, a camera is built into the housing of the superdirectional speaker 30, in other words, the superdirectional speaker and the camera are integrated. At a position on the far side as seen from the user, the superdirectional speaker 30L is disposed on the left side, and the superdirectional speaker 30R is disposed on the right side. The superdirectional speaker 30L and the superdirectional speaker 30R each have a camera unit 55C built into their housing and are integrated, in other words, they are a speaker-camera unit.

[0095] This configuration is not only superior in terms of space factor due to integration, but also enables stereo photography using two cameras (camera unit 55C) on the left and right. Therefore, the user's face position and the distance from the ultrasonic speaker to the face position can be calculated and obtained based on the images of each camera unit 55C.

[0096] Moreover, this speaker-camera unit (super-directional speakers 30L, 30R) may be fixed at a predetermined position of the housing 50. In this case, the distance between the left and right units is always constant. Therefore, the positional relationship between the user, the camera, and the super-directional speaker can be calculated more accurately. Furthermore, in this configuration, the position of the sound field formed by the two super-directional speakers 30L, 30R can be calculated with high accuracy. This makes it possible to set with high accuracy the focal area of ​​the sound field formed by the two super-directional speakers 30L, 30R, that is, the area where only the authorized user can hear the sound. As a result, the sound generated by the sound field formed by the two super-directional speakers 30L, 30R can be heard only by the authorized user of the ATM device, for example, and cannot be heard by others in the vicinity of the user (for example, on the left, right, or rear).

[0097] 16, a sound field is formed for the user's left ear by left superdirectional speaker 30L, and a sound field is formed for the user's right ear by right superdirectional speaker 30R, making it possible to form an optimal sound field overall. This makes it possible to provide audio that is easier for the user to hear, compared to the case of one superdirectional speaker 30 in FIG. 15.

[0098] 15 and 16, the position of the user's face is determined by a stereo camera. However, the present invention is not limited to this method, and the position of the face, eyes, etc. may be determined by using a thermal sensor or the like instead of a camera.

[0099] <Example 4 of a floating image display device: Fingertip tactile sensation generation> Next, a configuration for generating a tactile sensation on a fingertip when operating on a space-floating image in a case where the space-floating image is applied as a non-contact user interface as a space-floating image display device according to an embodiment will be described. In this embodiment, when a user touches an object (e.g., a push button) displayed as a space-floating image, the device has a function of generating a touch sensation on the fingertip as if the user had actually touched some object. This function is realized by using the fingertip tactile sensation generating unit 1230 in FIG. 1 described above.

[0100] [Space-floating image display device] FIG. 17 shows a schematic configuration of the space floating image display device 1000 of this embodiment, seen from the side including the user U. For the sake of explanation, the coordinate system and directions of the space are indicated by (X, Y, Z). The Z direction is the vertical direction, the up-down direction, the X direction and the Y direction are the horizontal direction, the X direction is the left-right direction as seen from the user U, and the Y direction is the front-back direction and the depth direction. The coordinate system and directions of the space floating image 3 are indicated by (x, y, z). The x direction and the y direction are two orthogonal directions that constitute the main two-dimensional plane of the space floating image 3, the x direction is the horizontal direction (in other words, the horizontal direction in the screen), and the y direction is the vertical direction (the vertical direction in the screen). The z direction is the direction perpendicular to the two-dimensional plane, and is the front-back direction related to the intrusion and approach of the finger UH. This space floating image display device 1000 is implemented as a part of, for example, a bank ATM device.

[0101] In this example, the floating image 3 is formed in an oblique direction from the position on the front side of the housing 50 as seen from the user U, with the angle C (FIG. 2) being about 45 degrees. The user U views the floating image 3 from the eye point UP corresponding to the eyes. In FIG. 17, the position and line of sight of the eye point UP of the user U correspond to the optimal line of sight direction LM (FIG. 10) described above, and corresponds to the optical axis a3. The user U performs an operation such as a touch operation on the floating image 3 with the finger UH (particularly the fingertip Uf). FIG. 17 shows a state in which the fingertip Uf touches, for example, a center position P1 of the floating image 3. This position P1 can be expressed as a position coordinate (X, Y, Z) in space, or a position coordinate (x, y) in a two-dimensional coordinate system of the floating image 3, and they can be converted.

[0102] In the configuration of FIG. 17, a fingertip tactile sense generating device 6 is provided as an implementation example corresponding to the fingertip tactile sense generating unit 1230 in FIG. 1. This fingertip tactile sense generating device 6 is provided as a separate body on the outside of the main housing 50 of the floating-in-space image device 1000. The fingertip tactile sense generating device 6 has a housing separate from the housing 50, and the housing has, for example, a rectangular parallelepiped shape, and an output plane of the ultrasonic element array 61 is arranged on the surface. An ultrasonic signal generating circuit 62 and the like are built in the housing. The ultrasonic signal generating circuit 62 is connected to the control device 10 through a wired or wireless signal line / communication. In the arrangement example of FIG. 17, the fingertip tactile sense generating device 6 is installed on a stand 52 (a part of the housing 50 or an optional part) standing vertically at a position on the back side in the Y direction on a base 51 corresponding to the upper surface or outer frame of the housing 50.

[0103] The image display device 1, the retroreflective member 2, and the like are housed and fixed inside the main housing 50 (housing 1190 in FIG. 1) as the aforementioned components. The image display device 1 is composed of a light source device 13, a liquid crystal display panel 11, an absorptive polarizer 12, and the like. The retroreflective member 2 is provided with a λ / 4 plate 2b. In this embodiment, the control device 10 and the light blocking member 120 are also provided inside the housing 50. The light blocking member 120 has the functions as shown in FIG. 6 above.

[0104] The control device 10 is an example of an implementation of elements such as the control unit 1110 and the image control unit 1160 in Fig. 1, and can be implemented as a control board or the like. Each element such as the image display device 1, the sensor 4, the camera 5, and the fingertip tactile sense generating device 6 is connected to the control device 10 via signal lines and communication. The control device 10 may communicate with an external device (e.g., a server) to exchange data.

[0105] The sensor 4 is an implementation example of the midair operation detection sensor 1351 in Fig. 1, and for example, a TOF sensor can be applied. As shown in the figure, the sensor 4 is disposed at a position on the front side of the housing 50 in the Y direction, at a position extending from the plane of the floating image 3 in space, and the direction of the detection axis is parallel to the direction of the plane of the floating image 3 in space (y direction). The sensor 4 may be disposed in the center in the X direction, for example, as in Fig. 11 described above, or two may be disposed on the left and right. The angle C between the floating image 3 and the sensor 4 is about 45 degrees in this example.

[0106] The camera 5 is an example of the implementation of the imaging unit 1180 in Fig. 1. As shown in the figure, the camera 5 is installed at a position on the back side of the housing 50 in the Y direction. One or more cameras 5 may be arranged in the X direction, and in particular, a stereo camera may be configured with two cameras on the left and right. The imaging direction of the camera 5 is set so as to capture the face UF of the user U in a normal position, as shown by the dashed-dotted arrow, and the imaging range (in other words, the angle of view) is set as a range that covers at least a part of the floating-in-space image 3 and the face UF.

[0107] The fingertip tactile sensation generating device 6 includes an ultrasonic element array 61 and an ultrasonic signal generating circuit 62, the details of which are shown in Fig. 18. The ultrasonic element array 61 is an array in which a plurality of ultrasonic elements 63 are arranged on a plane as shown in Fig. 18, and outputs ultrasonic waves. The ultrasonic signal generating circuit 62 is a circuit that generates an ultrasonic drive signal c5 for driving the ultrasonic element array 61 based on the control of the control device 10.

[0108] FIG. 17 illustrates a case in which phase-controlled ultrasound (in other words, a group of ultrasound waves) emitted from an ultrasonic element array 61 forms a focus due to sound pressure at a position (e.g., position P1) where a fingertip Uf of a user U touches an object on the surface (xy) of the floating-in-space image 3.

[0109] In the configuration example of FIG. 17, the path of the ultrasonic waves output from the ultrasonic element array 61 of the fingertip tactile sense generating device 6 is a path reflected by the transparent member 100. The group of ultrasonic waves from the ultrasonic element array 61 is once reflected by the surface of the transparent member 100, and forms a focus at a position corresponding to the object and the fingertip Uf of the floating-in-space image 3 as shown in the figure. First, ultrasonic waves are output from the ultrasonic element array 61 in a diagonally downward direction as shown by an axis a4. The angle of this axis a4 is similar to the angle A of the axis a1 of the emission of the image light from the image display device 1. Next, the ultrasonic waves are reflected (almost totally reflected) by the upper surface of the transparent member 100, and take a path in a diagonally upward direction as shown by an axis a5. The angle of this axis a5 is similar to the angle B of the axis a2 of the emission of the image light from the retroreflective member 2. Then, the ultrasonic waves reach a position P1 at the center of the floating-in-space image 3 on the axis a5.

[0110] Both the axis a2 of the image light and the axis a5 of the ultrasonic wave are set to be perpendicular (angle α1=about 90 degrees) to the surface of the floating image 3. Thus, in this configuration example, the fingertip tactile sense generating device 6 is disposed at the illustrated position outside the housing 50 so as not to obstruct the optical path of the image light from the retroreflective member 2 in the housing 50. Furthermore, in this configuration example, ultrasonic waves from the ultrasonic element array 61 are irradiated in a direction substantially perpendicular to the back side of the floating image 3. This allows ultrasonic waves to be applied to the fingertip Uf of the user U from a perpendicular direction like the axis a5, which makes it possible to provide a more suitable touch feeling compared to the case of applying ultrasonic waves from other directions.

[0111] Furthermore, in this embodiment, the ultrasonic element array 61 is configured to be able to form an area where the sound pressure of the ultrasonic waves is relatively high like a focus based on control. FIG. 17 shows a case where the focus of the ultrasonic waves is formed at a position P1 in the center of the floating-in-space image 3. The ultrasonic element array 61 can form an area where the sound pressure is high as a focus at a position at a predetermined distance in the direction of the path (axis a4 and axis a5) of the output ultrasonic waves. The fingertip tactile sense generating device 6 can variably control the position of the focus of the ultrasonic waves. That is, the focus of the ultrasonic waves can be formed so as to match a desired area corresponding to the touch position of the fingertip Uf within the plane (xy) of the floating-in-space image 3. The sound pressure of the ultrasonic waves can impart a tactile sensation such as a touch feeling to the fingertip Uf when the user U performs a touch operation.

[0112] The performance, position, orientation, etc. of the ultrasonic element array 61 are designed so that a focus of maximum sound pressure can be formed at a position at a predetermined distance on the ultrasonic path between the ultrasonic element array 61 and the floating image 3. The predetermined distance is designed as a suitable distance based on the performance of the ultrasonic elements 63.

[0113] Furthermore, the surface of the ultrasonic element array 61 is designed in terms of size, shape, number of elements, etc. so that it can cover the formation of focal points in all touch operable areas on the surface of the floating image 3 in space.

[0114] In addition to the above-mentioned functions, the camera 5 of the imaging unit 1180 may be used for a function such as detecting that the user U has come to a predetermined position in front of the space-floating image display device 1000 by detecting the face of the user U. For example, when the space-floating image display device 1000 detects that a person such as the user U has come to a predetermined position based on the camera 5, it may start a predetermined control (for example, displaying the space-floating image 3 or outputting sound, etc.), and when it detects that the person has left the predetermined position, it may stop the predetermined control.

[0115] [Fingertip tactile sensation generation unit] FIG. 18 shows a configuration example of the fingertip tactile sense generating device 6 corresponding to the fingertip tactile sense generating unit 1230. The ultrasonic element array 61 has an array configuration in which a plurality of ultrasonic elements 63 are arranged at approximately equal intervals on a plane so that ultrasonic waves with a frequency of, for example, about 40 kHz can be generated. The number of ultrasonic elements 63 is set to N. As an example, N=223. In this example, the ultrasonic element array 61 is a circular array in which a plurality of ultrasonic elements 63 (N=223) are arranged concentrically. An ultrasonic phased array is configured by the array of these ultrasonic elements 62. The ultrasonic phased array is an ultrasonic element array in which the position of the ultrasonic focus formation can be controlled.

[0116] In addition, the shape of the arrangement of the multiple ultrasonic elements 63 in the ultrasonic element array 61 is not limited to a concentric shape, and may be, for example, a square, a rectangle, a polygon, etc. In addition, the arrangement of the multiple ultrasonic elements 63 is generally arranged at approximately equal intervals without any gaps, but is not limited to this.

[0117] An example of an applicable ultrasonic element 63 (in other words, an ultrasonic transducer) is the MA40S4S manufactured by Murata Manufacturing Co., Ltd. In this example configuration of an ultrasonic element, the piezoelectric ceramics are housed in a cylindrical case with a diameter of about 1 cm for each element, and two terminals protrude from the case (specifications of the MA40S4S: https: / / www.murata.com / en-us / products / productdata / 8797589340190 / MASPOPSE.pdf).

[0118] As is well known, piezoelectric ceramics expand and contract when a voltage is applied, and their shape changes. When an ultrasonic AC voltage of, for example, 40 kHz is applied to the piezoelectric ceramics, the piezoelectric ceramics generates ultrasonic waves at the frequency of the ultrasonic AC voltage. In the case of the MA40S4S, which is the ultrasonic element 63 used in this embodiment, the basic performance is that when an ultrasonic voltage of 10 Vrms with a frequency of 40 kHz is applied between the above terminals, a sound pressure of about 120 dB can be obtained at a position 30 cm from the output side of the ultrasonic element 63.

[0119] As will be described later, the ultrasonic signal generating circuit 62 changes the phase (in other words, delay time) of the ultrasonic drive signal applied to the ultrasonic element 63 so that the ultrasonic waves emitted by each ultrasonic element 63 constituting the ultrasonic element array 61 reinforce or weaken each other. This makes it possible to form a point where the sound pressure of the ultrasonic waves is the strongest at an arbitrary point or small area on the ultrasonic element array 61. When the coordinate system of the plane of the array as shown in FIG. 18 is (Ax, Ay), this arbitrary point on the ultrasonic element array 61 corresponds to the position of the ultrasonic element 63 in that plane. For example, in the standard case, the area including the central point shown by the solid circle in the figure can be set as the point where the sound pressure is the strongest (corresponding to the area where the focus is formed). In addition, by controlling the ultrasonic drive signal c5, it is also possible to set, for example, an area shown by the dashed circle in the figure (for example, the upper side in the Ay direction). That is, in the example of FIG. 17, the point where the sound pressure is the strongest can be formed as a focus, for example, from the central position P1 to the position P2 on the upper side within the plane of the floating-in-space image 3.

[0120] In addition, the fingertip tactile sense generating device 6 can also form the point where the sound pressure is strongest as a focus at a position at a predetermined distance on the path of the ultrasonic waves (axis a4 and axis a5) by controlling the ultrasonic drive signal c5. In the example of Fig. 17, the point where the sound pressure is strongest can be formed as a focus not only at position P1 on the plane of the floating image 3 in space, but also at positions shifted to some extent in the z direction before and after that.

[0121] The floating-in-space image display device 1000 (particularly the control device 10) detects the touch position (in other words, the fingertip position) of the user U's fingertip Uf on the surface of the floating-in-space image 3 based on the sensor 4. The floating-in-space image display device 1000 then controls the output of ultrasonic waves from the fingertip tactile sense generating device 6 to match the touch position, so that the area with the highest sound pressure is formed near the fingertip Uf. As a result, the user U can feel the sound pressure of the ultrasonic waves on the fingertip Uf during the touch operation. In other words, the user U can get a touch sensation as if he or she is touching an object in the air, when there is actually nothing there.

[0122] The ultrasonic signal generating circuit 62 in Fig. 18 has a circuit group for generating and supplying an ultrasonic drive signal c5 to the ultrasonic element array 61 based on the control of the control device 10. This circuit group generates an ultrasonic drive signal c5 to be applied to each ultrasonic element 63 constituting the ultrasonic element array 61. The ultrasonic drive signal c5 is input from two terminals for each ultrasonic element 63.

[0123] The circuit group of the ultrasonic signal generating circuit 62 includes, in order from the input side to the output side, an ultrasonic carrier signal generating circuit 621, a square wave generating circuit 622, a phase shift circuit 623, an amplitude (AM) modulation circuit 624, and an inductance circuit 625. The ultrasonic carrier signal generating circuit 611 is an oscillator circuit that generates an ultrasonic carrier signal c1 having a frequency of, for example, 40 kHz. The generated ultrasonic carrier signal c1 is input to a square wave generating circuit 612. The square wave generating circuit 612 converts the ultrasonic carrier signal c1 into an ultrasonic carrier signal c2, which is a square wave. The ultrasonic carrier signal c2, which is a square wave output from the square wave generating circuit 612, is input to a phase shift circuit 613.

[0124] The phase shift circuit 613 is a circuit that generates an ultrasonic carrier signal c3, which is a rectangular wave in an ultrasonic band having multiple (e.g., eight) different phases. The phase shift circuit 623 is a circuit that generates a signal for forming a sound pressure (focus) according to the fingertip position. Here, "having different phases" is synonymous with "having different delay times." That is, for example, eight types of ultrasonic signals c3 are ultrasonic signals having eight different delay times. For example, in N ultrasonic carrier signals c3 corresponding to N ultrasonic elements 63, each ultrasonic carrier signal c3 is a signal having one phase selected by control from the multiple (e.g., eight) phases. By controlling the phase or delay time in this way, a focus can be formed at any point on the ultrasonic element array 61 where the sound pressure is strongest.

[0125] The phase shift circuit 623 and the inductance circuit 625 receive sensor detection information b1 from the input terminal. The sensor detection information b1 is detection information by the sensor 4 in FIG. 17 related to the user U performing an operation such as touching an object in the floating-in-space image 3, or information resulting from processing by a processor or the like based on the detection information. The sensor detection information b1 includes information such as the position of the fingertip Uf in FIG. 17, including, for example, the position coordinates (X, Y, Z) or (x, y) of the position P1. In this example, the sensor detection information b1 uses the two-dimensional coordinates (x, y) of the position of the fingertip Uf when the floating-in-space image 3 is a two-dimensional plane.

[0126] The phase shift circuit 623 uses the sensor detection information b1 (fingertip position information, etc.) to control the phase of the ultrasonic signal, that is, to change the phase of the ultrasonic drive signal c5 input to each ultrasonic element 63. As a result, a signal is generated for forming a maximum sound pressure point (focus) in accordance with the touch position, etc., for the group of ultrasonic waves emitted from the entire ultrasonic element array 61. Specifically, the phase shift circuit 623 can be configured with, for example, a shift register. By changing the number of stages of the shift register, the number of types of phases is not limited to eight, but can be any number of types.

[0127] The ultrasonic carrier signal c3 having multiple (eight types) phases output from the phase shift circuit 623 is input to the AM modulation circuit 624. The AM modulation circuit 624 is a circuit for superimposing an audio signal on the ultrasonic carrier signal c3, and is a circuit for AM-modulating the ultrasonic carrier signal c3 by the audio signal. An audio signal b2 is input from an input terminal to the AM modulation circuit 624. The audio signal b2 is a modulating audio signal for modulating the ultrasonic carrier signal c2. The AM modulation circuit 614 AM-modulates the ultrasonic carrier signal c3 by the audio signal b2 to obtain a modulated signal c4 (modulated ultrasonic carrier signal).

[0128] The audio signal b1 input to the AM modulation circuit 624 is a signal for generating audio associated with an object on which the user U has performed an operation such as touching the floating-in-space image 3 with his / her fingertip Uf. This audio is, for example, an audible band audio (e.g., "one") reading out a number (e.g., "1") displayed on a push button object. This audio signal b1 may also be an audio signal such as a predetermined operation sound or an error warning sound for informing the user U that the object has been operated. When there is no audio to be superimposed on the ultrasound, the input of the audio signal b1 can be turned off and the processing in the AM modulation circuit 624 can be skipped.

[0129] The modulated ultrasonic signal c4 (modulated ultrasonic carrier signal) output from the AM modulation circuit 624 is input to the inductance circuit 625. The inductance circuit 625 is a circuit configured with, for example, a coil, and generates N ultrasonic drive signals c5 corresponding to the N ultrasonic elements 63 based on the modulated ultrasonic signal c4. The generated N ultrasonic drive signals c5 are supplied to the N ultrasonic elements 63 of the ultrasonic element array 61.

[0130] When the AM modulation circuit 624 performs AM modulation using the audio signal b1, the ultrasonic waves emitted from the ultrasonic element array 61 are superimposed with the audio signal. As a result, audio corresponding to the audio signal b1 is emitted from the vicinity of the location where the user U manipulates the object of the floating in space image 3 with his / her fingertip Uf (for example, the touched position P1). In other words, when the ultrasonic waves hit the fingertip Uf, the audio is demodulated. The audio emitted from that location basically propagates in all directions and reaches the user U's ear UE. As a result, when the user U touches an object, the user U can get the above-mentioned touch sensation and can hear audio related to the object from the vicinity of the object. As a result, the user U can more reliably recognize that the object has been touched.

[0131] Fig. 19 shows a configuration example of the inductance circuit 625. Fig. 19 shows a configuration example in the case of generating an ultrasonic drive signal c5 having the above-mentioned eight kinds of phases. The inductance circuit 625 is configured to have a plurality of inductances that are variable capacitance inductances that can vary the inductance component. Fig. 19 also shows an example in which eight ultrasonic signals c4 (modulated ultrasonic carrier signals) having eight kinds of phases and after AM modulation are input to the inductance circuit 625 that is configured with eight variable capacitance inductances (L1 to L8).

[0132] In this example, the ultrasonic element array 61 is divided into eight regions based on the control of the above eight types of phases. FIG. 20 shows an example in which the surface of the ultrasonic element array 61 is divided into eight regions corresponding to the eight types of phases. In this example, eight regions are formed concentrically around the center point of the array. An ultrasonic drive signal c5 with a different phase is input to each region. One selected phase is associated with each region (in other words, each phase group). For example, the central region indicated by the solid line is the region of maximum sound pressure (point M), and the sound pressure decreases toward the periphery in the radial direction. For the sake of explanation, the multiple ultrasonic elements 63 constituting the region of one phase group are also described as an ultrasonic element group, and the number of ultrasonic elements 63 constituting the group is m. For example, in the central region shown in the figure, m=7.

[0133] 19 is L, and the capacitance component obtained by multiplying the capacitance per ultrasonic element 63 in the ultrasonic element array 61 (for example, about 2400 pF / piece) by the number m of ultrasonic element groups constituting each of the above-mentioned regions is C. Then, the inductance component L and the capacitance component C form an LC resonant circuit.

[0134] In FIG. 19, the capacitance components C1 to C8 corresponding to the eight regions of the eight types of phases are illustrated as an equivalent circuit 1900 of the ultrasonic element group. In FIG. 19, N signals are illustrated as signal lines grouped together for each of the eight phase groups. The input terminal 1902 actually has the above-mentioned two input terminals for each ultrasonic element 63 of the phase group. The inductance circuit 625 has inductance components L1 to L8 connected to the capacitance components C1 to C8. In this example, the inductance components L1 to L8 are adjusted so that the resonant frequency (f) of the LC resonant circuit is 40 kHz. As illustrated, the resonant frequency f is determined by 1 / {2π√(LC)}. As a result, eight types of 40 kHz sine wave signals with different phases are input as ultrasonic drive signals c5 from the inductance circuit 625 to the ultrasonic element groups of each of the eight regions divided into which the ultrasonic element array 61 is composed.

[0135] Here, in an experiment conducted by the present inventor, it was confirmed that an ultrasonic phased array can be configured by performing phase control on each ultrasonic element 63 of an ultrasonic element array 61 in which N=223 ultrasonic elements 62 are arranged concentrically as shown in FIG. 18. The fingertip tactile sense generating device 6 can form a point or small area (point M, for example, FIG. 20) where the sound pressure level of the ultrasonic waves is maximum in the area of ​​an arbitrary point on the array using the ultrasonic phased array by this ultrasonic element array 61. More specifically, the phase control using the phase shift circuit 623 is such that when the distance between any ultrasonic element 63 constituting the ultrasonic element array 61 and point M is far, the phase is advanced, and when the distance between the ultrasonic element 63 and point M is close, the phase is delayed. With this method, ultrasonic waves emitted from each ultrasonic element 63 strengthen or weaken each other, and a region where the sound pressure level of the ultrasonic waves is maximum can be formed as a focus in the area of ​​point M. As a more specific example of the sound pressure level, it was confirmed that when an ultrasonic signal of 10 Vrms is applied to each ultrasonic element 63, a sound pressure of about 0.6 gf (gram force) is generated at a position about 20 cm above the ultrasonic element array 61 (perpendicular to the plane on the output side, in the direction of the ultrasonic path). Note that this position of about 20 cm is one example corresponding to the performance of the element used, and can be designed without being limited to this. For example, if it is desired to generate a sound pressure of the same level as above at a longer position of 30 cm, the voltage applied to the ultrasonic element 63 should be increased to about 17 to 18 Vrms.

[0136] In the configuration of Fig. 17, the standard position where the maximum sound pressure is formed as described above is designed to be a position that matches the surface of the floating-in-space image 3. This allows the user U to get a sufficient sense of operation as a touch sensation when touching an object on the surface of the floating-in-space image 3 with his / her fingertip Uf. The touch operation is an operation of moving the fingertip Uf in the z direction from the front to the back of the surface of the floating-in-space image 3 to touch the object.

[0137] In this embodiment, when the ultrasonic signal c3 is AM-modulated by the audio signal b2 (for example, the voice of reading out numbers) in the AM modulation circuit 624, the audio can be emitted from the vicinity of the point M with the maximum sound pressure. In other words, the audio signal can be demodulated and emitted as a related audio from the vicinity of the fingertip Uf of the user U and the object. According to this configuration, when the fingertip Uf touches (in other words, is positioned) an object formed by the floating-in-space image 3 that is actually empty, the user U can get a touch sensation that feels like physical contact with the fingertip Uf. In addition, the user U can hear the audio related to the object emitted from the vicinity of the fingertip Uf. This allows the user U to more reliably recognize the operation he or she has performed as a touch operation from both the sense of touch and the audio.

[0138] The ultrasonic signal generating circuit 62 AM-modulates, with respect to the ultrasonic signals of each ultrasonic element 62 in the ultrasonic element array 61, at least the ultrasonic signals of a portion of the ultrasonic elements 63 corresponding to an area where sound pressure is to be generated by ultrasonic waves, using an audio signal.

[0139] Needless to say, when the audio signal b2 is not input to the AM modulation circuit 614, that is, when AM modulation is not performed, no sound is generated from the location where the fingertip Uf is located. For example, when touching an object, it is possible to generate a touch sensation using only the sound pressure. The control device 10 may set the audio signal b2 in association with the object or operation according to the contents of the object of the floating-in-space image 3. Also, for example, when operation A (e.g., touch) and operation B (e.g., swipe) are possible for object A, it is possible to control so that sound A is generated when operation A is performed, and sound B is generated when operation B is performed.

[0140] Although the ultrasonic signal generating circuit 62 in Fig. 18 is illustrated as an example of a functional block configuration using analog circuits, the implementation is not limited to this. For example, the ultrasonic signal generating circuit 62 may be implemented entirely using digital signal processing circuits. Also, each circuit unit may be implemented using one or more dedicated circuits. Furthermore, each circuit unit may be implemented using software program processing.

[0141] [Floating in space video] Fig. 21 shows an example of the positional relationship between the space-floating image 3 and the fingertip tactile sense generation device 6 when the space-floating image 3 produced by the space-floating image display device 1000 in Fig. 17 is viewed from the viewpoint of a legitimate user U. In Fig. 21, the fingertip tactile sense generation device 6 is installed in the center position on the left and right of the stand 52 (which is plate-shaped in this example, but is not limited to this), and cameras (5L, 5R) are arranged on the left and right. A linear sensor 4 is arranged on the front side of the base 51 of the housing 50, and a frame-shaped transparent structural member 60 is fixed diagonally above that point, and the space-floating image 3 is formed within the surface of the transparent structural member 60.

[0142] In Fig. 21, an object OB1 is displayed as a part of the floating-in-space image 3. The object OB1 is displayed, for example, at the center position within the plane (xy) of the floating-in-space image 3. The object OB1 is, for example, a push button object, and is defined in advance as an object that accepts a touch operation (or a tap operation). Fig. 21 shows a schematic diagram of a state when a user U touches the object OB1 with the index finger of his / her right hand as the fingertip Uf.

[0143] As described above, the fingertip tactile sense generating device 6 controls the phase of the ultrasonic drive signal c5 input to each ultrasonic element 63 of the ultrasonic element array 61, thereby generating ultrasonic waves from the ultrasonic element array 61 so that the maximum sound pressure occurs at the position of the fingertip Uf of the user U in the floating-in-space image 3, that is, the touch position of the object OB1. As a result, a sound pressure of, for example, about 0.6 gf is generated at the fingertip Uf, and the user U feels a touch sensation as if he or she were touching an object at the touch position of the fingertip Uf.

[0144] In the space floating image display device 1000 of this embodiment, the positional relationship between the space floating image 3 and the fingertip tactile sense generation device 6 is configured as shown in Fig. 17 and Fig. 21. This allows the fingertip tactile sense generation device 6 to generate a touch sensation by ultrasonic waves on the fingertip Uf of the user U without obstructing the optical path (axis a2, etc.) for generating the space floating image 3.

[0145] The position of the touch between the fingertip Uf of the user U and the object OB1 can be detected by the aerial operation detection unit 1350 in FIG. 1 based on the signal emitted by the sensor 4 (air-operation detection sensor 1351 in FIG. 1) and the signal received by the sensor 4 as described above. The control device 10 grasps information such as the touch position (for example, position P1 in FIG. 17) output by the aerial operation detection unit 1350. The fingertip tactile sense generating device 6 performs phase control of the output of ultrasonic waves based on the touch position coordinate information received from the control device 10. That is, as shown in FIG. 17, a group of ultrasonic waves from the ultrasonic element array 61 is emitted toward the touch position, so that a focus is formed by sound pressure at the touch position, and a touch sensation is generated on the fingertip Uf. A touch sensation can be generated by the same control for any position on the main plane (xy) of the floating-in-space image 3 in FIG. 21.

[0146] Furthermore, when generating the touch sensation of the object OB1, the control device 10 and the fingertip tactile sense generating device 6 input a predetermined audio signal corresponding to the object OB1 as the audio signal b1 for modulation to be input to the AM modulation circuit 624 in Fig. 18, as described above. As a result, the audio signal is superimposed on the ultrasonic wave group, so that a predetermined sound is emitted from the vicinity of the touch position of the object OB1 by the fingertip Uf, and the user U can hear the sound.

[0147] The predetermined audio signal associated with the object OB1 may be, for example, an operation sound (such as "pon") indicating that the push button has been pressed, a voice (such as "ichi") that reads out a number (such as "1") or a symbol written on the push button, or a guidance voice that is not written on the push button but is associated with it, etc. This allows the user U to confirm the operation of the object OB1 not only visually but also by sound.

[0148] [Combination of fingertip tactile sensation generator and ultra-directional speaker] As a modification of the above embodiment, it is also possible to use both the fingertip tactile sense generation device 6 of Fig. 17 etc. and the superdirectional speaker 30 of Fig. 13 etc. in combination. Fig. 22 shows an example of the installation of the fingertip tactile sense generation device 6 and the superdirectional speaker 30 in this combined form. In this example, in addition to the configuration of the fingertip tactile sense generation device 6 similar to that of Fig. 21, left and right superdirectional speakers 30L, 30R similar to that of Fig. 13 are provided.

[0149] Based on detection of the state of the fingers UH by the sensor 4 and detection of the face, etc. by the camera (camera unit 55C), the control device 10 controls the generation of tactile sensation and sound by the fingertip tactile sensation generating device 6 and the generation of sound by the superdirectional speaker 30. By using these in combination, for example, the user U can feel a sense of touch on his / her fingertips Uf, and can hear only the user U as sound associated with the object OB1 from the superdirectional speakers 30 (30L, 30R) in a way that cannot be heard by others nearby the user U. The above-mentioned form of combination is particularly effective when applied to a highly confidential system, such as an ATM device.

[0150] Furthermore, the above-mentioned combined form may be applied to a system that does not require a high level of confidentiality, such as a ticket vending machine at a station. In that case, for example, a station name selection button or the like may be used as an object of the floating-in-space image 3, and the sound associated with the object may be the sound of the station name. When the user U touches the button with the name of the station of his / her destination, the sound of the station name is output. The method of outputting the sound of this object may be output by the fingertip haptic sense generating device 6 or by the superdirectional speaker 30. In particular, when the superdirectional speaker 30 is used, there is no risk that the information such as the station name will be heard by others around the user U, and a ticket vending machine system that takes privacy into consideration can be configured.

[0151] Examples of control for using the two devices, the fingertip tactile sense generating device 6 and the superdirectional speaker 30, together or selectively will be given below.

[0152] First Example: When transmitting an arbitrary voice to the user U, the superdirectional speaker 30 is turned on and used. At that time, the voice output by the ultrasonic element array 61 is turned off and not used.

[0153] Second Example: Depending on the type of voice to be output to user U, output from superdirectional speaker 30 and output from ultrasonic element array 61 are used interchangeably. For example, they are used interchangeably depending on the level of secrecy of the target voice. When outputting a type of voice that requires high secrecy (such as the voice of a PIN number), superdirectional speaker 30 is used. When outputting a type of voice that does not require high secrecy (such as an operation sound), ultrasonic element array 61 is used.

[0154] Also, in the above combined form, when outputting sound to user U using superdirectional speaker 30, space floating image display device 1000 detects the face position etc. using camera 5 of imaging unit 1180, and controls superdirectional speaker 30 to output sound toward the detected face position. This further enhances the effect of superdirectional speaker 30.

[0155] [Effects of Example 4] As described above, the space-floating image display device 1000 of the embodiment (example 4) provides the following effects. The user U who sees and operates the space-floating image 3 as a non-contact user interface can see objects such as push buttons in the space-floating image 3 more reliably without ghost images. Furthermore, when the user U touches the object, he or she can get a touch sensation similar to touching a physical button. Furthermore, when the user touches the object, he or she can hear the sound associated with the object emanating from the vicinity of the fingertip. According to the embodiment, it is possible to provide a non-contact user interface that can minimize the risk of contact infection, has excellent visibility and operability, and can reduce erroneous operations and erroneous inputs.

[0156] According to the embodiment of the floating-in-space image display device, it is possible to display high-resolution and high-brightness image information as a floating-in-space image in a floating-in-space state. In particular, when this floating-in-space image is used as a non-contact user interface including objects such as push buttons, the user can easily operate it without worrying about contact infection with physical buttons, etc., and when the user touches an object displayed as a floating-in-space image with his / her fingertip, he / she can get a sensation (touch feeling, etc.) as if he / she had touched a physical button. Furthermore, when the user touches an object such as a push button, he / she can hear a sound such as a number associated with the object from the fingertip and the vicinity of the object. This makes it possible to prevent or reduce erroneous input by the user to the floating-in-space image.

[0157] According to the space floating image display device of the embodiment, when an object such as a push button is displayed as the space floating image 3 and used as a non-contact user interface, when the user U touches the object, the fingertip tactile sensation generating unit 1230 can generate a touch sensation on the fingertip. Although the object of the space floating image 3 can be visually recognized by the user U, there is actually no object other than air at that position, making it difficult to get a sense of operation. However, according to the embodiment, when the user U touches such an object, the user U can perceive a sensation as if he or she has touched an object, and can recognize that the touch operation has been confirmed (in other words, the device side has accepted it as a touch operation / input) by both vision and touch. Therefore, a non-contact user interface that is more suitable than the conventional one and has excellent information confidentiality can be realized.

[0158] Furthermore, according to the embodiment, when a non-contact user interface is used, the fingertip tactile sensation generating device 6 is used to modulate the ultrasonic signal with the audio signal, so that the ultrasonic signal is superimposed with the audio related to the object. As a result, when the user U touches an object, not only is a touch sensation generated, but the user U can also hear the audio related to the object from the vicinity of the fingertip Uf. That is, when the user U operates an object in the floating-in-space image 3, the user U can recognize that the operation has been performed correctly from the visual, tactile, and auditory senses. Therefore, a more suitable non-contact user interface can be realized.

[0159] The space-floating image display device of the embodiment and the space-floating image produced by the same can be applied as a non-contact user interface including various applications such as ticket issuing, reception, authentication, and payment in systems used by an unspecified number of users, for example, public facilities such as stations, government offices, and hospitals, facilities such as banks and department stores, and devices such as cash registers and elevators. Examples of GUI objects include push buttons that indicate station names, PIN numbers, and destination floors. The invention can be applied to various objects such as slide bars as well as buttons. With such a non-contact user interface, the user does not need to touch physical panels or buttons with his or her fingertips, minimizing the risk of contact infection and allowing the user to use applications without feeling anxious. In addition, the visibility and operability of the non-contact user interface using the space-floating image can be greatly improved compared to conventional methods.

[0160] In addition, the space-floating image display device of the embodiment includes an image display device 1 as an image source and a retroreflective member 2, and the divergence angle of the image light emitted to the outside is small, i.e., an acute angle, and further aligned with a specific polarization, so that only the normal reflected light is efficiently reflected by the retroreflective member 2. Therefore, this space-floating image display device has high light utilization efficiency, and can significantly reduce the occurrence of ghost images that occur in addition to the main space-floating image, which was a problem in the conventional retroreflective method, and can obtain a clear space-floating image. In addition, this space-floating image display device includes an image display device 1 including a unique light source device 13. This makes it possible to provide a new space-floating image display device that can significantly reduce power consumption and has excellent usability.

[0161] [Modification of Example 4] Various modifications of the embodiment of the above example 4 are possible as follows.

[0162] FIG. 23 shows a configuration of a modified example of the space-floating image display device 1000 seen from the side. This modified example is different from the configuration of FIG. 17 in that the path of the ultrasonic wave output from the fingertip tactile sense generating device 6 is not the above-mentioned reflection type, but a horizontally linear path, and is configured to focus on the fingertip Uf from the back side of the space-floating image 3. In this example, the fingertip tactile sense generating device 6 is separate from the housing 50 and is installed on the wall 57 (or the surface of another device) behind the space-floating image display device 1000. The plane of the output of the ultrasonic element array 61 is vertical, and the axis a4 of the ultrasonic output is horizontal. Even in this configuration, the path of the ultrasonic wave is designed not to overlap with or interfere with the path (axis a2, etc.) of the image light from the retroreflective member 2, etc. in the housing 50.

[0163] In addition, in this configuration, the angle of the ultrasound (particularly the focus of the sound pressure) with respect to the plane (xy) of the floating image 3 is, for example, about 45 degrees, as in angle α2. Even with this configuration, sound pressure can be formed on the fingertip Uf, so that a sufficient touch sensation can be given.

[0164] As another modified example, an ultrasonic element array 61 or the like may be provided inside the housing 50. In that case, the ultrasonic element array 61 is arranged so as not to interfere with elements such as the retroreflective member 2 inside the housing 50. In that case, the path of the ultrasonic waves from the ultrasonic element array 61 must be such that it does not hit the transparent member 100 or the like and reaches the surface of the external floating-in-space image 3. For this reason, for example, an opening that allows ultrasonic waves to pass through is provided in a part of the transparent member 100 or the housing 50.

[0165] Fig. 24 shows the state of the floating-in-space image 3 and the like in another modified example seen from the side. In this modified example, various operations are accepted in addition to touch operations as operations on the floating-in-space image 3 and objects. In Fig. 24, the floating-in-space image area 3V is shown as a three-dimensional area that encompasses the main plane (xy) of the floating-in-space image 3 and extends outward. The floating-in-space image area 3V is, for example, an area that takes a predetermined distance K1 in the front and back in the z direction perpendicular to the surface.

[0166] The space-floating image display device 1000 accepts various operations by the fingers UH of the user U on this space-floating image area 3V. The space-floating image display device 1000 detects the state of the position, movement, etc. of the fingers UH based on sensing by the sensor 4 and the camera 5, etc., and determines various operations. The sensor 4, etc. detects the state of the operation including the three-dimensional position and movement of the fingers UH on the surface of the space-floating image 3 or the space-floating image area 3V.

[0167] For example, when the user U's fingers UH are close to the front side of the surface of the floating image 3, the distance in the z direction between the fingertip Uf and the surface of the floating image 3 is, for example, distance K2. Also, when the user U's fingers UH are intruding into the back side of the surface of the floating image 3, the distance in the z direction between the fingertip Uf and the surface of the floating image 3 is, for example, distance K3. The control device 10 measures such distances within a predetermined floating image area 3V and obtains the position coordinates (X, Y, Z) of the fingertip Uf in space. Also, the control device 10 can grasp the movement of the fingers UH by obtaining the position at each time point in time series.

[0168] In addition to touch and tap operations, the operation using the fingers UH can also include swipe operations, flick operations, and pinch operations. For example, when applying to the swipe operation of an object, the position of the formation of the focus by ultrasound can be controlled based on the position of the movement of the fingertip Uf in the x and y directions by the swipe operation. These operation examples correspond to the operations on existing touch panels, but are not limited to these and any operation (gestures, etc.) can be defined. The operation using the fingers UH is not limited to operations using one finger, but can also be performed using two fingers or the entire hand. The operation object for the floating image 3 is not limited to the fingers UH, and can be similarly applied as an object such as a pen held by the fingers UH (however, in this case, sensations cannot be generated at the fingertips).

[0169] Furthermore, when accepting an operation on the floating in space image area 3V as described above, the control of the formation of a focal point by ultrasonic waves from the fingertip tactile sense generating device 6 is not limited to the control of directions within the above-mentioned plane (xy), but can also be controlled in the z direction perpendicular to the plane. For example, when touching an object, when the fingertip Uf penetrates deep into the floating in space image area 3V, a focal point by ultrasonic waves may be formed according to the position of the distance at that time. Based on the above-mentioned phase control, the focal point can be moved in the z direction.

[0170] In another modified example, as an example of another operation and control, when a state is detected in which the fingers UH are not touching the main surface of the floating-in-space image 3 but are sufficiently close, it may be determined that a specified operation such as a touch operation has been performed and a tactile sensation may be generated.

[0171] <Optimization of the diffusion characteristics of a space-floating image display device> Next, detailed configurations and characteristics of image display devices that can be applied to the space floating image display devices of the above-mentioned embodiments will be described. When a large liquid crystal display panel is used as the liquid crystal display panel 11 in Fig. 2, the liquid crystal display panel may be configured to face inward so that light around the screen is directed toward the user (eyes) when the user faces the center of the screen. This improves the overall brightness of the screen.

[0172] 25 is a graph showing the convergence angle between the long side and short side of the panel when the viewing distance L from the user's panel and the panel size (screen ratio 16:10) are used as parameters. When the screen is viewed vertically, the convergence angle can be set according to the short side. For example, when a 22-inch panel is used vertically and the viewing distance is 0.8 m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the user.

[0173] Similarly, when a 15-inch panel is used in portrait orientation and the monitoring distance is 0.8 m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed to the user. In this way, depending on the size of the LCD panel and whether it is used in portrait or landscape orientation, the image light from the periphery of the screen can be directed to the user who is in the optimum position to monitor the center of the screen, improving the overall brightness of the screen.

[0174] Next, as described above, the liquid crystal display panel 11 is directed inward so that when the user faces the center of the screen, the light around the screen is directed toward the user. Furthermore, assuming that the average distance between the eyes of an adult is 65 mm, the luminance difference in the horizontal direction of the screen of the floating image caused by the parallax between the left eye and the right eye was calculated using the viewing distance as a parameter. The results are shown in Figure 26. If the shortest viewing distance in normal use is 0.8 m, the difference in brightness due to parallax can be reduced by using a light source device with characteristics such that the relative luminance is not less than 50% at 12 degrees, which is the sum of the difference in viewing angle (5 degrees) and the convergence angle (7 degrees) on the long side shown in Figure 25.

[0175] In addition, since the short side direction of the space floating image display device can be accommodated by shifting the user's line of sight, the overall brightness of the screen is improved if the brightness difference due to parallax in the long side direction, which is a more severe condition, is taken into consideration.

[0176] <Reflective polarizing plate> The grid-structured reflective polarizer in the embodiment (for example, the reflective polarizer constituting the polarization separation member 101 in FIG. 2) has poor characteristics for light coming from a direction perpendicular to the polarization axis. For this reason, it is desirable for the reflective polarizer to be specified along the polarization axis, and the light source device 13 in the embodiment, which is capable of emitting the image light emitted from the liquid crystal display panel 11 at a narrow angle, is an ideal light source. Similarly, the horizontal characteristics are also poor for light coming from an oblique angle. Taking the above characteristics into consideration, in the configuration example of the present embodiment, the light source device 13 capable of emitting the image light emitted from the liquid crystal display panel 11 at a narrower angle is used as the backlight of the liquid crystal display panel 11. This makes it possible to provide a high-contrast floating image. This configuration example will be described below.

[0177] <Display device> A configuration example of the display device 1 will be described with reference to FIG. 27. The display device 1 includes a liquid crystal display panel 11 as an image display element and a light source device 13 constituting the light source. In FIG. 27, the light source device 13 is shown as an exploded perspective view together with the liquid crystal display panel 11. As shown by the direction of the arrow 3000, the liquid crystal display panel 11 obtains an illumination light flux having a narrow-angle diffusion characteristic, that is, a strong directivity (in other words, linearity) and a characteristic similar to that of laser light, by the light from the light source device 13, which is a backlight device. The liquid crystal display panel 11 emits image light modulated according to an input image signal based on the illumination light flux. Then, the image light is reflected by the retroreflective member 2 and passes through the transparent member 100 to form the space floating image 3 as a real image.

[0178] 27 includes a liquid crystal display panel 11, a light direction conversion panel 54 for controlling the directional characteristics of the light beam emitted from the light source device 13, and a narrow-angle diffuser plate (not shown) as necessary. That is, polarizing plates are provided on both sides of the liquid crystal display panel 11, and image light of a specific polarized wave is emitted with its light intensity modulated by a video signal (arrow 3000). As a result, a desired image is projected as light of a specific polarized wave with high directivity through the light direction conversion panel 54 toward the retroreflective member 2, and after being reflected by the retroreflective member 2, it is transmitted toward the user's eyes to form the space floating image 3.

[0179] It should be noted that a protective cover 250 shown in Figs. 28 and 29 may be provided on the surface of the light direction conversion panel 54 described above.

[0180] In this embodiment, in order to improve the utilization efficiency of the light flux (arrow 3000) emitted from the light source device 13 and to significantly reduce power consumption, in the display device 1 including the light source device 13 and the liquid crystal display panel 11, the light (arrow 3000) from the light source device 13 is projected toward the retroreflective member 2, and after being reflected by the retroreflective member 2, the directivity can be controlled by a transparent sheet (not shown) provided on the surface of the transparent member 100 so that the floating image in space is formed at a desired position. Specifically, by configuring this transparent sheet with optical components such as a Fresnel lens or a linear Fresnel lens, the imaging position of the floating image in space can be controlled while maintaining high directivity. According to this configuration, the image light from the display device 1 reaches the user efficiently with high directivity like laser light, and as a result, a high-quality floating image in space can be displayed with high resolution. At the same time, the power consumption by the display device 1 including the LED element 201 of the light source device 13 can be significantly reduced.

[0181] <Display device configuration example 1> As described above, FIG. 27 shows an example of a specific configuration of the display device 1. FIG. 28 is a cross-sectional view showing an example of a specific configuration of the light source device 13 (corresponding to the light source 1105 in FIG. 1) in FIG. 27. As shown in FIG. 28, the liquid crystal display panel 11 and the light direction conversion panel 54 are arranged on the light source device 13 in FIG. 27. The light source device 13 is formed, for example, of plastic on the case shown in FIG. 27, and is configured by storing the LED element 201 and the light guide 203 (corresponding to the light guide 1104 in FIG. 1) inside. As shown in FIG. 28 and other figures, the end face of the light guide 203 has a lens shape that gradually increases in cross-sectional area toward the opposite side to the light receiving part in order to convert the divergent light from each LED element 201 into a substantially parallel light beam, and gradually reduces the divergence angle by multiple total reflections during propagation inside. The liquid crystal display panel 11 is attached to the upper surface of the light guide 203. Furthermore, LED elements 201, which are semiconductor light sources, and an LED board 202, which mounts a control circuit for the LED elements 201, are attached to one side surface (the left end surface in this example) of the case of the light source device 13. A heat sink, which is a member for cooling heat generated by the LED elements and the control circuit, may be attached to the outer surface of the LED board 202.

[0182] In addition, the frame (not shown) of the liquid crystal display panel 11 attached to the upper surface of the case of the light source device 13 is provided with the liquid crystal display panel 11 attached to the frame, a flexible wiring board (FPC, not shown) electrically connected to the liquid crystal display panel 11, and the like. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light based on a control signal from a control circuit (not shown) constituting an electronic device together with the LED element 201, which is a solid light source. The image light generated at this time has a narrow diffusion angle and contains only specific polarization components, so that a new image display device that has not been seen before is obtained that is similar to a surface-emitting laser image source driven by an image signal. Note that, at present, it is technically and safety impossible to obtain a laser light beam of the same size as the image obtained by the above-mentioned display device 1 using a laser device. Therefore, in this embodiment, for example, a light beam from a general light source equipped with an LED element is configured to obtain light similar to the above-mentioned surface-emitting laser image light.

[0183] Next, the configuration of the optical system housed in the case of the light source device 13 will be described in detail with reference to FIG. 28 and FIG. 29. Since FIG. 28 and FIG. 29 are cross-sectional views, only one of the LED elements 201 constituting the light source is shown. In this optical system, the shape of the light receiving end surface 203a of the light guide 203 converts the incident light into approximately parallel light (collimated light). For this reason, the light receiving portion of the end surface of the light guide 203 and the LED element 201 are attached while maintaining a predetermined positional relationship. Each of the light guides 203 is formed of a light-transmitting resin such as acrylic. The LED light receiving surface at the end of the light guide 203 has, for example, a convex convex outer circumferential surface obtained by rotating a parabolic cross section, and at the top of the LED light receiving surface, a concave portion having a convex portion (i.e., a convex lens surface) formed in the center is formed, and the center of the flat surface has a convex lens surface protruding outward (or may be a concave lens surface recessed inward) (not shown). The outer shape of the light receiving part of the light guide 203 to which the LED element 201 is attached is a paraboloid that forms a conical outer surface, and the angles of the reflecting surface and the paraboloid are set so that the light emitted from the LED element 201 in the peripheral direction can be totally reflected inside the light guide 203.

[0184] On the other hand, the LED elements 201 are arranged at predetermined positions on the surface of the circuit board, that is, the LED substrate 202. The LED substrate 202 is arranged and fixed such that the LED elements 201 on the surface are located in the center of the recessed portion described above with respect to the LED collimator (light receiving end surface 203a).

[0185] According to this configuration, the shape of light-receiving end surface 203a of light guide 203 makes it possible to extract the light emitted from LED element 201 as substantially parallel light, thereby improving the efficiency of use of the generated light.

[0186] As described above, the light source device 13 is configured by mounting a light source unit in which a plurality of LED elements 201 serving as light sources are arranged on the light-receiving end surface 203a serving as a light-receiving section provided on the end surface of the light guide 203, and the divergent light beam from the LED elements 201 is converted into substantially parallel light by the lens shape of the light-receiving end surface 203a at the end surface of the light guide 203, which is guided inside the light guide 203 as shown by the arrow, and is emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11 disposed substantially parallel to the light guide 203. By optimizing the distribution (in other words, density) of the light beam direction conversion means depending on the shape of the inside or surface of the light guide 203, the uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled. The light beam direction conversion means 204 emits the light beam propagated inside the light guide 203 toward the liquid crystal display panel 11 arranged approximately parallel to the light guide 203 by configuring the surface shape of the light guide 203 or providing a portion with a different refractive index inside the light guide 203. At this time, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is placed at the same position as the diagonal dimension of the screen, if the relative luminance ratio between the center of the screen and the periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it is an even better characteristic.

[0187] 28 and 29 are cross-sectional layout diagrams for explaining the configuration and operation of a light source of this embodiment that performs polarization conversion in light source device 13 including light guide 203 and LED element 201 described above. In Fig. 28, light source device 13 is composed of light guide 203 having light beam direction conversion means 204 on the surface or inside formed of plastic or the like, LED element 201 as a light source, reflective sheet 205, reflective polarizing plate 206, lenticular lens, etc. On the upper surface of light source device 13, liquid crystal display panel 11 having polarizing plates on the light source light entrance surface and the image light exit surface is attached.

[0188] A film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (lower surface) of the liquid crystal display panel 11 corresponding to the light source device 13. The reflective polarizing plate 49 selectively reflects one polarized wave (e.g., P wave) WAV2 of the natural light beam 210 emitted from the LED element 201, and reflects it on a reflective sheet 205 provided on one (lower) surface of the light guide 203 so that it heads toward the liquid crystal display panel 11 again. Therefore, a λ / 4 plate, which is a retardation plate, is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizing plate 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through twice, thereby converting the reflected light beam from P polarized light to S polarized light, thereby improving the utilization efficiency of the light source light as image light. The image light beam (arrow 213 in Figure 28), whose light intensity has been modulated by a video signal in the liquid crystal display panel 11, enters the retroreflective member 2 and, as shown in Figure 2, passes through the transparent member 105 after reflection, thereby obtaining a real image, a floating image 3, on the outside.

[0189] 29 is similarly composed of a light guide 203 having light beam direction conversion means 204 on or inside the light guide 203 made of, for example, plastic, an LED element 201 as a light source, a reflective sheet 205, a reflective polarizing plate 206, a lenticular lens, etc. On the upper surface of the light source device 13, a liquid crystal display panel 11 having polarizing plates on the light source light entrance surface and the image light exit surface is attached as an image display element.

[0190] A film or sheet-like reflective polarizing plate 49 is provided on the light source light incidence surface (lower surface) of the liquid crystal display panel 11 corresponding to the light source device 13. One side polarized wave (for example, S wave) WAV1 of the natural light beam 210 emitted from the LED element 201 is selectively reflected by the reflective polarizing plate 49, reflected by a reflecting sheet 205 provided on one (lower) surface of the light guide 203, and directed again toward the liquid crystal display panel 11. A λ / 4 plate, which is a retardation plate, is provided between the reflecting sheet 205 and the light guide 203, or between the light guide 203 and the reflective polarizing plate 49, and the one side polarized wave of the natural light beam 210 emitted from the LED element 201 is reflected by the reflecting sheet 205 and passes through it twice, thereby converting the reflected light beam from S polarized light to P polarized light. This can improve the efficiency of using the light source light as image light. The image light beam (arrow 214 in Figure 29) that has been light intensity modulated by a video signal in the liquid crystal display panel 11 enters the retroreflective member 2 and, as shown in Figure 2, passes through the transparent member 105 after reflection, generating a real image, a floating image 3, on the outside.

[0191] In the light source device 13 shown in Fig. 28 and Fig. 29, in addition to the action of the polarizing plate provided on the light incident surface of the corresponding liquid crystal display panel 11, the polarized component on one side is reflected by the reflective polarizing plate, so that the theoretically obtained contrast ratio is the product of the inverse of the cross transmittance of the reflective polarizing plate and the inverse of the cross transmittance obtained by the two polarizing plates attached to the liquid crystal display panel 11. This results in high contrast performance. In fact, it has been confirmed by experiments that the contrast performance of the displayed image is improved by 10 times or more. As a result, it has been confirmed that a high-quality image that is comparable to that of a self-luminous organic EL can be obtained.

[0192] <Display device configuration example 2> FIG. 30 shows another example of a specific configuration of the display device 1. The light source device 13 in FIG. 30 is similar to the light source device in FIG. 29 and the like. This light source device 13 is configured by housing an LED, a collimator, a composite diffusion block, a light guide, and the like in a case made of, for example, plastic. A liquid crystal display panel 11 is attached to the upper surface of the light source device 13. In addition, an LED board on which LED elements, which are semiconductor light sources, and a control circuit for the LED elements are mounted are attached to one side of the case of the light source device 13. A heat sink 103, which is a member for cooling heat generated by the LED elements and the control circuit, is attached to the outer side of the LED board.

[0193] The frame of the liquid crystal display panel 11 attached to the top surface of the case is fitted with the liquid crystal display panel 11 attached to the frame, and further with an FPC and the like electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements, which are solid-state light sources, based on a control signal from a control circuit constituting the electronic device.

[0194] <Display device configuration example 3> Next, another example of a specific configuration of the display device 1 will be described with reference to Fig. 31. The light source device of this display device 1 converts a divergent light beam of natural light (a mixture of P-polarized and S-polarized waves) emitted from an LED (LED element of an LED substrate) into a substantially parallel light beam by an LED collimator 15, and reflects the substantially parallel light beam toward the liquid crystal display panel 11 by a reflective light guide 304. The reflected light is incident on a wave plate and a reflective polarizing plate 49 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (e.g., S-polarized wave) is reflected by the reflective polarizing plate 49, the phase of which is converted by the wave plate and returned to the reflecting surface, and the light passes through the retardation plate again to be converted into a polarized wave (e.g., P-polarized wave) that transmits through the reflective polarizing plate 49.

[0195] As a result, the natural light from the LEDs is aligned to a specific polarized wave (e.g., P polarized wave), and the specific polarized wave is incident on the liquid crystal display panel 11 and is luminance-modulated according to the video signal, thereby displaying an image on the panel surface. The configuration of FIG. 31 has a plurality of LEDs constituting a light source, as in the above-mentioned example, but since FIG. 31 is a vertical cross-sectional view, only one LED is illustrated. These LEDs are attached to a predetermined position relative to the LED collimator 15. Each of the LED collimators 15 is formed of a light-transmitting resin such as acrylic or glass. The LED collimator 15 has a cone-shaped outer peripheral surface obtained by rotating a parabolic cross section, and has a concave portion at the top of the convex portion (i.e., a convex lens surface) formed in the center. The center of the flat portion has a convex lens surface protruding outward (or may be a concave lens surface recessed inward). The parabolic surface forming the cone-shaped outer peripheral surface of the LED collimator 15 is set within an angle range that allows the light emitted from the LEDs in the peripheral direction to be totally reflected therein. Alternatively, the reflective surface is formed so that the light emitted from the LED in the peripheral direction is within a range of angles that allows total reflection therein.

[0196] The above configuration is the same as that of the light source device 13 of the image display device 1 shown in Fig. 28 and Fig. 29. Furthermore, the light converted into approximately parallel light by the LED collimator 15 shown in Fig. 31 is reflected by the reflective light guide 304, and a specific polarized light is transmitted by the action of the reflective polarizing plate 49, while the other reflected polarized light is transmitted through the reflective light guide 304 again and reflected by the reflector 271 provided on the other surface of the light guide that is not in contact with the liquid crystal display panel 11. At this time, the light is polarized and converted by passing twice through the retardation plate (lambda / 4 plate) 270 arranged between the reflector 271 and the liquid crystal display panel 11, and is transmitted through the reflective light guide 304 again and the reflective polarizing plate 49 provided on the opposite surface, and is made incident on the liquid crystal display panel 11 with the polarization direction aligned. As a result, all the light from the light source can be utilized, and the light utilization efficiency is significantly improved (for example, doubled).

[0197] In a conventional TV set, the light emitted from the liquid crystal display panel 11 had the same diffusion characteristics in both the horizontal direction of the screen (shown by the X-axis in (a) of FIG. 38) and the vertical direction of the screen (shown by the Y-axis in (b) of FIG. 38). In contrast, the diffusion characteristics of the light flux emitted from the liquid crystal display panel 11 of this embodiment are such that, as shown in example 1 in FIG. 38, the viewing angle at which the luminance is 50% of that when viewed from the front (angle 0 degrees) is set to 13 degrees, the viewing angle is approximately 1 / 5 of the conventional 62 degrees. Similarly, the vertical viewing angle is made uneven between the top and bottom, and the reflection angle and the area of ​​the reflection surface of the reflective light guide are optimized so that the upper viewing angle is approximately 1 / 3 of the lower viewing angle. As a result, the amount of image light heading in the viewing direction is significantly improved compared to conventional liquid crystal TVs, and the luminance is 50 times or more higher.

[0198] Furthermore, assuming the viewing angle characteristics shown in Example 2 in Figure 38, the viewing angle at which the brightness is 50% of that when viewed from the front (angle 0 degrees) is set to 5 degrees, which is 1 / 12 of the conventional 62 degrees. Similarly, the vertical viewing angle is optimized by optimizing the reflection angle of the reflective light guide and the area of ​​the reflective surface so that the viewing angle is approximately 1 / 12 of the conventional angle, with equal viewing angles both above and below. As a result, the amount of image light heading in the viewing direction is significantly improved compared to conventional LCD TVs, and the brightness is more than 100 times higher.

[0199] As described above, by setting the viewing angle to a narrow angle, the amount of light flux heading in the viewing direction can be concentrated, greatly improving the efficiency of light utilization. As a result, even if a conventional liquid crystal display panel for TV is used, it is possible to achieve a significant improvement in brightness with similar power consumption by controlling the light diffusion characteristics of the light source device. This makes it possible to create an image display device that is compatible with information display systems facing bright outdoors.

[0200] <Light source device configuration example 1> Next, a detailed description will be given of an example of the configuration of the optical system, such as the light source device 13, housed in the case, with reference to FIG. 32, FIG. 33, and FIG. 34. In FIG. 32 to FIG. 34, the LED elements 14 (14a, 14b) constituting the light source are shown, and these are attached at a predetermined position relative to the LED collimator 15. Each of the LED collimators 15 is formed of a light-transmitting resin, such as acrylic. As shown in FIG. 34, the LED collimator 15 has a cone-shaped outer peripheral surface 156 obtained by rotating a parabolic cross section, and at the top of the LED collimator 15, a concave portion 153 having a convex portion (i.e., a convex lens surface) 157 formed in the center thereof. In addition, the center of the flat portion of the LED collimator 15 has a convex lens surface 154 protruding outward (or may be a concave lens surface recessed inward) 154. The paraboloid forming the conical outer surface 156 of the LED collimator 15 is set within an angle range in which the light emitted from the LED element 14 in the peripheral direction can be totally reflected therein, or a reflective surface is formed.

[0201] The LED elements 14 are disposed at predetermined positions on the surface of the LED substrate 102, which is the circuit substrate. The LED substrate 102 is disposed and fixed to the LED collimator 15 such that the LED elements 14 (14a, 15b) on the surface are positioned at the center of the recess 153.

[0202] According to this configuration, the light emitted from the LED element 14 by the above-mentioned LED collimator 15, particularly the light emitted from the center portion toward the upper side (right direction), is collected and made parallel by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. The light emitted from the other portions toward the periphery is reflected by the parabolic surface that forms the conical outer circumferential surface of the LED collimator 15, and is similarly collected and made parallel. In other words, the LED collimator 15 that has a convex lens in its center and a parabolic surface formed on its periphery can extract almost all of the light generated by the LED element 14 as parallel light. This can improve the utilization efficiency of the generated light.

[0203] A polarization conversion element 2100 is provided on the light emission side of the LED collimator 15. As is clear from Fig. 33, this polarization conversion element 2100 is configured by combining a columnar (parallelogram prism) translucent member having a parallelogram cross section and a columnar (triangular prism) translucent member having a triangular cross section, and arranging them in an array in parallel to a plane perpendicular to the optical axis of the parallel light from the LED collimator 15. Furthermore, a polarization beam splitter (PBS film) 211 and a reflection film 212 are alternately provided at the interface between adjacent translucent members arranged in the array, and a λ / 2 phase plate 213 is provided on the emission surface from which the light incident on the polarization conversion element 2100 and transmitted through the PBS film 211 is emitted.

[0204] 33 is provided on the exit surface of this polarization conversion element 2100. That is, the light emitted from the LED elements 14 is converted into parallel light by the action of the LED collimator 15, enters the synthetic diffusion block 16, is diffused by the texture 161 on the exit side, and then reaches the light guide 17.

[0205] Light guide 17 is a rod-shaped member having a substantially triangular cross section, made of a light-transmitting resin such as acrylic, as shown in Fig. 34. As is clear from Fig. 32, light guide 17 includes light guide light incident portion (including the light guide light incident surface) 171 facing the exit surface of synthetic diffusion block 16 via first diffusion plate 18a, light guide light reflecting portion (including the light guide light reflecting surface) 172 forming an inclined surface, and light guide light exit portion (including the light guide light exit surface) 173 facing liquid crystal display panel 11, which is a liquid crystal display element, via second diffusion plate 18b.

[0206] As shown in Fig. 36, which is a partially enlarged view, a number of reflective surfaces 172a and connecting surfaces 172b are alternately formed in a sawtooth shape on the light guide light reflecting portion (surface) 172 of the light guide 17. The reflective surface 172a (a line segment sloping upward to the right in the drawing) forms an angle αn (n: natural number, 1 to 130 in this example) with the horizontal plane shown by the dashed dotted line in Fig. 32, and as an example, the angle αn is set to 43 degrees or less (but 0 degree or more) here.

[0207] The light guide light incident portion (surface) 171 is formed in a curved convex shape inclined toward the light source side. Accordingly, the parallel light from the exit surface of the synthetic diffusion block 16 is diffused and enters through the first diffusion plate 18a, and as is clear from the drawing, the parallel light is bent slightly upward by the light guide light incident portion (surface) 171 (in other words, deflected) and reaches the light guide light reflecting portion (surface) 172, where it is reflected and reaches the liquid crystal display panel 11 provided on the exit surface above.

[0208] According to the display device 1 described above in detail, the light utilization efficiency and the uniform illumination characteristics are further improved, and it is possible to manufacture the light source device including the modularized S-polarized light source device in a small size and at low cost. In the above description, the polarization conversion element 2100 is described as being attached after the LED collimator 15, but the present invention is not limited to this, and the same action and effect can be obtained by providing it in the optical path leading to the liquid crystal display panel 11.

[0209] In addition, in the light guide light reflection section (surface) 172, a large number of reflection surfaces 172a and connection surfaces 172b are alternately formed in a sawtooth shape, and the illumination light beam is totally reflected on each reflection surface 172a and directed upward, and further enters the light direction conversion panel 54, which is provided with a narrow-angle diffusion plate in the light guide light emission section (surface) 173 and controls the directional characteristics as a substantially parallel diffused light beam, and enters the liquid crystal display panel 11 from an oblique direction. In this embodiment, the light direction conversion panel 54 is provided between the light guide light emission section 173 and the liquid crystal display panel 11, but the same effect can be obtained by providing the light direction conversion panel 54 on the emission surface of the liquid crystal display panel 11. <Light source device configuration example 2>

[0210] Another example of the configuration of the optical system of the light source device 13 and the like is shown in FIG. 35. As in the example of FIG. 33, FIG. 35 shows a plurality of (two in this example) LED elements 14 (14a, 14b) constituting the light source, which are attached at predetermined positions relative to the LED collimator 15. Each of the LED collimators 15 is formed of a light-transmitting resin such as acrylic. As in the example of FIG. 33, the LED collimator 15 has a cone-shaped outer peripheral surface 156 obtained by rotating a parabolic cross section, and at the top of the LED collimator 15, a concave portion 153 having a convex portion (i.e., a convex lens surface) 157 formed in the center thereof is formed. Also, at the center of the flat portion, a convex lens surface 154 protruding outward (or a concave lens surface recessed inward) is formed. The parabolic surface forming the cone-shaped outer peripheral surface 156 of the LED collimator 15 is set within an angle range in which the light emitted from the LED element 14a in the peripheral direction can be totally reflected therein, or a reflective surface is formed thereon.

[0211] The LED elements 14 (14a, 14b) are disposed at predetermined positions on the surface of the LED substrate 102, which is the circuit substrate. The LED substrate 102 is disposed and fixed to the LED collimator 15 such that the LED elements 14 on the surface are located at the centers of the recesses 153.

[0212] According to this configuration, the light emitted from the LED element 14 by the above-mentioned LED collimator 15, particularly the light emitted from the center portion toward the upper side (right direction), is collected and made parallel by the two convex lens surfaces 157, 154 that form the outer shape of the LED collimator 15. The light emitted from the other portions toward the periphery is reflected by the parabolic surface that forms the conical outer circumferential surface of the LED collimator 15, and is similarly collected and made parallel. In other words, the LED collimator 15 that has a convex lens in its center and a parabolic surface formed on its periphery can extract almost all of the light generated by the LED element 14 as parallel light. This can improve the utilization efficiency of the generated light.

[0213] A light guide 170 is provided on the light emission side of the LED collimator 15 via a first diffusion plate 18a. The light guide 170 is a rod-shaped member having a substantially triangular cross section, made of a light-transmitting resin such as acrylic, as shown in Fig. 35. As is clear from Fig. 35, the light guide 170 includes a light guide light incident portion 171, which is an incident portion of the light guide 170, facing the emission surface of the synthetic diffusion block 16 via the first diffusion plate 18a, a light guide light reflection portion 172 forming an inclined surface, and a light guide light emission portion 173 facing the liquid crystal display panel 11 via the reflective polarizing plate 200.

[0214] If a material having the property of reflecting P-polarized light and transmitting S-polarized light is selected as the reflective polarizing plate 200, the P-polarized light in the natural light emitted from the LED light source will be reflected by the reflective polarizing plate 200, pass through the λ / 4 plate 201a provided on the light guide light reflecting section 172 shown in Fig. 34, be reflected by the reflecting surface 201b, and pass through the λ / 4 plate 201a again to be converted into S-polarized light. As a result, all the light beams incident on the liquid crystal display panel 11 are unified into S-polarized light.

[0215] Similarly, if a material having the property of reflecting S-polarized light and transmitting P-polarized light is selected as reflective polarizing plate 200, the S-polarized light in the natural light emitted from the LED light source will be reflected by reflective polarizing plate 200, pass through λ / 4 plate 201a provided on light guide light reflecting section 172 shown in Fig. 34, be reflected by reflecting surface 201b, and pass through λ / 4 plate 201a again to be converted to P-polarized light. As a result, all light beams entering liquid crystal display panel 52 are unified into P-polarized light. Polarization conversion can also be achieved with the above-mentioned configuration.

[0216] <Light source device configuration example 3> Another example of the configuration of the optical system of the light source device 13 and the like will be described with reference to Fig. 31. In this example, as shown in Fig. 31, the divergent light flux of natural light (a mixture of P-polarized light and S-polarized light) from the LEDs of the LED substrate 102 is converted into a substantially parallel light flux by the LED collimator 15, and is reflected toward the liquid crystal display panel 11 by the reflective light guide 304. The reflected light is incident on the reflective polarizing plate 206 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (e.g., S-polarized wave) is reflected by the reflective polarizing plate 206, transmitted through a surface connecting the reflective surfaces of the reflective light guide 304, reflected by a reflector 271 arranged facing the opposite surface of the reflective light guide 304, and polarized and converted by transmitting twice through a retardation plate (λ / 4 plate) 270, transmitted through the light guide and the reflective polarizing plate, and incident on the liquid crystal display panel 11, where it is modulated into image light. At this time, by combining the specific polarized wave with the polarized wave plane, the light utilization efficiency becomes twice as high as usual, and the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system. Therefore, by using the light source device 13 of this embodiment, the contrast ratio of the information display system can be significantly improved.

[0217] As a result, the natural light from the LED is aligned to a specific polarization (for example, P polarization). As in the above example, a plurality of LEDs (only one is shown in FIG. 31 because it is a vertical cross section) constituting the light source are provided, and these are attached at a predetermined position relative to the LED collimator 15. Each of the LED collimators 15 is formed of, for example, a translucent resin such as acrylic or glass. The LED collimator 15 has a cone-shaped outer peripheral surface obtained by rotating a parabolic cross section, and at the top of the LED collimator 15, a concave portion having a convex portion (i.e., a convex lens surface) formed in the center is formed. The center of the flat portion has a convex lens surface protruding outward (or may be a concave lens surface recessed inward). The parabolic surface forming the cone-shaped outer peripheral surface of the LED collimator 18 is set within an angle range that allows the light emitted from the LED collimator 15 in the peripheral direction to be totally reflected therein, or a reflective surface is formed thereon.

[0218] The LEDs are arranged at predetermined positions on the surface of the circuit board, that is, the LED board 102. The LED board 102 is arranged and fixed to the LED collimator 15 such that the LEDs on the surface are located at the center of the recess.

[0219] According to this configuration, the light emitted from the LEDs by the LED collimator 15, particularly the light emitted from the central part, is collected and made parallel by the two convex lens surfaces that form the outer shape of the LED collimator 15. The light emitted from the other parts toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the cone shape of the LED collimator 15, and is similarly collected and made parallel. In other words, the LED collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, can extract almost all of the light generated by the LED as parallel light. This can improve the utilization efficiency of the generated light.

[0220] <Light source device configuration example 4> Further, another example of the configuration of the optical system of the light source device 13 and the like will be described with reference to FIG. 39. In the configuration of FIG. 39, two optical sheets 207 (in other words, diffusion sheets) that convert the diffusion characteristics in the vertical and horizontal directions of the drawing are used on the light emission side of the LED collimator 15, and the light from the LED collimator 15 is made to enter between the two optical sheets 207. When this optical sheet 207 is configured with one sheet, the vertical and horizontal diffusion characteristics are controlled by the fine shapes of the front and back surfaces. In addition, multiple diffusion sheets may be used to share the function. Depending on the front and back shapes of the optical sheet 207, the diffusion angle of the light from the LED collimator 15 in the vertical direction of the screen is matched to the width of the vertical surface of the reflection surface of the diffusion sheet, and in the horizontal direction, the surface density of the light flux emitted from the liquid crystal display panel 11 is uniform, and it is preferable to optimally design the number of LEDs and the divergence angle from the optical element 500 as design parameters. That is, in this configuration, the diffusion characteristics are controlled by the surface shapes of multiple diffusion sheets instead of the light guide. In this embodiment, the polarization conversion is performed in the same manner as in the above-described configuration example 3 of the light source device. Alternatively, a polarization conversion element 2100 (FIG. 30) may be provided between the LED collimator 15 and the optical sheet 207, and the light source light may be made incident on the optical sheet 207 after the polarization conversion is performed.

[0221] If the above-mentioned reflective polarizing plate 206 is selected to have the property of reflecting S-polarized light and transmitting P-polarized light, it will reflect the S-polarized light in the natural light emitted from the LED light source, pass through retardation plate 270 shown in Fig. 39, be reflected by reflector 271, and pass through retardation plate 270 again to be converted into P-polarized light, which then enters liquid crystal display panel 11. The optimum thickness of this retardation plate must be selected depending on the angle of incidence of the light beam on the retardation plate, and the optimum value is in the range of λ / 16 to λ / 4.

[0222] Various embodiments have been described above in detail. The present invention is not limited to the above-described embodiments, and various modified examples are included. In the above-described embodiments, the entire system, etc., has been described in detail to clearly explain the present invention, but the present invention is not limited to having all of the components. Except for the essential elements, the components of each embodiment can be added, deleted, or replaced. Unless otherwise specified, each component may be singular or plural.

[0223] In the technology according to the present embodiment, the floating image is displayed as high-resolution and high-brightness image information in a floating state, which allows the user to operate the device without worrying about contact infection. If the technology according to the present embodiment is used in a system used by an unspecified number of users, it is possible to reduce the risk of contact infection and provide a non-contact user interface that can be used without worry. This contributes to "3. Health and well-being for all" of the Sustainable Development Goals (SDGs) proposed by the United Nations. In addition, the technology according to the present embodiment makes it possible to efficiently reflect only the normal reflected light on the retroreflective material by reducing the divergence angle of the emitted image light and aligning it to a specific polarization, thereby making it possible to obtain a bright and clear floating image with high light utilization efficiency. According to the technology according to the present embodiment, it is possible to provide a highly usable non-contact user interface that can significantly reduce power consumption. This contributes to "9. Build infrastructure, promote innovation and promote prosperity" and "11. Create sustainable cities and towns" of the Sustainable Development Goals (SDGs) proposed by the United Nations. Furthermore, the technology according to this embodiment makes it possible to form a floating image using highly directional (linear) image light. The technology according to this embodiment makes it possible to provide a non-contact user interface with little risk of people other than the user looking at the floating image, even when displaying images that require high security, such as those in bank ATMs and ticket machines at train stations, or highly confidential images that should be hidden from people directly facing the user, by displaying highly directional image light. This contributes to "Sustainable cities and towns" in the Sustainable Development Goals (SDGs) proposed by the United Nations. [Explanation of symbols]

[0224] 1...display device (image display device), 2...retroreflective member, 3...floating-in-space image, 4...sensor (air-operation detection sensor), 5...camera, 6...fingertip tactile sensation generating device, 10...control device, 11...liquid crystal display panel, 12...absorptive polarizing plate, 13...light source device, 50...housing, 61...ultrasonic element array, 62...ultrasonic signal generating circuit, 100...transparent member, 101...polarized light separation member, 1000...floating-in-space image display device, U...user, Uf...fingertip

Claims

1. A space floating image display device that forms a space floating image, A display device for displaying an image; A retroreflective member that retroreflects image light from the display device; Equipped with forming the floating image in space based on the light reflected from the retroreflective member; A sensor for detecting an operation state including a position of a user's finger relative to a spatial region including a surface of the space floating image or an object displayed on the surface; A tactile sensation generating device that generates a tactile sensation in the fingers by forming an area where the sound pressure of ultrasonic waves is the highest in the vicinity of a touch position on the surface of the floating image according to the position of the fingers based on the information detected by the sensor; an ultra-directional speaker that outputs ultra-directional sound toward the vicinity of the user's face; the superdirectional speaker detects a distance from a position of the user's face to the superdirectional speaker based on an image of the user captured by an imaging device, and forms a sound field by controlling a phase difference of an ultrasonic signal input to an ultrasonic output element of the superdirectional speaker based on the detection result; When outputting a type of sound that requires high confidentiality, a position of a face of the user is detected based on an image of the user captured by the imaging device, and the superdirectional sound is output toward the vicinity of the user's face by the superdirectional speaker; A space floating image display device, in which, when outputting a type of sound that does not require high confidentiality, the haptic generating device generates sound from near the position of the user's fingers or an object held by the user.

2. 2. The space floating image display device according to claim 1, the haptic sense generating device generates a sound from the vicinity of the position of the finger by modulating the ultrasonic signal with a sound signal; A floating image display device.

3. 2. The space floating image display device according to claim 1, The haptic sense generating device comprises: An ultrasonic element array in which a plurality of ultrasonic elements are arranged; an ultrasonic signal generating circuit that generates an ultrasonic drive signal to be input to each ultrasonic element of the ultrasonic element array based on information detected by the sensor; having The ultrasonic signal generating circuit generates the ultrasonic drive signal so as to provide one ultrasonic signal selected from ultrasonic signals having a plurality of different types of phases for each ultrasonic element. A floating image display device.

4. 2. The space floating image display device according to claim 1, The object includes an object that accepts a touch operation by the finger, A sound pressure is generated by the ultrasonic waves in the vicinity of the touch position of the finger on the surface of the floating image in space. A floating image display device.

5. 4. The space floating image display device according to claim 3, A housing in which the display device and the retroreflective member are housed; A transparent member provided in a part of the housing and transmitting light reflected from the retroreflective member; Equipped with The ultrasonic element array is arranged at a position outside the transparent member so that ultrasonic waves emitted from the ultrasonic element array are reflected by the transparent member and then strike the back side of the surface of the floating image in space. A floating image display device.

6. 4. The space floating image display device according to claim 3, A housing in which the display device and the retroreflective member are housed; A transparent member provided in a part of the housing and transmitting light reflected from the retroreflective member; Equipped with The ultrasonic element array is disposed at a position outside the transparent member so that ultrasonic waves emitted from the ultrasonic element array are not reflected by the transparent member and strike the back side of the surface of the floating image in space. A floating image display device.

7. 3. The space floating image display device according to claim 2, The modulation is an amplitude modulation. A floating image display device.

8. 3. The space floating image display device according to claim 2, The audio signal is a predetermined audio signal associated with a surface or object of the space floating image, or an operation on the surface or object. A floating image display device.

9. 2. The space floating image display device according to claim 1, The display device includes: A liquid crystal display panel for displaying images; a light source device that supplies light with a specific polarization direction to the liquid crystal display panel; having The retroreflective member retroreflects an image light beam having a narrow divergence angle as image light from the liquid crystal display panel, A light blocking member is provided in a space on an optical path connecting the liquid crystal display panel and the retroreflective member, and blocks image light having a divergence angle exceeding a specific angle from the liquid crystal display panel from entering the retroreflective member. A floating image display device.

Images